Evaluate the whole system before you connect anything

Every agent demo works. That is what makes them useless as evidence. The interesting behaviour is not what a system does on the path somebody rehearsed, it is what it does when a connector times out, a record is missing a field, or the answer genuinely is not in the corpus.

Seeded data is not a compromise

Running the whole system on seeded data is often assumed to be a weaker test than running it on real data. It is usually a stronger one, because you control the inputs. You can include the malformed record, the duplicate, the entry with a date in the wrong format, and the question whose answer does not exist. Real data rarely offers you all of those on demand.

What to watch for while it runs

The behaviours below separate a system that will survive production from one that will be quietly switched off in month three.

• Does it say when it does not know, or does it produce something fluent
• When a connector fails, does it stop or does it continue with partial data
• Can you see why it retrieved what it retrieved
• Does the same input produce a defensible answer twice
• When it escalates, is the handover complete enough to act on

The procurement argument

There is a second reason to build this way, and it is not technical. A system that requires production credentials before it can be assessed forces a security review to happen under time pressure, at the exact moment somebody wants to start. A system that runs without any live key can be evaluated by the people who need to evaluate it, on their own schedule, before anything is at stake.

If a vendor needs your API keys before you can see the system work, you are being asked to buy on trust.

We build demo mode into everything for both reasons, and it stays after launch, because it is also where you test changes.