Your AI agent works in the demo. That's when the work starts

Why does a home-built AI agent look finished so fast?

Because a demo tests the part that has become easy. With a model API, a knowledge base and vibe coding (describing what you want and letting an AI write the code), a small team can have a chat agent answering questions within days. It understands plain language, and its first answers are convincing.

A demo usually checks one question and one answer. We wrote about how to test more than that in How to test an AI customer support agent before you buy. This article is about what starts once the agent is live.

Gartner expects over 40% of agentic AI projects to be canceled by the end of 2027, because of rising costs, unclear business value or inadequate risk controls (Gartner, June 2025).

What breaks first when real customers use it?

Conversations that don't go in a straight line. A customer starts one request, asks about something else halfway through and then comes back.

A customer writes: "I was charged twice this month." The agent starts looking into it and asks for the dates on both invoices. The customer asks instead: "By the way, how do I add a teammate to our account?" The agent explains. A few messages later: "OK, but what about the double charge?"

That's an ordinary conversation. For the agent it's bookkeeping: which issue is still open, and what it has already collected for it. A simple build answers the latest message and forgets the charge. A more careful one keeps the whole conversation in context and still loses track of the unfinished task, or asks for the invoice dates again.

Researchers have measured this. In LLMs Get Lost In Multi-Turn Conversation, a team from Microsoft Research and Salesforce Research simulated more than 200,000 conversations. Top models performed 39% worse on average when the same task arrived over several messages instead of one. The authors found that models make assumptions early, and once a conversation takes a wrong turn they rarely recover.

So someone has to design how scenarios interact. What happens when one is interrupted? How does the system keep track of unfinished tasks, and how does it decide which one to continue? That person also needs to know the business well enough to say what each scenario needs from the customer.

In Misterio this is part of how the agent handles a conversation. It keeps the unfinished scenario open while it answers the side question, then picks it up from the step where it stopped, without asking again for what the customer already told it.

What does an AI agent need after launch?

Ongoing work in three places: the content it answers from, the models it runs on, and the tests that tell you whether a change broke something.

Does an AI support agent need retraining?

Not in the way most people expect. You don't retrain the language model on your conversations. In practice, "retraining" means this:

• updating the knowledge base when a policy, price or procedure changes;
• changing scenarios when real conversations show a gap;
• rerunning your tests after each of those changes.

The second one is the one teams underestimate. In the payment scenario from our first article, two of the rules that mattered most were added after launch, once we had read real conversations.

How often do AI models change?

More often than most plans assume. Providers retire models on their own schedule. Anthropic, for example, commits to at least 60 days' notice: Claude Sonnet 4.5 was deprecated on September 30, 2026 and is due to be retired on November 30, 2026, and cloud platforms that resell the same models keep their own dates (Anthropic model deprecations). In that list, models are often retired one to two years after release.

A retirement at least comes with a date. A new version that behaves differently doesn't. Here are four cases from our own logs, without names:

• A newer version of a model we use started reasoning by default. On short steps it spent its whole answer budget thinking and returned an empty reply.
• A provider documented a setting that its own servers rejected for one model. That step had never worked with that provider, and nobody noticed, because its backup model answered every time.
• A model returned broken JSON on 13 of 24 calls, and 10 of those were marked as a normal finish.
• Two models from the same provider used different names for the same answer-length limit.

None of these looked like an outage: replies kept coming, just worse or more expensive than they should have been, until someone noticed.

Here's what that costs us. In the 11 months since our first commit, 229 of 3,388 code changes at Misterio went into connecting, adapting or fixing AI models. That's about 1 in 15. In July and August 2026, when we were adding several new providers, the share rose to 15 to 19%. We've built integrations with 11 AI providers, several of them in more than one region. A step can have a backup model, and calls switch to it automatically when its provider keeps failing.

How do you know a change didn't break anything?

You run the same test conversations before and after it. Our release protocol does this through the agent's full pipeline, and requires a full run of more than 1,500 turns when the model that writes replies changes. The tests have to cover what your customers actually bring, including interruptions, wrong attachments and requests for a person, not only the FAQ.

Whoever owns the agent owns this job. When you look at vendors, ask how they test before they move your agent to a new model.

What do GDPR and security require from an AI support agent?

The same as any system that handles customer data, plus a few things specific to AI. If you build it yourself, each of these is yours:

• A data processing agreement with every AI provider you send conversations to, and an up-to-date list of subprocessors for your own customers.
• Transfers outside the EEA. If a provider processes data elsewhere, you need a legal basis such as the EU Standard Contractual Clauses.
• Retention and deletion in every copy. Conversation history, logs of AI calls, the search index behind your knowledge base and backups. A deletion request has to reach all of them.
• Masking personal data before it reaches a model where you can, and knowing which steps still see it.
• Telling people they're talking to an AI. Since August 2, 2026, Article 50 of the EU AI Act requires it (European Commission). The Commission's guidelines list chatbots in customer support among the examples, and the disclosure has to come no later than the first interaction.
• Prompt injection. A customer, or a document the agent reads, can try to change its instructions. OWASP ranks it first in its Top 10 for LLM Applications 2025.
• Access control. IBM's 2026 Cost of a Data Breach report found that roughly one in five organizations had a breach involving their AI models or applications, and 92% of those lacked proper AI access controls.

Then there's liability for what the agent says. In Moffatt v. Air Canada (2024), a Canadian tribunal held the airline responsible for wrong information its website chatbot gave a customer and rejected the argument that the chatbot was responsible for its own words (case comment on CanLII). For a European view, an expert report for the EDPB's Support Pool of Experts works through the privacy risks of an LLM-based chatbot for customer queries step by step (EDPB, April 2025).

At Misterio, retention periods for conversations and AI call logs are published in our Data Processing Agreement, which also lists every subprocessor and where it processes data. Personal details are masked automatically in the conversation history the agent works with, and our chat window marks the agent as an AI.

What should you check before buying a SaaS AI agent?

Buying removes the infrastructure work, but not every decision. A few questions show where the trade-offs are, and they're worth asking any vendor:

1. Can you change scenarios yourself, and can you see why the agent answered the way it did? Without that, every fix becomes a ticket to the vendor.
2. How does the price grow with volume? Per seat, per conversation and per resolved case behave very differently when your traffic doubles.
3. Who are the subprocessors, where do they process data, and how long are conversations kept? The answers belong in the DPA, not just in a sales call.
4. Can you export your knowledge base and conversation history if you leave?
5. Who sets the agent up and tests it before launch? If the answer is "you, in a few minutes", plan for your own team's time.
6. How does the vendor handle model changes? Ask whether they retest your scenarios before switching, and whether they tell you.
7. Does it work inside the helpdesk and systems you already use, or will your team have to move?
8. What does a person get when the agent hands a conversation over? In a 2026 Gartner survey, 87% of customers said companies using GenAI in customer service must give them access to a human agent (Gartner, August 2026).

How do you decide: build, buy self-serve or buy with setup?

Decide by who will own the work after launch.

My view: build in-house if AI customer support is part of your product and you have engineers who will own it after launch, not just build it. Buy if customer support is how you serve your product rather than the product itself. That advice is wrong for a company with a strong ML team and data or compliance requirements no vendor will accept. For them, building is often the only option.

At Misterio, our team sets the agent up with you: scenarios, the knowledge base, when to bring in a person, and tests before launch. After that we handle the models and retest when they change, and your team can still change scenarios without code. When a case needs a person, Misterio passes the whole conversation to your team in the helpdesk you already use.