From LLM Prototype to Production: What Actually Breaks

Why the Demo Was Never the Hard Part

Most LLM prototypes follow the same path: someone wires a model to a real prompt, feeds it a handful of well-chosen examples, and gets output that impresses in the meeting. The prototype works because every input is curated, every user is the builder, cost is invisible, and failure looks like a funny screenshot rather than a support ticket.

Production changes all five conditions at once. The inputs become whatever the world sends, the users are people with different permissions, the bill becomes line items finance reviews, and failure becomes a customer seeing the wrong thing. None of these are model-quality problems. They are engineering problems, and they are where projects actually stall.

Break One: Nobody Defined What Good Looks Like

A prototype is evaluated by its author looking at the output. That does not scale past ten users. In production you need an evaluation set: a fixed collection of real inputs with expected outputs or at least expected properties, run automatically whenever the prompt, model, or retrieval layer changes.

Without evals, every change is a judgement call made from memory. Teams end up either freezing the system so nothing regresses, or shipping prompt edits on vibes and discovering regressions through complaints. Both are slower than writing the eval set while the prototype's failure modes are still fresh.

• Build the eval set from the first week of the prototype, not after launch: fifty to a hundred real cases is enough to start.
• Score what matters for the business decision the output feeds, not generic fluency.
• Re-run the set on every prompt or model change, and keep failures as new cases.

Break Two: Permissions Stop at the Demo

In the prototype, the model is usually given everything: the full document store, the whole database export, every ticket. That is fine when the only user is the person who assembled it. In production, the model answers a specific user, and that user should only see what they are allowed to see.

The engineering answer is retrieval-layer permissions: every document, row, and ticket carries an access scope, and the retrieval step filters by the requesting user's scope before anything reaches the model. Retrofitting this after launch means re-indexing with permission metadata and re-testing every workflow that touches the model. Doing it in the first release is a fraction of the work.

Break Three: The Bill Nobody Metered

Token costs look trivial at prototype volume and stop looking trivial when the workflow runs on every document, every ticket, or every customer session. The two cost failures we see most often are the same two: no per-workflow metering, so nobody can say which feature costs what; and no caps, so a bug that loops a retrieval step runs up a bill overnight.

The fix is unglamorous: meter usage per workflow and per tenant, set budgets with alerts before launch, and cache aggressively where the same input will recur. Teams that do this before go-live treat token cost as a product metric. Teams that skip it treat the first surprise invoice as a incident review.

Break Four: Guardrails Are Not Optional

Any LLM that takes untrusted input is an attack surface. Prompt injection through a support ticket, a web page, or an uploaded document can steer output, exfiltrate context, or simply produce something the brand cannot publish. Guardrails means three concrete layers: input handling that treats user content as data rather than instructions, output validation that checks structure and blocks known-bad patterns, and human review for anything with external or financial consequence.

It also means redaction before model calls where personal data is involved. EU delivery makes this the baseline: the model should see what it needs to answer, not everything the system holds.

Break Five: Drift Is Invisible Without Instrumentation

Model behaviour changes when the provider updates a model version, when your retrieval corpus shifts, or when usage patterns move to edge cases you never tested. In a dashboard demo nothing looks wrong; in production the wrong answer looks exactly like a right one, only it is not.

Production LLM systems track input and output distributions, eval scores over time, latency, and cost per call, with alerts on each. This is the same monitoring discipline as any other production service, applied to outputs that cannot be validated by a type checker.

The Order That Works

When we take a prototype into production on a client project, the order is fixed: evals first, because they define done for everything after; permissions and redaction next, because they shape the data layer; then metering and budgets; then guardrails; then monitoring. The model itself is usually the last thing touched, and often not at all.

That is the honest summary of the gap. The demo proved a model can do the task. Production is the work of making a system that does the task reliably, affordably, and safely for people who were not in the room when it was built.