Why LLM Features Fail in Production: The Demo Data Was Clean

The demo used a handful of documents and every one of them was clean. Someone picked them. Not dishonestly, just naturally, because whoever built the prototype needed files to test against and reached for the ones that opened properly.

Then it went live. The first real batch arrived as photographs of forms shot at an angle, a scan where the top third is photocopier shadow, two languages inside one page, and a table that a human would struggle to read.

The model did not get worse. The input did. That gap is why LLM features fail in production while the prototype that convinced everyone still runs fine on the laptop it was built on.

science The demo proved the wrong thing

MIT's NANDA initiative looked at this across 52 organisations, 153 surveyed leaders and more than 300 publicly disclosed AI projects. Fortune's write-up of the report puts it plainly: roughly 5% of pilots reach real revenue acceleration and the rest stall. Lead author Aditya Challapally attributes it to a learning gap rather than model quality, with tools that cannot adapt to how a company actually works.

One line in that coverage deserves more attention than the 95% headline. More than half of generative-AI budgets go to sales and marketing tools, while the largest measured return sits in back-office automation. The money and the payoff are in different rooms.

That matches what I see. The features that survive are unglamorous. They process documents, reconcile records, route requests. Nobody demos them at a board meeting because there is nothing to look at, which is precisely why they were scoped properly in the first place.

description What a production corpus actually looks like

I built a document-processing platform that converts scanned PDFs and photographed forms into structured Markdown. It handles more than ten languages, including Urdu, Arabic, Amharic and Khmer. Manual keying on that work runs six to eight minutes a page and scales linearly forever, so the business case was never in question.

The engineering case was entirely in question.

Standard OCR falls over on non-Latin scripts. It also throws away the thing that makes a document useful: reading order, table structure, which line was a heading. You can get 95% character accuracy and still produce output nobody can query, because the invoice total ended up in a different row from the invoice.

Here is the part that catches teams out. A prototype corpus is curated by definition. Whoever assembled it opened each file to check it was suitable, and in doing so silently filtered out every case that would have broken the pipeline. The test set was cleaned by a human being who did not realise they were doing it.

build Four things that broke, and what fixed them

These are the decisions that made the difference between a pipeline that worked on a laptop and one that runs unattended. None of them involved changing the model.

The queue that starved itself

OCR is CPU-bound. Calling a hosted model is I/O-bound: you send a request and wait. Put both in one worker pool and they fight, because the pool is sized for one kind of work and is doing two.

Celery's own documentation notes that the prefork pool is the default and that concurrency defaults to the number of CPUs on the machine. That is a sensible default for compute. It is a terrible one for a task that spends most of its life waiting on someone else's API, where you have effectively sized your waiting capacity to your core count.

The fix was two queues rather than one.

# CPU-bound OCR: prefork, sized to cores.
#   celery -A app worker -Q ocr --pool=prefork --concurrency=4
#
# I/O-bound model calls: gevent, sized to how many
# requests you are willing to have in flight.
#   celery -A app worker -Q inference --pool=gevent --concurrency=64
 
app.conf.task_routes = {
    "tasks.run_ocr":        {"queue": "ocr"},
    "tasks.call_gemini":    {"queue": "inference"},
}

One caveat worth knowing before you copy that. The same Celery docs record that the gevent pool does not implement soft time limits and will not enforce a hard time limit if the task is genuinely blocking. Your timeout has to live in the HTTP client, not in the worker config, and that is not obvious until something hangs.

The request that should never have been a request

The instinct is to accept an upload and return the result. It works beautifully with a two-page PDF and falls apart with a 90-page scan, because now a web process is holding a connection open through a multi-minute pipeline.

Everything moved behind a FastAPI 202-accept pattern instead. The API takes the file, hands back an identifier, and gets out of the way. Work happens elsewhere and the caller polls. The web process stays around 300 MB of RAM regardless of what is being processed, because it is no longer doing the processing.

This is old, boring web architecture. It is also the single change that made the system operable, and it has nothing to do with AI.

The routing rule that had to be measured

Local OCR models are cheaper and faster, and they hit a hard accuracy ceiling on complex scripts. A hosted model handles those far better and costs more per page. So you route.

The temptation is to route on a hunch. What worked was routing on a measured threshold: pages where at least 25% of detected characters belong to a complex script bypass local OCR entirely and go to Gemini. Under that line, local wins on cost with no meaningful accuracy loss. Over it, local produces output that has to be redone by hand, which is the most expensive outcome available.

The number matters less than where it came from. It came from running both paths against real pages and finding the crossover, not from a meeting.

The failure that nobody saw

This one is the most dangerous, and it is the reason I built a separate system for it.

On a different project I needed per-application cost visibility from a single shared AI invoice, which meant capturing a usage receipt for every model call. Building that surfaced a second problem: when a provider degraded or an internal call started throwing, it went into a log and stopped there. No alert. The application kept running and kept returning something.

That is what makes LLM failure different from a crashed service. A crashed service is loud. A model that has started returning confident nonsense looks exactly like a model that is working, and the first person to notice is usually a customer.

So the capture path got a real-time sibling: dependency outages and internal errors raised as structured, de-duplicated alerts with an audit trail. The whole thing sits behind a durable local outbox with at-least-once, idempotent ingestion, which produces an exactly-once effect. Kill any single component and you lose freshness, never accuracy. It carries 144 passing tests across integrity, resilience and outage handling, and that number exists because I did not trust the design until something proved it.

payments The cost shape nobody models

Pilot budgets are built on a per-request average. Production does not bill you an average.

Gemini's rate-limit documentation is worth reading before you forecast anything. Limits apply across requests per minute, tokens per minute and requests per day at the same time, and the documentation is explicit that exceeding any one of them triggers an error. Twenty-one requests in a minute against a limit of twenty fails regardless of how few tokens you used. Overages return 429 RESOURCE_EXHAUSTED.

Two consequences follow, and both get missed:

1. Retries are a cost multiplier, not a safety net. A retry on a 429 is another billable call if it succeeds, and your naive cost model counted one.
2. Bursts are the real unit. Fifty pilot users trickling requests through the day behave nothing like five thousand production users arriving in the same fifteen minutes, even at identical daily volume.

If you want to size this properly before committing, the LLM API cost calculator will take token counts and model choice and give you a monthly figure. And the approach that cut one document-AI bill by 99% is task-based routing, which is the same idea as the threshold above applied to the whole system.

checklist What to check before you promise a date

Short list. Everything on it comes from something that went wrong.

• Pull fifty inputs at random from real data. Not curated, not reviewed. Random. Run them and look at the failures rather than the score.
• Separate fast work from slow work at the queue level before you need to, because retrofitting it under load is miserable.
• Make long work asynchronous from day one. Accept, acknowledge, process elsewhere.
• Decide what a wrong answer looks like and write something that detects it. A silent failure you cannot see is worse than a crash you can.
• Model cost in bursts, with retries counted, against every rate-limit dimension rather than the friendliest one.

None of that is model work. That is the point, and it is roughly where the effort actually goes.

quiz Frequently asked questions

The gap is engineering, not intelligence

The uncomfortable conclusion from the MIT work is that most of this money is not being lost to bad models. It is being lost between a prototype and a system, in integration and operations, on exactly the sort of work that never makes a good demo.

Both systems described here are written up in more detail: the document processing platform and the cost attribution and outage monitoring build. The architectures are worth more than the summaries.

If you have a pilot that impressed everyone and then stopped moving, that is a familiar shape and usually a fixable one. Tell me where it stalled and I will tell you whether it is a scoping problem or a plumbing one.