Why your AI feature isn’t working in production.

Most AI features fail the same way. The demo was impressive, the early sign-off was easy, and then production began to surface the gap between the prompt and the workflow. The model wasn’t the bottleneck. The bottleneck was the assumptions baked into the demo that production then quietly broke.

Evaluation, not enthusiasm

The first question we ask on any AI build is what success looks like in numbers — recall on a benchmark set, latency at the 95th percentile, cost per request, error rates by category. If the team can’t describe success this way, the project is being steered by feel, and feel is unreliable when the model is the part that can quietly drift.

Real data, real shape

Demos use clean data. Production sees ambiguous, incomplete, partially correct, partially malicious data. Before a feature ships, we run the model against the actual shape of inputs we expect, including the awkward ones. The result is usually a longer prompt, a tighter scope, and a fallback path the demo didn’t need.

Operational discipline

A working AI feature has logging, evals, a rollback plan, and a clear answer to the question of what to do when the model is wrong. None of that is glamorous. All of it determines whether the feature lasts past the first week.

The pattern, not the magic

When a build stalls, the fix is rarely a better model. It’s usually a tighter scope, a smaller decision the model is allowed to take, and a more honest measurement of how often it gets that decision right. The AI is the most exciting part of the project. The boring parts decide whether it ships.