The dangerous journey from AI proof of concept to production

A proof of concept gets clean inputs, attentive experts and a carefully chosen demonstration. Production gets ambiguous requests, stale content, concurrent users, vendor outages, adversarial input and somebody asking why a bad answer was allowed to affect a customer.

The proof of concept may be successful. It simply proved less than the organisation thinks.

The journey becomes dangerous when a compelling demo is treated as a nearly finished service. Production is not the pilot with more servers. It is a different evidence standard and a permanent operating commitment.

A pilot should test a hypothesis

Start with a falsifiable statement: for a defined group and task, the service can improve an outcome within constraints on quality, risk, latency and cost.

Specify:

• the users and workflow;
• baseline performance today;
• data and integrations required;
• expected benefit and acceptable harm;
• evaluation method;
• human decision and fallback; and
• stop, continue and scale criteria.

“Show that generative AI can help” is not a hypothesis. It invites a polished demo with no route to a decision.

Use representative difficult cases early. If the pilot excludes low-quality documents, long-tail requests, permission boundaries and peak load, record those as unproven—not future implementation details.

Production exposes the whole system

Data and context

Curated pilot data hides missing ownership, version conflict, retention, access control and deletion. Production needs ingestion and data contracts that keep evidence current.

Evaluation

A handful of impressive answers is not a test set. Define task-specific cases, expected evidence, safety cases and acceptable uncertainty. Separate retrieval, generation, tool execution and outcome quality.

Reliability

Plan for timeouts, malformed responses, rate limits, model or provider changes, partial connector failure and retry storms. Establish SLOs, queues, circuit breakers, idempotency and graceful degradation where needed.

Security and privacy

Threat-model prompt injection, data leakage, excessive tool agency, insecure output handling and supply-chain change. Propagate identity and least privilege. Protect traces and evaluation data, not only the production database.

Operations

Name the on-call and product owners. Give support teams diagnostic evidence. Define incident severity, rollback, kill switch, user communication and vendor escalation.

Economics

Model cost at expected and adverse load: tokens, retrieval, parsing, storage, external tools, observability, human review and engineering support. Average cost per query can hide a long tail of expensive sessions.

Adoption

A technically accurate system can fail because it interrupts work, produces answers people cannot verify or shifts accountability without permission. Test the workflow with the people who must use and supervise it.

Build a production-readiness gate

Before launch, require evidence proportionate to risk:

1. the business owner accepts a defined residual risk;
2. representative evaluations meet agreed thresholds;
3. forbidden and access-denied cases pass;
4. latency, concurrency, failure and cost tests cover expected load;
5. users can verify, correct and escalate outputs;
6. observability reconstructs a decision without retaining unnecessary sensitive data;
7. incident, rollback and continuity procedures are exercised;
8. vendor and model changes have a regression path; and
9. success and retirement measures are funded after launch.

An executive's enthusiasm is not a substitute for any of these.

Release in controlled slices

Move through increasing consequence:

• offline evaluation;
• internal sandbox with synthetic or approved data;
• shadow mode alongside the existing process;
• advisory use with mandatory human review;
• limited cohort or domain;
• broader release after measured performance; and
• automation only where risk, reversibility and evidence justify it.

Feature flags, quotas and scoped tool permissions keep the blast radius small. Preserve the existing route until the new service has demonstrated operational reliability.

Observe quality, not only uptime

An AI endpoint can return HTTP 200 while degrading badly. Track model and prompt version, retrieval evidence, tool calls, cost, latency, refusal, user corrections and outcome signals. Segment by domain and risk because averages conceal fragile areas.

Use traces to diagnose, not to collect everything forever. Apply access, minimisation and retention to prompts and retrieved content.

Production failures should become evaluation cases. Evaluation should gate the next change. That loop is the engineering system around the model.

Know when not to scale

Stop or redesign when:

• the task lacks reliable data or ownership;
• benefits disappear after human review and operating cost;
• users cannot recognise harmful errors;
• the required autonomy exceeds the evidence;
• latency or cost cannot meet the workflow;
• a deterministic rule, search or conventional model works better; or
• accountability remains unresolved.

Killing a pilot is not failure if it prevented an unreliable service from becoming institutional debt.

What to measure

• task outcome against the pre-AI baseline;
• quality and safety by risk segment;
• correct escalation and human override;
• p50, p95 and p99 latency and cost;
• tool and retrieval failure rates;
• incidents and near misses;
• adoption, abandonment and repeat use;
• support effort and time to diagnose;
• model/provider changes passing regression; and
• realised value after full operating cost.

The counterargument: this discipline kills speed

Applying the heaviest controls to every experiment would kill speed. Use risk tiers and reusable paved roads: approved providers, identity, logging, evaluation harnesses and deployment patterns.

Speed comes from making the safe path easy, not from pretending production concerns can wait until after success. A narrow, well-instrumented release is faster to learn from than a broad launch that nobody can explain.

Adovo’s view

A demo answers, “Can we make this look possible?” Production must answer, “Can we operate this responsibly on a bad day?”

The gap is not bureaucracy around the model. It is the product, engineering and governance work that makes the model worth trusting.

Primary documentation and further reading

• NIST AI Risk Management Framework
• Microsoft RAG evaluation guidance