从 AI 实验走向生产系统

Prototypes are easy to admire. A capable model, a well-chosen prompt, and a few sample documents can produce a convincing demonstration within days. The harder question comes afterward: can this run every day, for every user, on real data, at an acceptable cost, with someone accountable when it goes wrong?

In our engineering practice, when an AI initiative stalls between pilot and production, the cause is more often the missing system around the model than the model itself. A notebook or prompt playground hides the work production requires: identity and data access, integration with systems of record, latency and cost limits, failure handling, evaluation, monitoring, and clear ownership.

This guide walks through the transitions that turn an experiment into a supported production service, and the decisions each one forces.

Define the job before choosing the model

Start with the decision or task, not the technology. Who consumes the output? What happens downstream when it is wrong? Is a false positive cheaper than a false negative, or the reverse? An AI step that drafts a reply for a person to edit has a very different risk profile from one that automatically closes a claim, even if the underlying model is identical.

Write acceptance criteria before building anything substantial. A useful definition of production-ready includes:

• A quality threshold measured on a held-out evaluation set that business owners agree represents real work
• A latency budget expressed as a percentile, such as p95, rather than an average
• A cost ceiling per transaction or per successful outcome
• A defined fallback for when the AI step is unavailable or unsure
• A named business owner and a named engineering owner

Compare against a baseline. Sometimes a rules engine, a better search index, or a simple classifier captures much of the value with far less operational burden. Knowing the baseline also tells you whether the AI step is earning its place.

Build the evaluation harness first

The evaluation harness is the most valuable production asset in an AI system, because it is what allows you to change everything else safely. Assemble an evaluation set from real inputs, labeled by people who understand the domain. Hold a portion back so it is never used while tuning prompts. Organize cases into slices — document types, languages, customer segments, edge cases — because aggregate scores hide regressions in exactly the slices that matter most.

Choose metrics that fit the task. Extraction suits field-level precision and recall. Classification suits a confusion matrix, with attention to the costly errors. Open-ended generation needs a written rubric scored by a model-based grader that is periodically checked against human reviewers, plus regular human spot checks. Offline evaluation gates every release; online monitoring catches what offline evaluation missed.

Engineer the data and integration path

Pilots often run on exported files. Production runs on live data, which turns access control, freshness, and data protection into engineering requirements. Mirror the permissions of source systems instead of granting the AI service broad access. Minimize the personal data sent to models, redact where the task allows, and confirm where data is processed and retained — including by any model provider — against your data residency and contractual obligations.

Expose the capability through a stable internal API with a versioned contract, so consuming applications are insulated from model and prompt changes. Use asynchronous processing with queues for long-running or batch work, and reserve synchronous calls for interactions where users are waiting and latency allows. A model gateway is often worth building early: a single internal service that manages provider credentials, rate limits, logging, routing between models, and failover to an alternate provider. It reduces lock-in and gives you one place to enforce policy.

Design for failure, latency, and cost

Model APIs time out, rate-limit, and occasionally return malformed output. Treat them like any other unreliable dependency: set timeouts, retry with backoff where calls are safe to repeat, use circuit breakers, and validate structured outputs against a schema before anything downstream consumes them. Decide in advance what the degraded experience looks like — routing work to a manual queue, falling back to a rules-based path, or showing results without the AI enhancement.

Latency and cost are design inputs, not tuning exercises for later. Stream responses where users read them as they arrive. Route simple steps such as classification or field extraction to smaller, faster models and reserve larger models for the steps that need them. Cache stable context. Use batch interfaces for non-interactive workloads where providers offer them. Above all, measure cost per successful business outcome rather than per API call; a cheaper model that needs more human correction can cost more overall.

Release gradually and operate continuously

A production launch is a sequence, not an event. A staged rollout limits the impact of anything the evaluation set did not anticipate:

1. Shadow mode: the AI step runs on live inputs, but its outputs are logged and compared rather than used.
2. Internal users: a small group uses the feature with an easy way to flag problems.
3. Canary release: a limited share of real traffic, behind a feature flag, with predefined rollback criteria.
4. General availability: full traffic, with monitoring and the ability to disable the feature instantly.

After launch, monitor inputs as well as outputs. Input drift — new document formats, new products, seasonal language — often degrades quality before anyone files a complaint. Capture implicit feedback such as user edits and overrides, sample outputs for human review, and route confirmed failures into the evaluation set so each mistake becomes a regression test. Plan for model lifecycle events too: providers deprecate versions on their own schedules, and every upgrade should pass the same evaluation gate as a code change.

Make governance part of the pipeline

Governance works best when the delivery process produces it rather than when it is assembled afterward. The NIST AI Risk Management Framework and ISO/IEC 42001 both emphasize documented risk assessment, measurement, and accountability, and teams serving users in the European Union should also assess where each use case falls within the EU AI Act's risk categories. In practice, the evaluation reports, version history, approval records, and incident reviews generated by a well-run pipeline cover much of what these frameworks ask for, and they stay current because they are produced on every release.

What changes between pilot and production

The model is often the least changed component between a successful pilot and a production system. What changes is everything around it: explicit acceptance criteria, an evaluation harness that makes change safe, governed data access, engineered failure handling, measured unit economics, staged releases, and clear ownership.

Investing in those foundations once also makes the second and third use cases faster to deliver, because the gateway, the evaluation practice, and the release process can be reused. That is the real return on moving carefully from experiment to production: not one working feature, but a repeatable way to ship the next one.