Why “we only tweaked the prompt” is still a release
An AI service’s results depend on more than code. Responses are affected by the instruction, model and its version, generation settings, retrieval from a knowledge base, data templates, agent tools, and rules for handing work to a person. Changing any of these components can alter what a customer or employee receives.
That is why a prompt update should not be treated as a harmless text edit. For example, “answer in more detail” may improve consultations while also increasing cost, adding unsupported promises, or making the assistant more confident when the documents do not contain an answer. The more consequential the model’s action, the less reason there is to release the change to the entire workload at once.
A small business does not need to copy the processes of a large platform team. It needs a compact, repeatable control loop: record the current version, test a candidate on typical and risky examples, enable it for a limited share of tasks, observe the result, and have a simple rollback.
Start with the process outcome
Before testing, define what should improve. “Answers should be better” does not help make a decision. An operational requirement should describe observable behavior and the cost of a mistake.
For a sales assistant, criteria might include:
- does not invent stock availability or delivery dates;
- answers from the current price list and says when information is missing;
- does not disclose information about other customers;
- hands unusual requests to a sales representative;
- stays within an acceptable response time and per-task budget.
For an agent that drafts a request, add checks for format and required fields, as well as a prohibition on sending without an employee’s approval. For search over internal procedures, test not just whether the wording is useful but also whether it is grounded in the right document.
Separate criteria into three groups:
1. **Critical prohibitions.** Data exposure, an invented financial fact, or an action without authorization. One such failure may be enough to stop a release.
2. **Quality.** Completeness, relevance, correct routing, and an understandable refusal.
3. **Operations.** Response time, escalation rate, retries, cost, and the workload placed on employees.
You do not have to turn this into one universal score. First set thresholds for critical failures; an average score must not hide a dangerous individual case.
Build a small but honest test set
Take real types of requests or documents from the process and remove identifying information. You can begin with 30–100 examples if that is enough to cover different scenarios; this is a practical starting range, not a statistical guarantee. Include routine cases, rare exceptions, incomplete input, typos, conflicting information, and questions outside the assistant’s scope.
For each example, keep:
- the input and the context the system should retrieve;
- the expected result or list of required properties;
- a risk label: routine, sensitive, or requires a human handoff;
- the evaluation criterion and, where needed, a short expert explanation.
Do not make one “perfect answer” the only ground truth. In customer support, wording may vary while the facts, limitations, and next action remain correct. For these tasks, check concrete properties: whether a source is cited, whether a nonexistent date was promised, whether required fields are filled, and whether the correct queue was selected.
Use programmatic checks for exact fields: date format, allowed values, and required data. For semantic assessment, use expert review or a separate evaluator, but automated evaluators can also be wrong. Periodically compare their judgments with a person, especially for ambiguous and high-risk cases.
Compare the candidate with the current version
First run the same set through the active version and the candidate. Compare more than the overall success rate: inspect each important criterion and each scenario group separately.
A useful results table can be simple:
| Check | Current | Candidate | Threshold |
|---|---:|---:|---:|
| Critical violations | 0 | 0 | strictly 0 |
| Correct routing | 91% | 94% | at least current |
| Answers grounded in a source | 86% | 88% | at least 85% |
| Average task cost | RUB 7 | RUB 9 | within limit |
| Human handoff | 12% | 16% | investigate increase |
These figures only illustrate a comparison format; they are not industry benchmarks. The point is to judge whether the quality improvement is worth the added cost and whether it comes at the expense of safety.
Keep the test set and versions alongside the release record. If an issue appears later, the team can see which change preceded the regression. OpenAI’s evaluation guidance describes defining an objective, preparing data and criteria, and running comparisons as the basis of evaluation. You can choose any suitable tool: a spreadsheet with manual review, a custom script, a test environment, or a specialist system.
Release in stages, not with one big switch
After testing the set, check the candidate in staging or a copy of the process. Synthetic examples help test formats and edge cases, but do not replace anonymized real requests. Make sure the test environment cannot email a customer, change an order, or write to the production CRM.
Then use a limited rollout:
- **Shadow mode.** The candidate prepares an answer in parallel, but the old version responds; a person compares the results.
- **Small canary.** The candidate handles a small segment of low-risk requests, such as some internal queries.
- **Expansion.** Increase the share only after checking quality, errors, cost, and employee workload.
If the system takes actions rather than only suggesting text, start with recommendations or drafts. Keep sending an email, changing account details, issuing a refund, or writing to an accounting system under human control until the process and permissions have been verified.
Do not watch averages alone. Track the rates of errors, escalations, empty answers, and employee corrections; latency and cost per completed task; complaints; and cases where the model acted without sufficient grounds. Compare segments separately: an overall metric may hide that the new version works well on routine requests but fails on returns or warranty questions.
Make rollback shorter than the meeting about it
Before release, appoint a decision owner, observation period, and stop conditions. Specify who can restore the prior prompt, model, settings, or routing and how. Test rollback in a test environment first. If the version is stored separately from code, record its exact identifier and switching method; if it lives in application configuration, preserve the last known-good package.
Stop the rollout or revert if:
- a critical error or access-control violation appears;
- quality for an important scenario falls below the threshold set in advance;
- cost or latency rises enough to make the process unacceptable;
- employees have to correct more results than before the change.
Thresholds depend on the process. A draft internal answer can tolerate a different level of error from a payment calculation or contract change. Rollback is not an admission of defeat; it is a normal change-control mechanism.
Where prompt management fits
If instructions change often, store versions separately from application code and link them to evaluation results. In Langfuse, for example, prompt versions have a history and labels can point to staging and production; restoring an earlier version is done by moving the label. This illustrates an approach, not a required product choice: the same principle can be implemented in Git, a configuration store, or an internal service.
The same process applies to a local model. Record the exact weight version, quantization, chat template, system instruction, generation settings, and retrieval-index version in the test. If both the model and prompt change at once, it is hard to identify the cause of a failure. Change one factor at a time or explicitly compare combinations.
Also account for configuration propagation delays: a cache or long-lived process may still use the old prompt. Check the version actually used in the request log, not only the version shown in an administration panel.
The economics of a small control loop
A process owner does not need an expensive platform to begin. The minimum is a person responsible for business criteria, a developer or administrator able to switch versions, an anonymized sample, and a log containing the version identifier and evaluation outcome.
Include the full cost of a change:
- preparing and labeling examples;
- model calls during testing and production;
- human review and correction;
- retries and escalations;
- maintaining the test set and infrastructure.
Measure the cost of a successfully completed task, not the price of one model call. A cheap candidate that sends more requests to employees may end up costing more. Conversely, a more expensive response may be justified if it demonstrably reduces risk or saves specialist time.
A plan for the first week
- Choose one process and one outcome owner.
- Write down 3–5 critical criteria and stop thresholds.
- Collect and anonymize a small sample of routine and risky cases.
- Save the current version as a control point.
- Compare the candidate and active version on the same sample.
- Run in shadow mode and review discrepancies with an employee.
- Expand gradually; test rollback in advance.
The principle is simple: do not ask, “Did the answer sound nicer?” Ask whether the system completes the required task at an acceptable risk, cost, and level of manual work—and whether you can restore the previous behavior quickly. Then releasing AI changes stops being a leap of faith and becomes a normal, manageable process.