Why a new model is more than a new file

A local LLM is often upgraded like an ordinary library: download newer weights, change a configuration path and restart the service. That is enough for a demonstration. In a business workflow it can introduce a silent regression: the model changes its JSON structure, stops following the required tone, extracts dates less accurately, takes longer to produce the first token or invokes tools more often.

Weights are only one part of the behaviour. A release also includes the tokenizer, chat template, quantisation, generation settings, system prompt, LoRA adapter, structured-output schema, tool list, retrieval-index version and inference-server configuration. Changing one component can alter the entire system.

The unit of deployment should therefore be an immutable release bundle. Each candidate records:

  • the exact weight revision and cryptographic digest;
  • tokenizer and chat-template versions;
  • quantisation format and parameters;
  • adapter version when LoRA is used;
  • system prompt, templates and JSON schemas;
  • embedding and reranking models used by RAG;
  • inference-server image and launch parameters;
  • evaluation suite, approval date and owner;
  • tool restrictions and human-approval mode.

MLflow Model Registry demonstrates a useful mechanism: models have versions, tags and aliases such as `candidate` and `champion`. An alias can be reassigned without changing client code. However, an alias is a mutable pointer, not evidence of what actually served a request. Each request log still needs the resolved version and release-bundle digest.

Gate one: evaluation on a fixed dataset

A public benchmark compares general capabilities, but it does not show whether a new version correctly fills a customer record or creates an ERP draft. Before shadow deployment, build a business-specific golden dataset that represents the real task classes.

Include successful and difficult examples, short and long requests, Russian professional abbreviations, empty fields, contradictory documents and attempts to push the model beyond its role. Remove or replace personal data and restrict access to original records.

Each example needs a verifiable outcome:

  • required fields are present and use allowed types;
  • dates, amounts and identifiers are supported by a source;
  • answers cite only authorised documents;
  • prohibited tools are never invoked;
  • external actions remain drafts until approval;
  • missing information triggers a clarification request;
  • average cost and latency remain inside the agreed range.

One aggregate score is not enough. Code should verify structures, permissions and business rules. Process specialists should judge usefulness on a representative sample. Release gates must be established before testing rather than inferred from a few impressive answers.

Gate two: shadow traffic without side effects

After offline validation, the candidate receives copies of real requests but does not answer the user. The current `champion` continues to serve the workflow, while candidate outputs are written separately for comparison.

Mirroring happens after authentication and access-policy evaluation. If a request contains sensitive fields, the shadow instance receives the same access, masking, retention and deletion controls. Calling the deployment “only a test” does not change its data-processing obligations.

Candidate tools must be disabled or placed in dry-run mode. The model may propose CRM JSON but not create an opportunity, draft an email but not send it, and display accounting parameters but not post a document. Otherwise, the shadow test becomes a second source of side effects.

Each result pair shares one request identifier and records both release-bundle versions. Comparison runs asynchronously so the candidate cannot increase production response time. Automated checks, specialist ratings and the reason for each meaningful difference are stored.

Shadow mode answers the important question: is the candidate better on today's traffic, not just on an old dataset? It also reveals real GPU load, context lengths and queue behaviour.

Gate three: a canary with limited blast radius

Once shadow results pass the gates, a small share of live traffic moves to the candidate. Kubernetes documentation describes the common pattern of running a new release beside the previous one and sending it production traffic before full rollout.

For LLMs, routing should be stable by session, department or process type rather than random for every message. Otherwise, a conversation jumps between models and context differences make the comparison unreliable. A rollout can start with one internal team or 1–5% of eligible sessions.

At first, the new server handles only low-risk scenarios. External actions continue through the existing deterministic executor and approval rules. If the model changes tool-argument structure, the gateway rejects the call rather than guessing intent.

The canary stops when any critical threshold is breached: invalid structures, manual corrections or tool failures increase; p95 latency worsens; queues grow; or accepted outcomes decline. The threshold and observation window must be recorded before rollout.

Metrics to collect

vLLM exposes Prometheus metrics through `/metrics`. Useful release comparisons include running and waiting requests, KV-cache utilisation, time to first token, inter-token and end-to-end latency, input and output token volumes, and completed-request counts. The model label separates serving instances.

Infrastructure metrics need a business layer beside them:

  • outcomes accepted without editing;
  • invalid JSON or business-rule violations;
  • human escalation and review time;
  • required-field accuracy on the controlled sample;
  • failed and repeated tool actions;
  • cost per accepted outcome;
  • complaints, cancellations and post-action corrections;
  • results segmented by language, document type and department.

Traces generally need identifiers, versions, timings, statuses and de-identified ratings. Full prompts and responses should not automatically flow into a shared observability system because they may contain personal or commercially sensitive information. Diagnostic content access should be enabled separately, for a limited period and for named roles.

Rollback is a separate operation

Reassigning an alias or route makes switching fast, but a genuine rollback requires a prepared previous environment. The old release bundle must remain available, its image must be verified and clients must remain compatible with both versions.

With two GPUs or nodes, old and new models can stay warm in parallel. If a single card cannot hold both bundles, a zero-downtime promise may be physically impossible. The choices are a temporary second node, a smaller candidate quantisation or an honest maintenance window. Measure model-loading time in advance.

Before routing changes, new requests stop entering the faulty instance; in-flight work either drains or terminates under a defined policy. Side effects use idempotency keys so a retry after rollback cannot create a second invoice, message or record.

API and tool schemas remain backward-compatible throughout the rollback window. If the candidate requires different JSON, a versioned adapter performs the translation. Changing the model, data schema and business workflow at the same time makes failures much harder to diagnose.

Security and change governance

New weights should not enter production automatically from an external model hub. Files go through quarantine and format and licence checks; the revision is pinned, digests are computed, and remote code execution is denied by default. The approved bundle is copied into an internal repository.

Candidate permissions must never exceed those of the current model, particularly network access, secrets and tools. A shadow instance that “does not show anything to users” can still read data or call systems if it has credentials.

NIST AI RMF offers a useful responsibility model: the process owner defines an acceptable outcome, IT owns deployment and rollback, security owns data and access controls, and a named manager authorises traffic expansion. Approval history and rollback reasons stay with the release record.

Upgrade economics

A parallel instance, specialist evaluation and temporary GPU headroom have a cost. That cost should be compared with the expected cost of regression, not with zero.

Consider a modelled example: a workflow produces 20,000 outputs per month. An untested upgrade increases errors by two percentage points. Each correction takes eight minutes and fully loaded labour costs RUB 1,200 per hour. The additional monthly cost is about RUB 64,000: `20,000 × 2% × 8/60 × 1,200`.

This is not a forecast, but a calculation with explicit assumptions. A real budget also includes the second node, electricity, specialist time, trace storage and the risk of external actions. In a high-risk workflow, one incorrect transaction can cost more than the entire pilot.

A minimum implementation plan

A safe first cycle can follow these stages:

1. Freeze the current release bundle and build a golden dataset.
2. Register the candidate with digests, an owner and admission criteria.
3. Run functional, load and abuse-resistance tests.
4. Enable shadow traffic with all side-effecting tools disabled.
5. Review differences and obtain process-owner approval.
6. Route a small, stable cohort of sessions to the candidate.
7. Test rollback under controlled load.
8. Expand traffic only after a sufficient observation window.

The management conclusion is simple: a local LLM upgrade changes a production process; it is not a contest between model numbers. If the organisation cannot identify the exact serving version, measure its outcome and restore the previous one, the release is not ready.