Why “reply in JSON” is not enough
A local language model is often connected to a CRM, service desk, or accounting platform through a simple contract: the model receives an email or document and returns JSON containing the request type, extracted details, and next action. It works in a demo. In production, an extra comma appears, a field is renamed, free text precedes the object, or the model invents a value outside the approved catalogue.
A conventional prompt only asks the model to follow the format. It does not prevent the model from selecting a token that makes the final string incompatible with the expected structure. A repair request such as “fix the JSON” may reduce errors, but it introduces latency, compute cost, and another nondeterministic step.
Guided generation addresses a narrower problem. At every decoding step, it permits only tokens that can still lead to a string accepted by a grammar or schema. The `llama.cpp` documentation describes GBNF as a way to constrain output; the stack can also convert a supported subset of JSON Schema into a grammar and accept schemas through its server parameters.
This is a major interface improvement, but it is not an automatic guarantee of a correct business result. Valid JSON can contain a wrong tax identifier, a nonexistent customer, a date in a closed accounting period, or a confidently fabricated amount. A production architecture therefore needs several sequential gates, not just one schema.
What constrained generation actually guarantees
A grammar controls response shape: brackets, fields, types, enumerations, and selected length or item-count constraints. The Outlines authors describe the general mechanism as computing allowed continuations through finite-state-machine transitions and masking other model tokens.
In practice, this delivers three benefits:
- parsers stop receiving accidental prose around the object;
- integrations receive predictable field names and types;
- some errors are prevented during generation rather than discovered after a system write.
The guarantee ends at syntax and explicitly described structure. A schema does not verify factual truth, user authority, deal state, or the consequences of an action. `llama.cpp` also notes that the schema constrains output but is not injected into the prompt, so the meaning of the expected fields still has to be explained to the model. Only a subset of JSON Schema is supported, and complex grammars may affect performance.
The resulting rule is: **constrained generation replaces JSON repair, but it does not replace data validation or action control**.
A minimal schema for request classification
Assume an incoming message needs to become a CRM card. Start with a small object instead of trying to mirror the entire system interface:
```json
{
"type": "object",
"properties": {
"request_type": {
"type": "string",
"enum": ["sales", "support", "billing", "unknown"]
},
"customer_id": {"type": ["string", "null"]},
"summary": {"type": "string", "minLength": 1, "maxLength": 500},
"confidence": {"type": "number", "minimum": 0, "maximum": 1},
"evidence": {
"type": "array",
"items": {"type": "string", "maxLength": 300},
"maxItems": 5
}
},
"required": ["request_type", "customer_id", "summary", "confidence", "evidence"],
"additionalProperties": false
}
```
`required` must be explicit: listing fields under `properties` does not make them mandatory in JSON Schema. `additionalProperties: false` blocks unexpected keys. The enumeration limits routes to an approved catalogue, while `unknown` lets the model avoid choosing between inappropriate alternatives.
Do not interpret `confidence` as a measured probability. It is a routing signal that needs calibration on your own examples. `evidence` is often more useful because it asks the system for short supporting excerpts that a person or deterministic rule can compare with the source message.
The seven-gate architecture
**1. Input normalisation.** Email, form, or document is transformed into a common object. Attachments pass malware scanning, OCR, and cleaning. The source receives an immutable identifier.
**2. Instruction and data separation.** The system instruction describes the task and field semantics. Customer text is handled as untrusted data. Instructions embedded in an email do not override application policy.
**3. Schema-constrained generation.** The local model server applies JSON Schema or a prebuilt GBNF grammar. Model, chat-template, schema, and server versions are recorded.
**4. Independent validation.** A normal JSON Schema library validates the result again outside the model. This catches unsupported engine functionality, configuration mistakes, and regressions after upgrades. Public `llama.cpp` issue reports have described configurations where a parameter was accepted but the constraint did not take effect; an integration test must detect this failure mode.
**5. Business rules.** `customer_id` is looked up in the CRM, an amount is matched against the document, a date is checked against the open period, and a category is checked against available queues. These deterministic checks do not belong to the model.
**6. Exception queue.** Low confidence, conflicting fields, an unknown customer, or a rule violation creates a human task. The operator sees the source, extracted values, evidence, and the exact reason the pipeline stopped.
**7. Idempotent write.** Only after approval does the system create or update a record using the source event as an idempotency key. Redelivery of an email must not create a second opportunity. External email, payment, or deletion needs a separate explicit confirmation.
This pipeline supports a local model where confidentiality matters without making the model the owner of business catalogues or rules.
Data and infrastructure requirements
A pilot needs one model server, a small orchestration service, a validator, an exception queue, and a test instance of the target system. GPU requirements depend on model choice, context size, and target latency; structured generation does not remove the need for load testing.
Prepare four focused datasets rather than “all company data”:
- 100–300 real, anonymised inputs from the selected process;
- reference structured outputs approved by the process owner;
- catalogues of valid customers, categories, statuses, and units;
- rare and conflicting cases: empty attachments, multiple customers, contradictory amounts, and unknown formats.
Store the schema as a versioned contract next to the integration code. Renaming a field or changing the status catalogue is an interface migration, not an informal prompt edit. During transition, consumers either understand both versions or receive an explicit contract version.
Logs need only contain a source hash or identifier, component versions, structural and business validation results, the human decision, processing time, and cost. Sensitive source text does not need to be duplicated across every log.
Testing before connecting the CRM
Begin in draft-only mode. Measure the following separately for every example:
- the share of responses passing independent schema validation;
- accuracy for each business field, not only an object-level average;
- recall of mandatory facts;
- the `unknown` rate and the human-queue rate;
- critical false positives, such as linking the wrong customer;
- median and 95th-percentile latency;
- cost per accepted result and human review time.
A mandatory negative test should deliberately ask for a field or value forbidden by the schema. If the server returns it with an apparently successful HTTP response, the constraint is not active. Repeat the test after every engine, model, or chat-template update.
For business quality, split examples into tuning and holdout sets. Do not keep editing the prompt and schema while looking at the holdout set; that produces an attractive test result without evidence about new messages. Critical fields still require a rule or human check even when average accuracy is high.
Limits and common mistakes
The first mistake is an overly broad schema. Free strings thousands of characters long and permitted extra properties recreate most of the uncertainty of ordinary text output. Every field should support a concrete next step.
The second is an excessively strict schema without `unknown` or `null`. The model is then forced to choose a formally valid but false value. The ability to stop is part of reliability.
The third is trusting `confidence`. A model's self-score is not calibration. Thresholds must reflect the cost of different errors: a missed lead and an incorrect invoice have different consequences.
The fourth is writing immediately after JSON validation. A correct string does not prove that the customer exists or that the user may change the record.
The fifth is upgrading `llama.cpp` without contract tests. Its documentation states that a subset of schemas is supported and identifies performance considerations for some grammars. Pin a working build and move upgrades through a staging environment.
The sixth is one giant request. Extraction, classification, customer matching, and the action decision should be separated. Deterministic identifier lookup should not be delegated to a generative model.
Modelled pilot economics
Consider a service team handling 4,000 messages per month. Manual classification and transfer of five fields takes two minutes on average, or about 133 hours. If the system automatically accepts 70% of messages and reviewing each exception takes one minute, the modelled capacity released is about 73 hours: 93 hours of automated work minus 20 hours spent reviewing the remaining 30%.
This is a model, not a promise. It assumes a stable flow, no time spent correcting hidden errors, and zero infrastructure cost. A real budget must include the server, integration development, dataset labelling, schema maintenance, and quality control.
The primary metric is neither invocation price nor the percentage of syntactically valid JSON. Measure the cost of a card accepted without material edits and the cost of a critical error separately. If the human queue remains large, a smaller schema or a deterministic rule before the model may be more economical than a larger LLM.
A two-week pilot
During the first two days, select one object such as a support request. Agree on 5–8 fields, valid values, and actions the system is not permitted to perform.
Use days 3–5 for an anonymised sample and JSON Schema. Add `unknown`, forbid extra properties, and set an explicit contract version.
In week two, run the local model in shadow mode. Outputs do not enter the CRM; they are compared with operator work. Run negative constraint tests, then measure field accuracy and exception reasons.
Enable writes only for low-risk records and through an idempotent API. Expand autonomy after a stable holdout result and a review of every critical miss.
Management takeaway
Start with the contract between AI and the system, not with “which model writes better JSON”: a compact schema, independent validation, business rules, an exception queue, and action approval. That layer usually adds more reliability than another round of prompt engineering.
Vnutrik may fit every cube through the correct opening. People and process rules still decide which drawer receives the request and whether the CRM may be changed.