The short answer

Before selecting a model, a business needs a small gold-standard dataset: real examples with a predefined correct outcome, acceptable exceptions, and the cost of an error. Without it, the company cannot compare an API with a local model fairly, validate a new RAG version, or demonstrate that a pilot saves time.

Annotation is not a minor support line in the budget. The company is paying for guidelines, specialist work, repeated assessment of difficult examples, disagreement resolution, data protection, and dataset maintenance. A budget that counts only the hours spent clicking will look cheap and produce an unreliable test.

A practical pattern for a small or mid-sized business is to have one annotator cover the complete dataset, a second person independently label a risky or random subset, and a domain expert adjudicate disagreements. The overlap rate should depend on the cost of failure and measured human agreement rather than a universal percentage.

What a gold-standard dataset is

It is neither an archive of all available data nor a random CRM export. A gold set is a controlled sample on which the team makes a go or no-go decision.

Depending on the workflow, one item may be:

  • a customer request with the correct category and routing decision;
  • a RAG question with required evidence and an acceptable refusal;
  • an invoice with reference fields and an arithmetic check;
  • a call with a verified transcript and outcome;
  • an image with a defect class and region;
  • an agent action with the expected approval or prohibition.

Each item needs more than an answer. It also needs context: policy version, user permissions, language, source, decision rationale, and the business cost of an error. Otherwise the test measures string similarity rather than process fitness.

The NIST Measure function recommends documenting test sets, metrics, and tools and evaluating systems under conditions similar to deployment. In operational terms, the sample must represent the real workflow, including rare expensive failures rather than only convenient examples.

What the budget contains

The full cost has at least seven components.

1. **Guideline design.** The process owner and analyst define classes, acceptable decisions, examples, and escalation rules.
2. **Data preparation.** The team exports records, removes duplicates, masks personal data, reconnects attachments, and records versions.
3. **Primary annotation.** One worker applies the guidelines to the entire set.
4. **Overlap.** A second person independently annotates a subset. If the second person sees the first answer, that is review rather than independent evaluation.
5. **Adjudication.** An expert resolves difficult cases and updates the guidelines when needed.
6. **Dataset control.** The team checks class coverage, leakage between training and test data, departmental distribution, and document freshness.
7. **Operations.** The dataset is versioned, expanded with newly observed failures, and rerun against each material model or process release.

The annotation interface also changes cost. Keyboard shortcuts, prefilled fields, and relevant context can accelerate work. Machine pre-annotation, however, can anchor the human reviewer on an incorrect suggestion. The control subset should preserve independent judgement or measure the effect of suggestions separately.

Why annotator agreement matters

When only one specialist assigns a label, the team does not know whether the rule is reproducible. A second specialist is useful not because two votes are inherently correct but because disagreement reveals ambiguity.

Label Studio documentation describes overlap—assigning the same task to multiple annotators—and inter-annotator agreement metrics. Low agreement can indicate three different problems:

  • the guideline permits several interpretations;
  • workers do not receive enough context;
  • the business process itself has no single correct decision.

The third problem cannot be solved by changing the model. The process owner must first define a rule or explicitly allow multiple valid outcomes.

The metric must fit the task. Simple classification can use exact agreement and a confusion matrix. Image boxes may use IoU, transcription may use edit distance, and entity extraction may use precision, recall, and F1. A single aggregate agreement percentage can hide material differences between error types.

Complete double annotation of the entire dataset is rarely necessary at the start. It is more efficient to duplicate:

  • a random subset for an estimate of overall stability;
  • all rare critical classes;
  • new categories introduced after a guideline change;
  • cases where the model and a human disagree;
  • examples associated with previous incidents.

If agreement declines, the overlap rate can be increased temporarily. If the guidelines are stable and the risk is low, the rate can be reduced while retaining a control sample.

A model calculation for 1,000 examples

The following is not a market quote. It is a worked example with explicit assumptions. Suppose a company prepares 1,000 documents to evaluate field extraction.

Assumptions:

  • primary annotation takes two minutes per document on average;
  • a second worker independently annotates 20% of documents, also at two minutes each;
  • 10% of documents require four minutes of expert adjudication;
  • fully loaded annotator cost is RUB 900 per hour;
  • fully loaded expert cost is RUB 1,800 per hour;
  • guideline design, sample preparation, and control require 16 expert hours;
  • software, infrastructure, and integration are calculated separately.

The result is:

  • primary annotation: 1,000 × 2 minutes = 33.3 hours, or about RUB 30,000;
  • overlap: 200 × 2 minutes = 6.7 hours, or about RUB 6,000;
  • adjudication: 100 × 4 minutes = 6.7 expert hours, or about RUB 12,000;
  • guidelines and control: 16 expert hours, or RUB 28,800.

The total is approximately RUB 76,800 before tooling, export, anonymisation, and project-management costs.

If the estimate includes only the first line, the budget is understated by more than half. If the company double-labels every item, quality may not rise in proportion to the additional RUB 30,000. Overlap is therefore a managed parameter that changes after agreement is measured.

A reusable spreadsheet formula is:

**Budget = N × t₁ × C₁ + N × r × t₂ × C₂ + N × q × tₐ × Cₐ + preparation + tools**, with time converted to hours, `r` representing independent overlap, and `q` representing the adjudication rate.

The estimate also needs contingency. A revised guideline usually changes some existing labels, and part of the source data will prove unusable because of a damaged file, wrong document version, missing context, or irreducible ambiguity.

Avoiding false savings

The lowest unit price does not guarantee the cheapest accepted dataset. Annotation errors surface later: a model scores well against incorrect labels and then creates manual rework in production.

Vendors and internal teams should be compared using cost per **accepted** example after quality control. A pilot batch should measure:

  • task time by complexity category;
  • skipped and returned items;
  • agreement on overlap;
  • adjudication rate;
  • guideline revisions;
  • cost per accepted item;
  • speed of correcting a systematic mistake.

Skill levels should be separated. A domain expert does not need to label every simple example. The expert designs the rules, creates examples, adjudicates disagreements, and reviews the critical class. A trained annotator or process employee with a lower fully loaded hourly cost handles the routine portion.

A local annotation tool is appropriate when examples contain contracts, customer correspondence, production images, or other sensitive information. Self-hosting is not free, however: it requires accounts, backups, export auditing, updates, and project isolation. In some cases, an anonymised sample in a managed service is cheaper than a private deployment. Data requirements, rather than enthusiasm for local infrastructure, should decide.

A four-week build plan

In week one, select one process, define the cost of key errors, and draft the guideline using 20–30 real examples. In week two, give an initial set of 100 items to two people independently and review their disagreements. In week three, revise the rules, annotate the main sample, and route difficult cases to the expert. In week four, freeze the version, inspect its distribution, and compare models for the first time.

Acceptance criteria should be set before annotation begins:

  • minimum agreement for each critical class;
  • maximum unresolved-item rate;
  • no entity or document leakage between training and test splits;
  • a defined proportion of fresh examples from the current process version;
  • a list of errors that prevent the model from taking an automated action.

A gold set must not become a museum. Confirmed production failures, new formats, and changed rules are added after launch, but the historical version is not silently rewritten. This lets the team distinguish a better model from an easier exam.

The management takeaway

Buying a model before creating a gold-standard dataset reverses the investment sequence: the company pays for technology and only later discovers it cannot measure the outcome. It is safer to allocate a separate budget for guideline design, independent overlap, and expert adjudication.

For an initial decision, 500–1,000 carefully selected examples can be sufficient when they cover real categories and costly errors. This is a working range for a pilot, not a universal statistical rule; the required volume depends on process diversity and the confidence needed.

The main saving does not come from making annotation as cheap as possible. It comes from reusing one versioned dataset to select a model, validate upgrades, accept vendor work, and investigate failures.