Real-Time or Batch AI: The Business Cost of Unnecessary Urgency

Urgency is a capacity requirement, not a user-interface feature

Many local AI projects become “online” by default. An email arrives and the model is expected to answer immediately. A document is uploaded and its fields should appear within seconds. A knowledge base changes and every embedding is supposed to be recomputed at once. This looks impressive in a demo, but the budget then includes an always-ready server, peak headroom, continuous monitoring and operational coverage.

The management question is different: how much does the business lose if a particular result is ready in ten minutes, an hour or by the start of the next working day? If there is no measurable loss, immediacy is a paid feature without proven value.

Batch processing does not mean slow manual work. vLLM supports offline inference for generation, classification, embeddings and reranking, along with asynchronous queue APIs. NVIDIA Triton can dynamically combine incoming requests and configure batch delay, priorities, queue limits and time-outs. A company can therefore build one controlled service with several urgency classes rather than two unrelated stacks.

Three SLA classes instead of one “real-time” promise

Classify the work before choosing hardware.

1. Interactive

A person is waiting in the current screen and cannot continue. Examples include an operator assistant, knowledge-base search or a short request check before a customer reply. Time to first token, end-to-end p95 and predictable overload behaviour matter here.

2. Nearline

The result is needed within 5–60 minutes, but nobody is watching a spinner. Examples include inbound-email classification, call summaries, CRM-card preparation and new-document checks. The request can wait for a larger batch and yield capacity to the interactive queue.

3. Batch

There is a fixed deadline rather than an immediacy requirement. Examples include nightly catalogue enrichment, RAG reindexing, a daily report, archive classification or embedding recomputation after an approved update. The objective is throughput, repeatability and cost per accepted record.

Urgency belongs to an operation, not to an entire product. In one system, search over an existing index may be interactive, new documents may be processed every 15 minutes and a full reindex may run overnight.

Online batching and an overnight job are not the same

Dynamic batching keeps a service continuously available but delays a request briefly so it can be combined with its neighbours. Triton's documentation recommends measuring latency and throughput with the default configuration first, then increasing batch size or queue delay only while the result remains inside the latency budget. The same scheduler supports priorities and time-outs.

Offline batch inference operates on a known dataset. It can load a model once, split the data into shards, checkpoint progress and write results under stable identifiers. Current vLLM documentation describes offline APIs and a Ray Data integration where continuous batching keeps replicas saturated while sharding, load balancing and fault tolerance support long-running jobs.

These are different economic instruments:

• online batching lowers the cost of an interactive request but does not remove the cost of service readiness;
• offline batch removes continuous-readiness requirements from the non-urgent share of the workload;
• a hybrid keeps a small urgent lane and moves most work into a cheaper processing window.

The Sarathi-Serve paper illustrates the underlying trade-off: larger batches improve serving capacity, while the scheduler must control tail latency. Its gains were measured on specific models and GPUs and cannot be copied into a business estimate. The transferable lesson is to measure throughput and p95 together on the company's own hardware.

A shared-queue architecture

An SME can implement a transparent design without adopting a heavy platform.

1. The intake service creates a job with a stable `job_id`, input reference, SLA class, deadline, model version and access policy.
2. A router places it in an urgent, nearline or batch queue. Priority must be a visible job field rather than an implication hidden in an API method name.
3. One or more workers form batches according to deadline, input length and available memory. Urgent jobs may pass non-urgent jobs, but they must not starve them indefinitely.
4. Results are written idempotently. Re-running the same `job_id` updates status or returns the existing result rather than duplicating a CRM or ERP record.
5. Failures enter a bounded retry queue and then a dead-letter queue with a concise reason. Unlimited retries turn savings into hidden load.
6. Critical fields require human approval. The business metric counts accepted results, not successful model HTTP responses.

A simple system scheduler or Kubernetes CronJob can launch the batch. Kubernetes documentation warns that a CronJob may create multiple concurrent Jobs under some conditions. Use a concurrency policy, a starting deadline and idempotent writes. A schedule is not an exactly-once guarantee.

A modelled calculation: the price of immediacy

The following is neither a market tariff nor a universal benchmark. It is a template with explicit assumptions.

A company processes 3,600 non-urgent text jobs per day: classification, short summarisation and extraction of several fields. The same model achieves the same acceptable quality in both tested modes.

Assumptions:

• the latency-oriented mode produces 12 accepted results per minute;
• the batch mode produces 36 accepted results per minute because the queue is denser;
• the interactive stack requires dedicated 24×7 availability;
• batch runs daily with a 20% time reserve for retries and variable input length;
• fully loaded GPU cost, including depreciation, electricity and operations, is modelled at RUB 300 per GPU-hour.

The computation itself would take roughly five hours per day in the online mode, but the dedicated service reserves 720 GPU-hours per month. Batch completes the daily volume in about 100 minutes; including the reserve, it consumes roughly 60 GPU-hours per month.

Modelled infrastructure cost:

• always-ready dedicated service: `720 × 300 = RUB 216,000/month`;
• daily batch on a shared GPU: `60 × 300 = RUB 18,000/month`;
• immediacy premium: approximately `RUB 198,000/month`.

The difference shrinks or disappears if the online server performs other valuable work, the GPU is already paid for and capacity has no alternative use. It grows if the p95 target requires a second instance, N+1 redundancy or overnight operational coverage. The relevant input is therefore the cost of reserved capacity share, not only the purchase price of the card.

When real time pays for itself

An urgent lane is justified when acceleration changes money or risk. The formula is simple:

`value of urgency = genuinely urgent events × benefit of accelerating one event`.

Latency has value when a fast answer prevents a production stoppage, retains a customer during checkout or lets an operator process more cases. That effect must be validated. “Users prefer faster” is not enough evidence for a dedicated GPU.

Often only 1–5% of events are truly urgent. A better design is then to:

• create a narrow online route with a limit and short context;
• move the remaining jobs to nearline or batch;
• degrade predictably under load by returning an expected completion time rather than silently growing the queue;
• measure the share of results actually used before the batch alternative could have delivered them.

What to include in a pilot

The company can run a two-week pilot without buying new infrastructure.

• Collect an hourly arrival profile and users' actual deadlines.
• Label every operation interactive, nearline or batch and assign a cost of lateness.
• With one dataset and one model, measure accepted results per minute, p50/p95, GPU-hours, retry share and human-review time.
• Test three modes: immediate request, short dynamic delay and a fixed batch window.
• Introduce a queue limit, time-out, idempotency key and bounded retry before scaling.
• Compare cost per accepted result and cost of reserved availability, not just tokens per second.

The management takeaway

Do not buy “real time” for the whole system. Buy it only for operations where delay has a measured cost. Move the remaining work to nearline and batch while keeping a shared audit trail, common model versions and the same quality gates.

The first practical step is to take one week of real traffic and calculate what share of results was needed in less than 15 minutes. If the share is small, the largest economic opportunity is not another quantisation trick but an honest revision of the SLA. Vnutrik may sprint with every folder, but management still receives the bill for the sprinter's readiness.