When a small task reserves a large server

In customer support, AI may do several quite different jobs: prioritise new tickets, suggest the next phrase to an agent, search a knowledge base and draft a customer response. Keeping a separate powerful machine running around the clock for each action raises costs even when there are no requests. The server waits patiently; the bill for waiting does not.

Support AI provider Forethought described in an AWS case study how it separated these workloads. It placed multiple models on shared inference endpoints and moved some small classifiers to on-demand execution. According to the published AWS interview with the company, the first change reduced the relevant costs by up to 66%, while the second saved about 80% on the classifiers that were moved. These are results from one system compared with its earlier deployment, not a promise of equivalent savings for another business.

According to the source, three engineers handled the company's machine-learning infrastructure and its products supported more than 30 million customer interactions a year. What matters beyond that scale is the management method: break “AI in support” into tasks with different response-time requirements, then choose a suitable execution route for each one.

What happened in the case

Forethought previously served its models through Kubernetes in the cloud. As it grew, it had more models, including models tuned to particular customer scenarios. According to AWS, it moved some inference to SageMaker multi-model endpoints, where several models share infrastructure instead of each occupying a separate always-on endpoint. The vendor gives text autocomplete while a user is typing as one example. Forethought reports cost reductions of up to 66% for that area and better latency; the calculation method and original invoices are not public.

It moved another group, small classifiers such as a ticket-priority model, to SageMaker Serverless Inference. AWS reports a saving of about 80% on the associated cloud costs. These figures must not be added together: they cover different workloads and different previous hosting arrangements. Nor should they be projected onto the cost of a large language-model answer. AWS documentation explicitly states that Serverless Inference does not support GPUs and may suffer cold-start delays after idle periods.

That is the boundary of this case. A shared model pool can help under recurring demand when memory and compute can actually be shared. On-demand execution suits some small, irregular jobs when the business can tolerate startup time. An interactive customer answer with a strict latency target may need permanently ready capacity. The choice requires measurements of the company's own traffic, not a product label.

What transfers to an on-premises setup

Local AI need not reproduce AWS services. The principle applies to an owned server: measure the different request paths first, then separate inexpensive deterministic rules and small models from heavy generation. Ordinary code can sometimes route a simple ticket. A classifier, speech recognition, embeddings and a large language model differ in memory use, runtime and acceptable queueing delay. Sending all of them to the same expensive accelerator without measurement is a poor default.

A local pilot can follow this sequence:

  • Give each incoming request an identifier, task type and response deadline.
  • Let rules or a small classifier choose a route when their quality has been verified on historical tickets.
  • Retrieve knowledge-base documents under the employee's permissions; use generation only where a coherent answer or ambiguous text analysis is needed.
  • Send heavy batch jobs to a separate queue. Give urgent requests a waiting limit and a route to a human agent.
  • Record latency, error rate and cost per accepted result for each stage without filling logs with unnecessary customer-message content.

Existing inference servers such as NVIDIA Triton document multi-model serving and different schedulers and batching options. That is a technical capability, not automatic savings: models must fit in memory, loading one must not displace frequently used weights, and batching must not break the response-time target. For language models, also account for input and output lengths, KV-cache size and competition for memory.

Independent research on Mélange demonstrates the broader principle for language-model serving: the economical accelerator depends on request length, arrival rate and the latency constraint. Its authors evaluated specific experimental settings, so their savings figures are not a purchasing forecast for a particular company. The narrower useful lesson is that peak tokens per second in a promotional benchmark cannot select a business server on its own.

Data to gather before buying hardware

Collect several weeks of real request statistics without sending sensitive message contents into an unapproved test environment. For each route, record hourly request counts, peaks, text length, required response time, repeated-question share and handoffs to agents. Separate overnight and weekend traffic: a monthly average hides both spikes and idle hours.

Measure three options on the same sample: a simple rule or small CPU model; a shared local inference server; and an external API where data policy permits it. This is an architecture comparison, not a recommendation to export customer conversations. For each route, assess decision quality, p95 latency, errors and human corrections. A cheap classifier is not useful if it mistakenly sends urgent requests to a slow queue.

Define the useful unit in advance. For classification, it might be a ticket whose priority is correct after sampled review. For generation, it could be an answer an agent accepts without a material edit. “Cost per million tokens” excludes retries, validation, server idle time and the people correcting outputs.

Economics: an illustrative calculation

Suppose a dedicated local system for one function costs a hypothetical RUB 30,000 per month including depreciation, electricity, administration and a failure reserve. This is a model assumption, not equipment pricing or a Forethought result. With 3,000 verified useful operations, the fixed component is RUB 10 per operation. If the same system handles 6,000 such operations without worse quality or latency, it falls to RUB 5. If the workload vanishes for a month, the fixed cost remains.

A shared server helps only when combined workloads complement one another and memory and response-time constraints still hold. If every department sends requests at the same hours, sharing may simply create a longer queue. On-demand execution removes some payment for empty waiting but may add a cold start and cannot host every model. An external API changes the cost structure toward variable spending, while requiring separate assessment of data, contract terms, access and network charges.

Compare the full monthly cost: depreciation or rental, electricity, support, storage, failover, data transfer, retries and reviewer time. Divide it by results the business actually accepts, not all model calls. State a demand range and the point at which another option becomes cheaper. Recalculate that point when traffic or tariffs change.

A safe first step

Do not begin by buying an accelerator “for all AI”. Split one support workflow into tasks and measure frequency, peaks, p95 latency and accepted-result share for two weeks. Then test simple rules and a small model for routing, enabling generation only where it helps an agent. If a giant server has been standing by to produce a single short “urgent” label, Vnutrik can find a more suitable trolley.

Sources: the published AWS Forethought case for the company's reported outcomes; AWS documentation on Serverless Inference constraints; NVIDIA Triton documentation on multi-model serving; and the Mélange research paper on how inference cost depends on workload characteristics. The architecture recommendations and ruble calculation are editorial analysis and an illustrative model.