Local does not automatically mean private

“The model runs on our premises, so access is already secure” is a convenient but false shortcut. Local deployment answers where computation happens. It does not decide who can send a request, which methods are available, how much GPU time a client may consume, or where prompt data travels. Our eager little robot intern would happily run a cable to every desk; a cable cannot check permissions.

According to Ollama's official documentation, its local API does not require authentication. By default, the server binds to 127.0.0.1:11434, so only the local machine can reach it. OLLAMA_HOST changes the bind address. Listening on 0.0.0.0 to serve a local network is technically easy, but access control then becomes a separate obligation. Ollama also documents reverse-proxy use; its example demonstrates routing, not a complete security policy.

vLLM differs in detail but leads to the same conclusion. Its current Security documentation expressly warns that --api-key protects only specified API prefixes, including /v1, /v2, /inference, and /cohere. Other routes on the same HTTP server may remain outside that check. The documentation names /invocations as one example that can also perform inference. This does not mean every vLLM deployment is exposed: version, configuration, and actual network reachability matter. It does mean a single API key must not be assumed to protect every function.

What can go wrong

First, an unauthorized or simply unaccounted-for client occupies capacity. Even when the model server contains no company documents, generation consumes compute and delays legitimate work. OWASP identifies missing limits as an API resource-consumption risk. For a small team sharing one GPU, the impact may appear as a staff queue as much as a cost line.

Second, privileges get mixed. A service intended only to answer application requests may expose additional technical routes for model inspection, control, or integrations. Their set can change by version. Protecting only today's familiar chat path is insufficient if another route or extension bypasses the check. The safer pattern is to explicitly allow the needed routes rather than blacklist a few dangerous ones.

Third, teams may mistake compute isolation for data isolation. A model API by itself is not a knowledge base: access to private documents usually comes through the surrounding application, RAG layer, or tools. But if browsers and arbitrary internal services can contact the model or application directly, the company loses one consistent point for user checks, document rights, and consumption accounting. CORS constrains browser behavior; it is not a replacement for server-side authentication and authorization.

A minimum architecture for a small business

A single model does not justify an entirely new, heavy service by default. If the environment already has a suitable gateway or reverse proxy, use it as the only allowed entry point. Keep the model server on loopback or in an isolated internal segment, and use the firewall to prevent bypassing the gateway. When the application and model run on different hosts, secure the transport with TLS or another protected internal channel, and limit which addresses may connect.

The gateway needs four independent decisions:

  • who calls it: a distinct application or user identity, not one shared key copied into every client's source code;
  • what is allowed: only the methods and routes needed for the business workflow, with default denial for the rest;
  • how much is allowed: request size, maximum output, concurrency, queue length, timeout, and per-client quotas;
  • what is logged: time, caller, model, duration, volume, and outcome without automatically retaining every prompt body.

For example, an internal support assistant may need only a generation request. It need not load new models, change configuration, or use an operations dashboard. If it later needs another operation, add that path separately with an access test. vLLM's own documentation recommends minimizing exposed routes, deploying behind a reverse proxy, and constraining network exposure. These are verifiable settings, not a request for the model to “behave.”

The gateway does not settle data authorization inside RAG. Document permissions must be enforced before retrieval and checked again before an answer is delivered. The boundary discussed here comes earlier: who may reach the compute API at all. Do not confuse the two layers.

Economics without fictional ROI

There is no honest universal price for an “unprotected port.” The impact depends on GPU hardware, context length, concurrency, and the cost of delays. A model calculation can still be useful. Suppose an unwanted request occupies one minute of compute. Thirty such calls could use up to 30 minutes of aggregate work; parallel execution changes wall-clock time and memory demand. This is not an observed company result, only a way to see why limits matter even on a private network.

When evaluating controls, count more than the gateway's price. Include administrator time for identities and rules, proxy latency, logging, upgrades, and bypass tests. Also count the downside of overly strict limits: employees may wait or find a workaround. Measure ordinary load and peaks first, then set quotas with headroom. An engine's built-in queue helps with overload, but it cannot distinguish an authorized client from an accidental neighbor on the network.

A one-day check

Inventory every model address and port, container publication, proxy, and tunnel. Confirm staff applications use the gateway and cannot reach the model directly from their network segment. Then test without secrets or a full generation: an unauthenticated request to every published route should be rejected, while an authenticated client should see only its allowed operations. Test alternate paths and version upgrades separately, because a new route can appear without much ceremony.

Next, apply a modest per-client limit and run a controlled load test outside production hours. Record normal-request latency, behavior when the queue fills, and the fields written to logs. If the check finds a gateway bypass, close that network path before expanding model usage. The eager robot can remain the hardest-working intern on the team; a human still decides where the cable goes.