What “local” actually covers
Local execution means model computation takes place on your hardware. By itself, it does not prove that the entire service operates without external connections. A production setup usually also has a weight downloader, API server, user interface, search index, CRM and document connectors, observability tools and update mechanisms. For each component, answer separately: what data does it receive, where can it connect, and what happens when the network is unavailable?
This is not a claim that every local product sends conversations outside the organisation. Different features transmit different kinds of data. Model usage statistics, for example, are not the same as prompt text; cloud inference, by contrast, processes prompt content remotely. Procurement and architecture decisions therefore need traffic categories rather than a single “local” tick box.
Four paths worth checking
**Model downloads.** Hugging Face Hub documentation says a normal `hf_hub_download` call may contact the server even when a file is cached, to check whether the version is current. `HF_HUB_OFFLINE=1` prevents HTTP calls to the Hub: available local files are used, while a missing file causes an error. This is a library setting, not a network prohibition for the whole application. For a reproducible startup, stage weights, tokenizer, configuration and dependent files in advance; pin the revision and test that the service starts without Hub access.
**Cloud features alongside local ones.** Ollama documents cloud-hosted models separately. For local-only use, its FAQ recommends `OLLAMA_NO_CLOUD=1` or `disable_ollama_cloud` in configuration, followed by a restart. Ollama cloud models and web search are then unavailable. That setting does not block arbitrary outbound connections from other programs, such as a separate connector or your application.
**Technical usage data.** In the vLLM 0.31.0 source code, usage-statistics collection is enabled by default and can be disabled with, among other options, `VLLM_NO_USAGE_STATS=1` or `VLLM_DO_NOT_TRACK=1`. This is a reason to agree a telemetry policy and inspect network logs, not a reason to claim prompt leakage. Check the fields sent by the particular version in use. Other components may have different defaults.
**Agents, RAG and interfaces.** Even if inference is entirely local, the application may send a question to cloud search, an external OCR service, an error analytics platform or monitoring provider. Trace the real route from the user interface and file processor to the model, vector search and agent tools. In the architecture record, name the owner of each call and the allowed data type. This is a system-design requirement, not a property of one model.
Build a boundary that can be tested
For a small organisation, an explicit boundary diagram is usually a better starting point than an immediate full air gap. Separate an update-preparation zone from a production zone. Approved model and package versions enter the former under a controlled procedure; business documents are processed in the latter. Transfer only approved artefacts with a recorded version and checksum. If the business needs an external API, route precisely that call through a managed gateway rather than granting the entire server general internet access.
Network rules should enforce policy at the host and gateway. Library options help, but they do not replace outbound controls: a misconfiguration or a new plug-in can violate the team's assumptions. The UK's NCSC recommends host firewalls that restrict unnecessary inbound and outbound connections. Blocking the internet, however, does not replace user authentication, document-level permissions, backup protection or careful log handling.
Write down two distinct policies:
- **Strict offline:** no outbound internet connections from the production zone. Updates enter through approved import. Loss of external connectivity must not stop agreed local tasks.
- **Controlled hybrid:** approved destinations, purposes and data classes are listed; traffic passes through a proxy or gateway, is logged and can be revoked. Users know when a request leaves the boundary.
Do not call a hybrid service “fully offline.” Equally, do not impose a blanket ban when a process owner genuinely needs an external register: support cost can rise without a corresponding benefit if the route can instead be restricted and approved.
A short acceptance pilot
Take ten representative tasks using de-identified data: a text question, a knowledge-base query, document upload, an agent using an approved tool and an abnormal restart. First record DNS queries and outbound connections; then block external addresses and repeat the scenarios. Look beyond the final answer to retries, startup delays, file-loading errors, logs and background jobs. Every unexpected destination needs an owner and explanation.
Set acceptance criteria before the test. For strict offline operation: zero outbound connections from the production zone and successful completion of the agreed local tasks. For a hybrid setup: only approved destinations and evidence that sensitive content is absent from inappropriate channels. One run is not enough; also test updates, restarts and an external-service outage.
Economics and the next step
A protected local environment has costs: compute capacity, storage for multiple weight versions, update import, monitoring, support and recovery. Comparing it with an API only on price per token is misleading. First estimate sensitive request volume and permissible routes, then total cost of ownership and downtime exposure. Controlled hybrid may be sufficient for many teams; contractual or regulatory constraints may demand strict isolation. The data owner, IT and security functions should make that decision together.
The practical first step is a one-page inventory of components, destinations, transmitted data classes and owners. Then run a network test on de-identified workflows. If that inventory cannot be completed, it is too early to claim that data never leaves the system.