How can you run a local language model securely?

Define the boundary before choosing a runtime

"Local" answers only one question: where inference executes. It does not answer which account runs the process, what data it can read, where it can connect, who can reach its API, whether it downloads updates, or whether the model can invoke tools. A useful design begins with a short threat statement, not with a model name or a container command.

For example, write down: "One developer uses a model on a workstation to summarize text they paste into a desktop client. The service must not accept network clients, read the home directory, retain chats, or use external tools." That is a much smaller system than: "A team shares a web interface that can search company documents and create tickets." The first can plausibly start with a local-only, chat-only runtime. The second needs user authentication, authorization, tenant and document access controls, monitoring, and a review of data handling.

The table is a decision aid, not a guarantee. A chat-only process still parses untrusted text and model artifacts, and a locally reachable service can still be attacked by other software or users on the same machine. A tool-equipped assistant is different in kind because model output can influence action. OWASP identifies prompt injection, tool abuse, sensitive-data exposure, excessive autonomy, and supply-chain issues as agent risks. OWASP AI Agent Security Cheat Sheet

Identify assets and attackers

List the assets first: the model and runtime binaries, source documents, local chat history, API tokens, SSH keys, browser profiles, business systems, GPU resources, and the host itself. Then identify realistic attackers and failure modes: a tampered model download, a vulnerable runtime, a malicious document, another local user, a device on the same network, an accidentally public port, or a model-induced but unauthorized tool call.

Security controls should reduce a specific risk. An isolated model directory helps contain a compromised downloader or runtime. Binding an API to loopback prevents ordinary remote access. It does not protect a host already compromised by malware. A reverse proxy with TLS protects a network hop, but it cannot make a model safe to give arbitrary shell access. This distinction prevents a common mistake: treating a container, a private network, or local inference as a complete security solution.

Example of a small but defensible workstation setup

This hypothetical setup is for a developer who wants private drafting help on a laptop. They create a dedicated non-administrator account for the runtime, place an approved model in a directory that the account can read but not modify, give it an empty working directory, and configure the client to connect only through a loopback socket. The process has neither browser cookies nor cloud credentials, and it is not allowed to open the developer's home directory.

The developer disables network egress for the process after the approved downloads complete, retains only a version manifest and security events, and removes conversational content when the client closes. The result is not invulnerable, but a malicious or flawed model runtime has far fewer useful things to read, send, or alter than it would inside the developer's normal account.

Acquire model artifacts and runtimes as a supply chain

Loading a model introduces software supply-chain risks. A model format may be primarily numeric data, but loading logic, conversion scripts, custom code, container images, tokenizer files, GPU libraries, and helper tools expand the attack surface. Do not install a model because its filename is familiar or a social-media post links to it.

Use a known publisher or an organization-controlled registry. Record the publisher, exact release or commit, model identifier, format, cryptographic digest, license, download date, runtime version, and any conversion steps. Verify a publisher-provided signature or checksum before the file reaches the directory served to the runtime. A checksum copied from the same compromised page is weaker evidence than a signed release, a trusted registry, or a checksum obtained through an independent channel.

Where the loader supports it, choose a loading path designed to reduce unsafe deserialization and keep its restrictions enabled. PyTorch explicitly warns never to load data from an untrusted source. Its weights_only mode narrows the remote-code-execution surface, but PyTorch also documents that it does not prevent denial of service and cannot eliminate all downstream risk. PyTorch serialization security documentation Treat that as a reason to verify provenance before loading, not as permission to trust an arbitrary checkpoint.

If you use a container, pin the image to a reviewed digest rather than a mutable tag, inspect its declared entrypoint and exposed ports, and track its base-image updates. Containers are packaging and isolation mechanisms, not a substitute for host hardening. NIST's container guidance calls out risks across image, registry, orchestrator, and host layers, including access control and configuration management. NIST SP 800-190, Application Container Security Guide NIST's Secure Software Development Framework also includes collecting and sharing provenance data for release components. NIST Secure Software Development Framework

Restrict the runtime rather than trusting it

Run inference under a dedicated, non-privileged operating-system identity. Give that identity a read-only model directory and a small, dedicated writable directory for caches or temporary files. Do not mount the user's home directory, credential stores, source-code checkout, browser profile, SSH directory, cloud configuration, or a container engine socket into the runtime. Avoid privileged containers, host networking, and broad device mounts unless a documented need outweighs the added access.

Use the isolation features that fit the operating system and deployment: a separate local account, filesystem permissions, mandatory-access-control policy where available, a non-root container user, a read-only root filesystem where practical, CPU and memory limits, and a request-size limit. The point is not to accumulate controls for their own sake. It is to ensure that a compromised component cannot silently become the user's general-purpose program.

GPU access deserves the same care. It may be required for performance, but it is still access to a hardware and driver interface. Grant only the device access the runtime needs, keep driver and runtime versions in the inventory, and do not assume that a GPU workload is isolated merely because it is in a container.

Make network exposure deliberate

For a service used on one computer, bind the listener explicitly to loopback, such as 127.0.0.1 for IPv4 and ::1 for IPv6. Do not assume a framework's default bind address is private. In Docker, for example, a generic published-port mapping can expose a container on the host's external interfaces. Docker documents that publishing to a localhost address limits access to the host, while a published port can otherwise be available outside it. Docker port publishing and mapping

Do not expose an unauthenticated model server directly to the internet. If colleagues need access, put an authenticated and encrypted gateway or a private, managed access path in front of it. Require a named identity, authorize the requested model and data scope, rate-limit requests, set maximum input and output sizes, and retain a minimal audit event. A network firewall remains useful, but it should complement rather than replace application authentication.

Decide separately whether the inference process may make outbound connections. A chat-only local system normally needs no egress after installation. If it needs a package mirror, a license check, telemetry endpoint, retrieval source, or update service, make each destination explicit, restrict it in a network policy or firewall where possible, and document what data leaves. Automatic downloads and outbound telemetry can defeat the privacy expectation implied by the word "local."

Remote access to a single user's model is usually safer through an existing authenticated private-network or remote-access service than by forwarding a router port. This is not a promise that any particular VPN is secure. It is a practical choice to avoid inventing public authentication, TLS, revocation, and monitoring for a hobby model endpoint. For a regulated, public, or multi-tenant service, obtain a security review and apply the organization's standard identity, network, backup, and incident-response controls.

Treat tools as privileged services

A model that only returns text can be wrong or manipulated, but it has limited direct impact if nothing interprets its output as an instruction to the operating system. Adding tools changes that. A file reader can disclose data, a web fetcher can reach internal services, a database tool can change records, and a shell tool can be equivalent to giving a probabilistic component a user account.

Put a small broker between the model and every tool. The model should request a structured action, and the broker should validate the schema, identify the human or service principal, check authorization, narrow the resource scope, enforce rate and spend limits, and log the decision. The broker should reject an action even if the model asks confidently. Do not let the model construct arbitrary shell commands, choose arbitrary URLs, or use credentials that are more powerful than the individual operation requires.

Retrieved webpages, attachments, emails, and documents are data, not trusted instructions. They can contain text intended to redirect the model or induce a tool call. Keep tool responses distinct from authority, avoid persisting them as unrestricted memory, and test with hostile inputs before adding a new capability. OWASP recommends per-tool least privilege, separate tool sets for trust levels, explicit authorization for sensitive operations, and testing that high-impact actions cannot bypass approval. OWASP AI Agent Security Cheat Sheet

Credentials should be per tool and per environment, short-lived where the platform supports it, and stored in a secret-management mechanism rather than in prompts, model directories, source files, or container images. A tool that needs to read one project should receive a credential constrained to that project, not an all-company administrator token. Revoke the credential when the tool is disabled or an incident occurs.

Update, observe, and recover without collecting everything

Maintain an inventory of the operating system, inference runtime, model digest, tokenizer, prompts or policy version, enabled tools, container image digest, and network configuration. Subscribe to security advisories for the runtime and its major dependencies. Test updates in a non-production profile before broad use, keep a known-good previous configuration, and record who approved exceptions such as temporary egress or a broad file mount.

Logs should support investigation without becoming an archive of sensitive conversations. Record structured events such as service start and stop, model and runtime version, authentication outcome, policy denial, tool name and result class, rate-limit action, configuration change, and error category. Avoid raw prompts, completions, file contents, access tokens, passwords, session IDs, and full document paths by default. OWASP advises that logs can contain personal and sensitive information and lists tokens, passwords, keys, sensitive personal data, and connection strings among data that should normally be removed, masked, sanitized, hashed, or encrypted. OWASP Logging Cheat Sheet

If prompt or output samples are genuinely needed for quality evaluation, collect the minimum sample set under an explicit policy. Separate it from operational logs, define retention and deletion rules, restrict access, redact where feasible, and make backups subject to the same rules. For customer or employee data, apply the organization’s access and retention rules to local copies and backups too.

Prepare a stop path before the first use. It should be possible to stop the service, remove its network route, disable or revoke tool credentials, block the model digest from deployment, preserve the minimum needed evidence, and notify the responsible owner. Test this path. A kill switch that exists only in a wiki is less useful than one an operator can exercise in a few minutes.

Build the baseline in a deliberate order

1. Define the intended use. Name the users, data classes, host, whether remote access is allowed, which tools are allowed, and what event should trigger shutdown. If these answers are unknown, do not expose a listener or add tools yet.

2. Install a reviewed runtime and model. Obtain them from the selected trusted source, verify the expected digest or signature, and save the provenance manifest. Reject a model that requires unexplained custom code or a loader setting that weakens deserialization protections.

3. Create a narrow execution identity. Use a non-administrator account, a read-only model directory, and an empty writable work directory. Start with no credentials, no host mounts, no source repositories, and no access to personal directories.

4. Limit network access. For one device, bind to loopback and deny outbound access after setup. For a shared service, deploy approved authentication, TLS termination, authorization, rate limits, and an explicit firewall rule before onboarding users.

5. Run an abuse check. Try oversize inputs, malformed requests, a request from a second device, a prompt asking the model to reveal configuration, and a document containing instructions to access unrelated data. Confirm that the service rejects or contains each case and that the log records the event without retaining the sensitive content.

6. Add one tool at a time. Begin with read-only, allowlisted access. Test that an untrusted prompt cannot make it access another user's data or take a write action. Only then consider a new tool, and keep approval gates for consequential actions.

7. Operate it as software. Patch it, review its inventory and exposed ports, exercise the stop path, and remove unused models, tool configurations, accounts, and credentials. A forgotten local server is often less safe than an actively maintained one.