Can llama.cpp run a small model entirely in GPU VRAM, and what system memory is still required?

What the source question adds

The observed Reddit discussion describes a Linux machine with a headless RTX 4070 Super, 12 GB VRAM, and 32 GB system RAM. The author tried a small GGUF model with all requested GPU layers, KV-cache quantization, --cache-ram 0, and --no-mmap, yet still saw roughly 1.3 GiB in the server's host-memory breakdown. That is a useful real-world symptom, but community replies are not used here as technical authority.

The important correction is that the question has two answers:

1. Can the model's inference-critical tensors be resident on the GPU? Often yes, if the memory budget fits and the chosen backend supports it.
2. Can the whole program avoid system RAM? No. A GPU backend is controlled by a host process, and OS and driver allocations remain.

Where llama.cpp memory goes

On a discrete GPU system, track several pools rather than one “model size” number.

The official help makes several distinctions that matter here. --n-gpu-layers accepts an exact number, auto, or all; --kv-offload is enabled by default; --no-kv-offload moves in the opposite direction; and the default KV types are F16. The same reference documents --cache-ram, --parallel, model load modes, device selection, and --fit. GPU, KV, and load options

Memory mapping is not a GPU-offload switch

With the default auto load mode, llama.cpp normally memory-maps the model unless a device does not support it. A file mapping creates virtual address space backed by the model file, not an instruction to keep the model weights on the CPU. Linux defines mmap() as mapping a file or device into the calling process's virtual address space. Linux mmap(2) reference

When layers are offloaded, the GPU still needs its own device-resident copy of those tensors. The mapped file can nevertheless appear in process accounting or the operating system file cache. That does not, by itself, prove that each generated token is fetching model weights across PCIe. This is an inference from the documented loading model and normal virtual-memory behavior, so use throughput and llama.cpp's own memory breakdown to diagnose a real bottleneck.

Conversely, --no-mmap or --load-mode none does not mean “load directly into VRAM.” It disables the memory-mapped load path. The current help says it can have slower load time and may reduce pageouts only in some circumstances. It can make host-RAM use larger, not smaller. Current load-mode behavior

A transparent memory budget

The GGUF file's real byte size is the right starting point for weights. A parameter-count shortcut is only an estimate because quant formats include scales and some tensors are not stored at the nominal bit width.

Hypothetical 9B Q4 weight estimate

For a dense 9-billion-parameter model at a nominal 4 bits per parameter:

9,000,000,000 parameters × 4 bits / 8
= 4,500,000,000 bytes
≈ 4.19 GiB

That is not a prediction of a particular GGUF's file size or its VRAM allocation. Use the actual file size and llama.cpp startup output. In a 12 GB card, the remaining budget must still cover the KV cache, compute buffers, the CUDA context and allocator overhead, and room for desktop or display use. Filling the last free megabyte is a recipe for an out-of-memory failure, not a performance target.

Hypothetical KV-cache calculation

For a conventional transformer with grouped-query attention, an approximate one-sequence KV-cache size is:

layers × context tokens × KV heads × head dimension
× (bytes per K element + bytes per V element)

Assume, only for illustration, 32 layers, 8 KV heads, 128 dimensions per head, 8,192 tokens, and F16 K plus F16 V:

32 × 8,192 × 8 × 128 × (2 + 2) bytes
= 1,073,741,824 bytes
= 1.00 GiB

If both K and V were approximately one byte per value, the comparable arithmetic gives about 512 MiB. If K is about one byte and V about half a byte, it gives about 384 MiB. Actual GGML quantized cache types use blocks with metadata, so regard those latter figures as directionally useful rather than exact. The model's attention layout, sliding-window behavior, recurrent state, and concurrent sequences can also change the real total.

Two consequences are reliable:

• Doubling --ctx-size approximately doubles an ordinary KV cache.
• More active server sequences raise the KV requirement. The llama.cpp multi-GPU guide specifically advises reducing context size first and then --parallel when memory is tight. KV-cache and slot guidance

There is no equally portable closed formula for compute workspace. Batch shape, backend kernels, Flash Attention support, and the version of llama.cpp all matter. Read the startup lines that report model, context, and compute buffer sizes, then test the actual workload.

A sensible minimal-RAM llama-server baseline

First record the exact binary with llama-server --version, then inspect its own --help. llama.cpp changes quickly, so copied flags from old guides are a poor source of truth.

For a single discrete GPU and a text-only model that is expected to fit, this is a good diagnostic baseline:

llama-server \
  --model /models/example.gguf \
  --n-gpu-layers all \
  --ctx-size 8192 \
  --parallel 1 \
  --cache-ram 0 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

This is not a universal fastest configuration. It answers a narrower question: where does memory go when one GPU serves one active sequence and host prompt caching is disabled?

The wording and defaults in this table come from the current official llama-server reference. Options and defaults

Do not add --no-mmap, --no-kv-offload, or CPU-MoE options to this baseline. Each can increase host work or host memory for this use case. Also leave --op-offload at its documented default of enabled; it can offload eligible host tensor operations, but it does not turn the HTTP server, tokenizer, and driver into GPU code.

Why host RAM may remain even after full offload

After disabling --cache-ram, a residual few hundred MiB to low GiB can still be legitimate. Its exact size is backend- and build-specific, but common causes include:

1. Process and model metadata: GGUF headers, tensor descriptors, tokenizer structures, sampling state, request buffers, threads, logs, and allocator bookkeeping live on the host.
2. GPU runtime work: the host creates GPU allocations, submits kernels, tracks streams and events, and owns driver-facing objects. GPU execution is not host-process-free.
3. Staging or pinned memory: some transfer and multi-GPU paths use host-visible or pinned buffers. The current CUDA source, for example, contains explicitly named pinned staging buffers for its multi-GPU all-reduce path. That does not establish a fixed single-GPU allocation, but it shows why “all tensors on GPU” and “no host buffer exists” are separate claims. CUDA all-reduce source
4. Mmap and the OS file cache: mapped model pages are file-backed host virtual memory. They should be interpreted separately from anonymous application heap and from device VRAM.
5. Features outside the text model: a multimodal projector, LoRA adapters, lookup caches, or an additional draft/speculative model have their own allocations and offload choices.

Host RAM only becomes a likely generation-speed bottleneck when important work is actually using it: CPU-resident layers, CPU-resident MoE experts, a KV cache kept off the GPU, or GPU memory spilling into system RAM. A small fixed host allocation is not evidence of per-token PCIe weight transfers.

Linux, macOS, and Windows are not equivalent

Linux with a discrete NVIDIA or AMD GPU

Linux normally gives a discrete GPU separate VRAM and system RAM. For the CUDA backend, llama.cpp documents GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 as an opt-in Linux setting that allows swapping to system RAM instead of failing when VRAM is exhausted. Keep it unset for a strict discrete-VRAM test, because it deliberately permits the behavior the test is trying to avoid. CUDA unified-memory note

Use device monitoring such as nvidia-smi alongside llama.cpp's startup memory report. For host memory, distinguish anonymous process memory from file-backed mappings where the tool can do so. Do not conclude that a large mapped virtual range is a duplicate resident copy of the whole model.

macOS, especially Apple silicon

On Apple silicon, CPU and GPU share unified memory. Apple describes hasUnifiedMemory as indicating that the GPU shares all its memory with the CPU, and states that the CPU and GPU on Apple silicon share memory. Metal unified-memory reference and Apple silicon guidance

That makes “all in VRAM, none in system RAM” the wrong question on those machines. Budget the total unified-memory working set, memory pressure, and the GPU's recommended working-set limit instead. A discrete Intel Mac GPU is different: Apple documents separate system and video memory for the discrete model. Metal storage modes

Windows with a discrete GPU

Windows WDDM manages GPU virtual addresses that can refer to dedicated GPU segments or system memory via an aperture. As a result, Windows tools can show dedicated and shared GPU-memory figures that do not map one-for-one to “the GGUF weight file.” Windows GPU segments and WDDM GPU MMU model

For CUDA specifically, the llama.cpp build guide says that Windows exposes the corresponding system-memory fallback setting in NVIDIA Control Panel. If strict VRAM residency matters, verify the driver setting and test under load, rather than relying only on Task Manager's headline counters. llama.cpp CUDA backend notes

A repeatable test plan

1. Confirm the backend. Run llama-server --version and --list-devices; make sure the expected CUDA, HIP, Vulkan, or Metal device is selected. A GPU may still accelerate some operations with -ngl 0, so that observation alone does not show full weight offload. Backend note
2. Remove unrelated pressure. On a discrete GPU, stop games, browsers, and desktop applications that use the same device. Record free VRAM before each run.
3. Start from the baseline above. Use one slot, an explicit 4K or 8K context, --cache-ram 0, and --n-gpu-layers all.
4. Read the startup report. Record model, context, and compute buffer sizes for both the selected GPU and host. Record GPU memory from the vendor tool and host RSS or a more detailed mapping report.
5. Test generation, not only load. Send a representative prompt and generate enough tokens to exercise the steady state. Watch for VRAM growth, host-RAM growth, OOM, fallback, and token rate.
6. Change one variable at a time. First context, then KV type, then --parallel, then batch settings. Record each change in a small table.
7. Only then test advanced knobs. Try --no-host, alternate backends, or Flash Attention only with a known baseline and an output-quality check.

This sequence makes the decision clear. If the all-layer GPU baseline is fast and host memory is stable, a residual host allocation is normal. If speed rises sharply when context or --parallel falls, the GPU memory budget is the limiting factor. If speed rises sharply only when CPU-MoE or partial layers are removed, the model was not fully resident on the GPU.

Failure modes and alternatives

• “I set -ngl 999, but RAM is still high.” In current llama.cpp use --n-gpu-layers all and verify the startup log. Then check --cache-ram, context, slots, and MoE settings.
• “--no-mmap reduced virtual mappings but increased RAM.” That is expected in many cases. Revert to the default load mode unless page faults are the demonstrated problem.
• “The model fits at 4K context but fails at 32K.” The KV cache scales with context. Lower context, reduce concurrent slots, or choose a cache type after validating quality.
• “The system did not crash, but generation became much slower.” Check whether unified-memory or system-memory fallback has allowed spillover. On a discrete GPU, lower the working set or use a larger GPU rather than treating spillover as a performance optimization.
• “I need zero host RAM for isolation or security.” llama.cpp is not the appropriate guarantee. The operating system, driver, and controlling process still need host memory. A separate process/container boundary or a different deployment architecture may meet the actual isolation requirement, but it is a different problem from model offload.