If an LLM receives conversation history, why is the model or API still called stateless?

Four different things called memory

The word “memory” is overloaded. A useful explanation begins by locating each kind of state, who owns it, and what changes it.

The first row is the neural network itself. In normal inference, its weights are fixed while it computes a probability distribution for the next token from the supplied input. A user saying “my preferred language is Spanish” does not rewrite those weights. It can affect a later answer only if a later context contains that sentence, a trustworthy summary of it, or a retrieved memory record that says the same thing.

The other three rows are system design choices. They can make an assistant feel persistent, but they belong to the caller, the application, or a provider service. Calling the whole product “stateless” without naming the layer is imprecise. Calling the base model's ordinary inference stateless is useful because it says that the model has no user-specific history to consult unless the current request supplies one.

What happens on each turn

In a simple stateless API pattern, the client owns the history. It stores the first user message and the model's answer, then sends the selected prior turns plus the new message on the next request. The model receives a new sequence of tokens and generates another response. It does not need a durable model-side chat slot to make the conversation coherent.

In a stateful service pattern, the client may send only the new message and a prior-response or conversation identifier. The service looks up stored conversation items, builds the actual input internally, and runs the model. This can reduce client-side orchestration, but it does not turn the model weights into a per-user notebook. It moves the persistence boundary from the application to a provider-managed resource.

Current APIs show both patterns. OpenAI documents that a Responses conversation can prepend its stored items to a request, while previous_response_id is another way to continue a multi-turn interaction. The same reference explicitly describes a stateless path using returned encrypted reasoning items when storage is disabled. OpenAI Responses API reference Google similarly documents an optional server-side interaction record continued with previous_interaction_id, and also a stateless mode where the caller sends the full conversation history. Gemini Interactions API

That distinction matters operationally. A server-side conversation resource has retention, deletion, access-control, and regional-processing implications. A client-managed history has the same considerations, but the application owns more of the implementation. In either case, the model only receives the state selected for the current inference.

A three-request example

Assume an application uses the same model and the same system instruction for all three requests. The user first says, “For this project, the deployment color is amber.” The assistant acknowledges it. The user then asks an unrelated question about release notes.

Nothing about the weights changes between request 3A and 3B. The input is different. A model can still guess, hallucinate, or infer from unrelated wording, so absence of history is not a mathematical guarantee of a particular sentence. But it has no evidence from this conversation for the fact that was left out, and a well-designed assistant should say it cannot determine the color rather than invent one.

This is why “working memory” is a useful but limited analogy. The model attends to the tokens in the active context while generating a response, so it can use them as short-lived working information. The words do not mean it owns a durable personal record, nor do they imply that it can recall a previous session whose information was never reintroduced.

Context windows are capacity limits, not history stores

A context window is the maximum token budget that the model can consider in one inference. The effective budget is shared by more than a visible chat transcript: system and developer instructions, tool definitions and results, documents, retrieved snippets, images or other inputs where supported, prior assistant text, the new user message, and room for the answer all consume capacity. Exact limits and automatic-truncation behavior vary by model and endpoint, so they must be checked in the current documentation for the model actually deployed.

When the assembled prompt is too long, the system must choose what to omit or compress. Some services may apply endpoint-specific truncation. For example, OpenAI's Realtime reference describes a mode that drops the oldest messages once the input token limit is exceeded, or returns an error when truncation is disabled. That is an API behavior, not model learning or forgetting in the human sense. OpenAI Realtime truncation reference

Applications commonly combine four techniques:

• Recent-turn window: retain the latest turns verbatim so references such as “that table” or “the second option” remain usable.
• Rolling summary: replace older material with a concise factual summary. This saves tokens but is lossy, can omit nuance, and can preserve an earlier mistake unless the source record remains available.
• Retrieval: search a conversation archive, user profile, or document collection for information relevant to the new question and add only the selected results. Retrieval can miss the right item or surface stale and conflicting facts, so provenance and timestamps matter.
• Explicit reset: begin a new task or conversation when old context should not influence the next answer. This is often safer than treating every chat as indefinitely continuous.

Use an authoritative transcript or business record outside the prompt. A summary and retrieval index are derived aids, not the sole record of what a user said. If a summary is updated in place without a source trail, it becomes difficult to correct or explain a wrong remembered fact.

Longer context also has a cost. More input must be tokenized, transmitted, and processed, and providers may bill or optimize it differently. Prefix caching can improve latency or input economics for repeated material, but it does not remove the need to manage the active context or make irrelevant history useful. Measure token usage for representative long conversations and set a budget for current turns, summaries, retrieved evidence, tool output, and requested answer length.

Why a KV cache is not durable conversational memory

During autoregressive generation, a transformer predicts one token at a time. Recomputing the attention keys and values for every earlier token at every step would be wasteful. A key-value, or KV, cache keeps tensor representations calculated from preceding tokens so the next-token computation can reuse them. Hugging Face describes the cache as an inference optimization that stores keys and values derived from attention layers to avoid recomputing them. Transformers caching documentation

That cache is useful within a generation or a carefully managed continuation, but it is not a user-facing database of messages or a weight update. It is an implementation-level representation tied to a model, prompt prefix, tokenizer, and execution environment. It may be freed at the end of a request, evicted under memory pressure, or reused only when a compatible prefix matches. An application cannot safely treat the existence of a KV cache as proof that a later request will remember a user fact.

Some platforms retain or reuse cached prefixes beyond one call to reduce latency. This changes performance and sometimes retention obligations, not the conceptual boundary. OpenAI, for example, documents that prompt caching can store encrypted key-value tensors in GPU-local application state for a limited period, while conversation resources have their own storage lifecycle. OpenAI data controls A robust application still keeps its own explicit conversation and memory policy instead of relying on an opaque cache as a source of truth.

Optional product memory needs a policy

Product memory is a deliberate feature built on top of inference. A system might extract “prefers concise answers,” “uses metric units,” or “works on Project Atlas” into a user-scoped record, then retrieve a few relevant facts before each request. This can improve continuity without replaying an entire transcript, but it has different failure modes from a conversation window.

First, decide what qualifies as memory. A stable preference may be appropriate; an offhand request, a sensitive health detail, a credential, or an inferred attribute usually needs much stricter rules or should be excluded. Make the source, confidence, timestamp, scope, and user control visible enough that an incorrect memory can be corrected.

Second, decide how it is used. Insert only relevant, validated entries into context and label them as stored preferences or past facts. Do not let an unverified model-generated summary silently override an explicit recent user correction. When user information conflicts, prefer the newest explicit instruction and preserve the conflict for review rather than blending it into a vague profile.

Third, decide how it is governed. Memory records and conversation logs can be personal data even when model weights do not change. Define access controls, encryption, retention, deletion, export, tenant separation, and incident handling. If a provider-managed conversation or memory feature is in scope, read its current storage and retention controls rather than assuming that a request identifier is ephemeral. OpenAI's data-controls documentation, for example, distinguishes endpoints with no application state from conversation resources retained until deletion. OpenAI data controls

Provider state is real state, but it is not model state

The word “stateless” sometimes becomes misleading when it is used to describe an entire API product. Provider-managed conversation identifiers, threads, session records, files, vector stores, and memory features are genuine stored state. They can be convenient, and they may be exactly the right product choice. The precise claim is narrower: stored state is retrieved and supplied to a model invocation, not absorbed as a private weight change caused by that individual chat.

The differences are significant enough that API code must not make assumptions across providers or endpoints. Google's current Interactions API says a continued interaction preserves conversation history but requires the caller to re-specify tools, system instruction, and generation configuration for the new interaction. Gemini Interactions API OpenAI documents a different conversation-resource and prior-response model. A production integration should specify which fields are persisted, which are repeated, what is counted against context, who can retrieve the state, and how it is deleted.

Use provider state for convenience only after choosing its data boundary deliberately. Keep application identifiers and authorization checks outside the prompt, even when a provider supplies a thread or conversation ID. A model should never be allowed to select another user's conversation merely because it can emit a plausible identifier.

Implementation checklist

• Name the state boundary in the design: model weights, active context, application transcript, provider conversation resource, retrieval index, and optional user memory.
• Store an append-only or versioned authoritative transcript with message roles, timestamps, source identifiers, and access controls. Treat summaries and embeddings as derived records.
• Build each request under a token budget. Reserve output capacity before packing recent turns, a summary, retrieved records, tool results, and the new message.
• Define a deterministic policy for truncating, summarizing, and retrieving. Test whether the critical fact is retained, corrected, or intentionally forgotten at each boundary.
• Use explicit user and tenant scopes in every conversation and memory lookup. Do not rely on a model instruction alone to protect cross-user data.
• Surface a way to inspect, correct, reset, export, and delete saved memory where the product promises persistence.
• Log only the operational data needed for debugging. Separate sensitive transcripts from general analytics, and apply encryption and retention controls to both.
• Test a missing-history case such as request 3B. The assistant should acknowledge uncertainty instead of fabricating a prior fact.
• Track input tokens, output tokens, truncation events, summary revisions, retrieval hit quality, context-build latency, and privacy deletion completion.
• Recheck provider documentation whenever changing endpoint, model, storage settings, data-retention terms, conversation mode, or prompt-caching configuration.