How should production RAG systems detect and handle stale context?

Distinguish the clocks in a RAG corpus

“Timestamp” is too vague for production retrieval. A document can be published years ago but still valid, updated today but not effective until next month, fetched recently but unchanged, or indexed recently from a source already revoked. Store the clocks separately so the pipeline can make a defensible decision.

Keep the raw fetched representation, normalized representation, and their hashes separate. A new raw hash may result from a harmless layout change, while a new normalized-content hash may indicate a semantic change. Do not silently overwrite an old document. Create a revision record, state why it changed, and retain enough protected lineage to explain an earlier answer. The W3C PROV family defines provenance in terms of entities, activities, and agents, and specifically supports representing revision and derivation. W3C PROV primer W3C PROV namespace

For HTTP sources, collect validators when available. RFC 9110 recommends Last-Modified for representations where a modification date can be reasonably and consistently determined, and ETag plus conditional requests can avoid re-downloading an unchanged representation. RFC 9110 HTTP Semantics A source that lacks validators is not automatically untrustworthy, but it needs a more conservative polling, hashing, and review plan.

Decide eligibility before applying temporal decay

Use a two-stage decision. First, evaluate whether a candidate is eligible for the query and requested time. Second, rank eligible candidates. This prevents a time-decay formula from converting an invalid document into a merely lower-ranked document.

An illustrative eligibility rule is:

eligible(chunk, query_time, user, use_case) =
  access_permits(user, chunk) and
  source_status(chunk) == "active" and
  chunk.effective_from <= query_time and
  (chunk.effective_to is empty or query_time < chunk.effective_to) and
  not superseded_for_scope(chunk, query_time, use_case) and
  validation_is_current_enough(chunk, use_case)

The exact policy varies by source class. A controlled corporate policy might require a current revision, approved owner, effective date, and explicit supersession chain. A technical tutorial might remain eligible after its review date but acquire a freshness warning. A user asking “What was the policy in 2023?” changes query_time to an historical date, so an earlier revision can become the correct answer. Make the “as of” time an explicit query parameter rather than assuming every question means now.

This resembles a basic caching principle. RFC 9111 defines a fresh response as one whose age has not exceeded its freshness lifetime and distinguishes explicit from heuristic expiration. It also states that must-revalidate responses cannot be reused after becoming stale until they are successfully validated. RFC 9111 HTTP Caching RAG systems do not need to copy HTTP behavior exactly, but the separation between explicit validity and a heuristic is a useful design rule.

Model source records, document revisions, and chunks separately

The common failure is to attach one created_at field to every chunk and call the problem solved. A chunk is a retrieval artifact, not the source of truth. Its metadata should link to a revisioned source record and to the rules that made it visible.

Source
  canonical_url, publisher, authority_class, jurisdiction, access_policy

Document revision
  document_id, revision_id, source_version, raw_hash, normalized_hash
  published_at, source_updated_at, fetched_at, validated_at
  effective_from, effective_to, review_due_at, status, supersedes

Chunk revision
  chunk_id, document_revision_id, section_anchor, text_hash
  embedding_model, embedding_version, indexed_at, vector_namespace
  visibility_state, metadata_schema_version

Retrieval evidence
  answer_id, query_time, index_snapshot, candidate_ids, filters_applied
  rank_features, cited_chunk_ids, generation_model, response_time

status should be an explicit finite state such as draft, scheduled, active, superseded, revoked, deleted, quarantined, or unknown. Do not infer all of those states from age. A “deleted” source needs an immediate search visibility change. A “quarantined” source may have been flagged for poisoning, rights, access-control, or quality review even if it is new.

The record also needs source authority and scope. A current vendor release note can supersede an earlier release note about that product, but not a statute. A policy from one business unit might be current only for that business unit. Treat scope, jurisdiction, product version, and audience as structured eligibility fields, not prose the model must interpret.

Re-index through validated source revisions

Use event-driven updates when a source offers webhooks, release feeds, a database change stream, or a content-management event. Supplement them with scheduled reconciliation, because events can be missed and publishers can alter a document without a clean notification. For HTTP sources, conditional fetch with If-None-Match or If-Modified-Since reduces unnecessary transfer when validators are reliable. A 304 response updates validated_at, not source_updated_at or embedded_at. RFC 9110 conditional requests

When a source changes, use an idempotent revision workflow.

1. Fetch the source, record validators and raw content, then canonicalize it for comparison without discarding the original.
2. Detect whether the normalized content, structured effective dates, authority, access policy, or source status changed. Route ambiguous changes to review for high-risk source classes.
3. Create a new document revision. Extract chunks, attach inherited and revision-specific metadata, and calculate embeddings in a staging namespace.
4. Validate the staged revision. Check metadata completeness, access labels, source lineage, chunk count, parsing errors, and a small set of retrieval regression queries.
5. Atomically change the active pointer from the old revision to the new one for the relevant scope. Retire old chunks from “current” retrieval, while retaining protected lineage for historical queries and audit if policy permits.
6. Emit an invalidation event for result caches, citation caches, materialized summaries, and any derived answer store that referenced the prior revision.
7. Monitor the job until the new revision is searchable, the old one is not served as current, and downstream invalidations have completed.

Do not delete an old revision before the new one is validated. That creates a gap in recall and makes rollback difficult. Do not keep both active without a version-precedence rule. That creates a conflict in which vector similarity can choose an obsolete source by chance.

For a source that disappears, returns an explicit deletion response, loses authorization, or is withdrawn by its owner, create a tombstone event. The online retrieval filter should exclude it immediately, before asynchronous physical deletion from vector shards, replicas, caches, and backups. Preserve a minimal tombstone record with source ID, event time, cause, and retention policy so an audit can explain why a past citation no longer resolves. RFC 9111 likewise distinguishes invalidating a stored response from physically removing it, which is a helpful operational distinction for this design. RFC 9111 invalidation

Use freshness-aware ranking only among eligible candidates

After hard filters, rank candidates with features whose roles are visible. A simple starting model might combine lexical or semantic relevance, authority, scope match, freshness, document quality, and diversity. The numbers are an engineering hypothesis to evaluate, not an universal formula.

final_score =
  0.45 * topical_relevance +
  0.20 * authority_match +
  0.15 * scope_match +
  0.15 * freshness_signal +
  0.05 * source_quality

Compute freshness_signal from an explicit source-class policy, not a single global half-life. Inputs can include time since last successful validation, time to review due date, source update velocity, observed change frequency, and a query's temporal intent. A new but low-authority blog post should not automatically outrank a current regulator or controlled policy. A recently re-embedded unchanged document should not become “fresh” merely because the pipeline ran yesterday.

Temporal decay helps when all candidates remain acceptable and the question benefits from a newer operational detail. It should be disabled, reversed, or constrained when the user asks for a historical answer, when a source has a long-lived authoritative edition, or when source lineage already tells you which revision is current. Treat it as a reranking feature, not a truth metric.

Log feature contributions for the selected citation. A citation record such as “retrieved because it was semantically close” is weak. A record that says “active revision, effective at query time, validated 8 hours ago, matches product version 4.2, ranked second after authority adjustment” is inspectable and debuggable.

Resolve conflicting versions before generation

Two chunks can contradict each other for legitimate reasons. They may concern different product versions, regions, effective dates, user roles, or source authorities. They may also reveal a real source error. Do not delegate this distinction to the generator alone.

Use a conflict resolver that groups candidates by topic or claim, then applies deterministic precedence where possible.

1. Partition by scope. Separate product versions, jurisdictions, customer plans, audiences, languages, and “as of” periods before comparing statements.
2. Follow verified lineage. Prefer a revision that explicitly supersedes an earlier revision in the same scope. Treat a revocation as a hard exclusion.
3. Apply source authority. A controlled policy, official release note, or system of record can take precedence over an uncontrolled copy, summary, or community post when they address the same claim.
4. Check effective intervals. A scheduled future version should not override the active version before its effective date. A historical query may need the opposite result.
5. Surface unresolved conflict. If two active authoritative sources disagree and no declared precedence exists, cite both, describe the scope of disagreement, and route high-stakes questions to a human or system of record.

This is also a security control. OWASP identifies vector and embedding weaknesses in RAG, including data poisoning, access-control failures, and conflicts when sources contradict each other. It recommends validating documents before they enter a RAG knowledge base and applying permission-aware retrieval in multi-tenant systems. OWASP LLM08 Vector and Embedding Weaknesses

Link citations to specific source revisions

A user-facing citation should link to a stable URL, title, source, and relevant section. The internal evidence record needs more. For every generated answer, persist enough information to reproduce what the system was allowed to see at that moment.

Store this evidence under access and retention rules appropriate to the corpus. Provenance can itself reveal sensitive relationships, so do not make all internal lineage public. W3C notes that provenance can support trust and compliance assessments, while provenance access can also create privacy concerns. W3C PROV primer W3C PROV Access and Query

Invalidate result and answer caches as part of the update

RAG commonly has several caches: source-fetch cache, parsed-document cache, embedding cache, vector-search cache, retrieved-context cache, reranker cache, generated-answer cache, and client or CDN cache. Each can reintroduce stale context after the index has been fixed.

Every cache key should include the values that determine the answer: normalized query, query time or “as of” time, user or authorization scope, use case, source-selection policy, corpus or index snapshot, embedding and reranker version, prompt version, and locale. A broad cache key such as only the user question risks serving yesterday's answer under today's policy and access rules.

Set a cache entry's maximum lifetime to no later than the earliest applicable source expiry, next validation time, policy review deadline, or authorization expiry. Prefer event-driven invalidation keyed by document revision and source scope. When source X is superseded or deleted, publish an event that invalidates all cache entries and materialized answers whose evidence set contains X. If fan-out is expensive, use a corpus-generation or source-version vector in the cache key so old entries become unreachable when the active generation changes.

Do not use a time-to-live as the only deletion mechanism. A deleted or revoked source should be excluded immediately, not when an arbitrary cache timer ends. HTTP caching offers a useful reference point: state-changing responses can require invalidation, and a response that must be revalidated cannot be served stale as a convenience. RFC 9111

Monitor freshness as a production SLO

Vector-search relevance metrics can remain high while the corpus is historically wrong. Add freshness and validity metrics to the same operational posture as latency, recall, and cost.

Make these measures source-class specific. A daily validation objective can be excellent for a software documentation site and unacceptable for an outage-status feed. Do not invent a universal half-life. Define an owner, a target cadence, a failure action, and a safe fallback for each class.

NIST's AI RMF calls for inventory, pre-deployment and ongoing evaluation, documentation of test sets and metrics, monitoring in production, and tracking existing and emergent risks. Those ideas map naturally to source lifecycle and stale-context monitoring. The AI RMF is voluntary and is being revised, so use it as a risk-management guide rather than a claim of RAG compliance. NIST AI RMF Core NIST AI RMF update notice

Evaluate with time as part of the test set

Offline retrieval evaluation should contain time-scoped questions and labels. For each question, record the intended “as of” date, permitted source classes, valid document revisions, invalid but semantically tempting revisions, and the authority expected to support the answer. Create hard negatives from superseded policies, old product documentation, revoked advisories, and sources from the wrong region or product version.

Measure ordinary retrieval quality, such as Recall at K, precision at K, mean reciprocal rank, and nDCG, separately from validity. NIST TREC has long used precision, recall, reciprocal rank, and nDCG-family measures for ranked retrieval tasks. NIST TREC evaluation overview Add RAG-specific measures such as:

• valid-source precision at K, the proportion of top candidates eligible at the query time;
• superseded-citation rate, the proportion of answers citing a retired revision as current;
• conflict-resolution accuracy, based on known precedence cases;
• deletion-retrieval rate, which should be zero after the service objective;
• citation reproducibility, the share of sampled answers whose evidence can be reconstructed from stored lineage;
• freshness-loss rate, where a decay rule hides a still-valid authoritative answer without a policy reason.

Run replay tests after source, embedding, chunking, ranking, policy, or cache changes. Keep a temporal holdout that arrives after the training and tuning period. For high-impact domains, have subject-matter experts review source authority, temporal validity, and the system's refusal behavior, not only answer fluency.

A concrete example

Example

Hypothetical setup: a company operates a RAG assistant over internal security standards and vendor advisories. A user asks, “What is the required remediation for affected gateway version 4.2?” The corpus contains a six-month-old advisory with a high semantic score, a current replacement advisory that changed the remediation steps, and a current release note for version 4.3 that does not apply to version 4.2.

At ingestion, the current advisory is recorded as the later revision of the earlier advisory, with its effective date, affected-version range, source authority, content hash, and review deadline. The first advisory is marked superseded for current queries. The release note is active but its product-version scope does not match the query. The pipeline re-indexes the new advisory, changes the active pointer, tombstones the old revision for current retrieval, and publishes an event that invalidates cached answers citing it.

At query time, the resolver filters the old advisory because it is superseded and filters the 4.3 release note because its scope does not match. It ranks the current 4.2 advisory, cites its revision and effective date, and answers with the stated remediation. If the advisory source is unreachable and its validation deadline has passed, the assistant should say that it cannot verify the current remediation rather than serving the older instructions. In this case, version relationships and product scope determine which advisory applies.

An incremental implementation path

1. Inventory sources and classify risk. Start with sources that change frequently or carry high consequence. Assign each a source owner, authority class, scope, target validation cadence, and safe fallback.

2. Add revision metadata. Store canonical URL, source and content version, hashes, fetched and validated times, effective interval, status, and predecessor or successor links. Make new fields mandatory for high-risk ingestion.

3. Introduce eligibility filters. Exclude deleted, revoked, expired, unauthorized, out-of-scope, and superseded content before ranking. Support explicit historical queries with an as_of parameter.

4. Version the indexing pipeline. Stage new revisions, validate them, atomically activate them, retire prior chunks, and create tombstones. Make jobs idempotent and test out-of-order source events.

5. Add cache invalidation and lineage. Include snapshot and scope information in cache keys. Emit invalidation events on revision and tombstone events. Persist evidence records for user-facing citations.

6. Evaluate and tune freshness-aware ranking. Start with source-class rules and a small, interpretable freshness feature. Use time-scoped evaluation data before adding learned decay or temporal rerankers.

7. Operate with monitoring and drills. Set alerts for validation lag, expired retrieval, deletion propagation, conflicts, cache errors, and citation failure. Exercise source withdrawal, broken feed, provider outage, and manual fallback paths.

Failure modes and viable alternatives

For a small, stable corpus, a versioned relational database plus full-text search and deterministic metadata filters may be safer and easier to operate than a complex temporal vector stack. For a larger heterogeneous corpus, use hybrid retrieval with the same validity filter in front of keyword, vector, graph, or reranking stages. The architecture can change. The invariant is that an answer must not treat an unverified or invalid source as current evidence.