How should a multilingual chatbot design semantic cache keys and similarity matching across languages?

Why multilingual semantic caching is a response problem

A semantic cache tries to reuse a prior result when a new request has the same meaning, rather than the same string. A multilingual embedding model can place translations and paraphrases near each other in a shared vector space. That is valuable for finding potential matches across languages. For example, an English request about airport parking and an Italian translation may retrieve the same candidate. Multilingual models are explicitly designed for cross-lingual retrieval, and benchmark suites such as MTEB expose multilingual and cross-lingual tasks. BGE-M3 paper and MTEB documentation

But a cache returns an answer, not an abstract intent. The response can vary with more than meaning:

• The user may need a different answer language from the input language.

• A regional variant can change vocabulary, currency, date and time formats, legal information, product availability, or escalation instructions.

• The same intent can be authorized for one tenant, role, subscription, or data region but not another.

• A tool result, price, timetable, policy, inventory level, or knowledge-base revision may have changed.

• A new system prompt, model, safety policy, or retrieval corpus may change what a correct answer should contain.

Therefore, “same vector neighborhood” must never mean “same cache entry.” A semantic cache should be modeled as a policy-controlled reuse system with embeddings as one ranking feature.

Choose the cache shape deliberately

The hybrid design is usually the right default. It lets “best museums” and “museos recomendados” share a candidate intent while requiring the final answer to be in the user’s chosen language and grounded in the same current city data. If the result is a static, language-neutral object, such as an internal article identifier and version, it can be reused. If it is a prose answer containing local advice, retrieve a matching locale-specific rendering or generate a fresh one.

Separate per-language caches remain a sound choice when your content changes meaning by jurisdiction, the languages have very little traffic, or the costs of a false hit are high. A shared index is not inherently better. It only pays off when your evaluation proves that cross-language candidates increase safe reuse more than they increase false hits.

Normalize language, locale, and intent without collapsing them

Use standard language tags rather than home-grown values such as english, espanol, or zh. BCP 47 defines language tags that can include language, script, region, and variants. The script and region are not decorative: zh-Hans-CN and zh-Hant-TW, or sr-Cyrl-RS and sr-Latn-RS, can require different presentation and can signal different supported content. RFC 5646 also cautions that related-looking tags do not guarantee mutual intelligibility. RFC 5646

Keep at least these fields separate:

Prefer an explicit language choice in the user profile, conversation setup, or first-turn UI. On WhatsApp or another channel without browser headers, a saved language preference is still better than re-detecting every two-word message. Let the user change it with a short command or menu. If no preference exists, use detection only to route a fresh response or to ask, “Would you like this in Italian, English, or Spanish?” Do not let a classifier with low confidence select a cached response language.

Code-switching needs the same caution. “Museums vicino al station” may contain an Italian request, an English noun, and a misspelling. Store the raw text for processing only if your retention policy permits it, record the observed languages and confidence, and use the selected answer locale for the response. It is often safer to retrieve multilingual candidates, make a fresh answer in the preferred language, and then establish the preference for later turns.

Cache key and record design

An exact key should represent the complete response contract. A semantic vector is not itself a cache key, because nearest-neighbor search is approximate and because the vector usually omits authorization and freshness conditions.

Exact cache key

Create an L1 exact-cache key from a canonical, privacy-minimized request shape. Normalize whitespace, Unicode form, application command syntax, and stable parameters, but do not normalize away a meaningful locale, date, entity, or negation. Hash a canonical serialization of fields such as:

{
  "tenant_id": "tenant_7",
  "authorization_scope_hash": "role:support_agent|kb:public_plus",
  "policy_region": "EU",
  "requested_answer_locale": "it-IT",
  "content_locale": "rome-it",
  "intent_contract": "travel.airport_transit.v3",
  "normalized_input": "metro airport sunday morning",
  "conversation_state_hash": "no_prior_required_context",
  "model_id": "provider-model-2026-08-15",
  "system_prompt_version": "support-v14",
  "safety_policy_version": "safety-v6",
  "knowledge_snapshot": "rome-transit-2026-08-30",
  "tool_contract_version": "transit-api-v2"
}

The resulting hash can be the exact-cache key. Do not put a raw bearer token in a key or log. Create an authorization-scope fingerprint from stable, revocable permission claims after authenticating the requester. Make the scope broad enough to prevent data leakage and narrow enough to preserve useful reuse. A tenant identifier alone is often insufficient when two roles in the same tenant can see different data.

This follows a basic resource-access principle: identity and authorization should be evaluated for the resource, not assumed from network location or affiliation. NIST’s zero-trust guidance makes that distinction explicit. HTTP caching makes a similar conservative distinction for shared caches and authorized requests. A semantic cache serving a personalized or authorized answer should be at least as strict. NIST SP 800-207 and RFC 9111

Semantic cache record

Store a semantic candidate as a record with hard filters, retrieval features, and evidence of why it may be reused. One implementation could use this shape:

{
  "entry_id": "sc_01J...",
  "scope": {
    "tenant_id": "tenant_7",
    "authorization_scope_hash": "role:support_agent|kb:public_plus",
    "policy_region": "EU",
    "data_residency_region": "eu-west",
    "content_locale": "rome-it"
  },
  "request": {
    "query_text_redacted": "metro airport sunday morning",
    "observed_languages": [{"tag": "it", "confidence": 0.41}],
    "intent_id": "travel.airport_transit",
    "intent_contract_version": "v3",
    "embedding": "vector-reference",
    "embedding_model_id": "multilingual-embed-v5"
  },
  "response": {
    "answer_locale": "it-IT",
    "rendering_id": "airport-transit-it-IT-v9",
    "structured_result": {"route_id": "metro-airport", "source_revision": "2026-08-30"},
    "model_id": "provider-model-2026-08-15",
    "prompt_version": "support-v14",
    "safety_policy_version": "safety-v6"
  },
  "reuse": {
    "mode": "structured_then_render",
    "translation_policy": "approved_localized",
    "created_at": "2026-09-01T10:00:00Z",
    "expires_at": "2026-09-01T16:00:00Z",
    "invalidated_by": ["rome-transit-feed", "support-policy"],
    "quality_status": "approved"
  }
}

For private or sensitive applications, do not make raw user prompts the primary lookup artifact. Apply data minimization and redaction before storage, encrypt cache records and backups, limit access, and align retention and deletion with the source-data policy. Treat embeddings, intent labels, and cached answers as potentially sensitive as well. A vector may still reveal that a user asked about a sensitive topic, and an answer can contain confidential tenant data.

Lookup flow and reuse gates

Run the inexpensive, deterministic checks before vector search.

1. Authenticate the request and calculate the tenant, permission, policy, and residency scope. Resolve the selected answer locale and current content locale.

2. Check the exact L1 cache using the full response contract. This is cheap, explainable, and has the lowest false-hit risk.

3. If L1 misses, search the multilingual semantic index only with hard filters for scope, policy region, data residency, intent contract, live-data class, and compatible knowledge snapshot. Never use vector similarity to bridge tenants or authorization scopes.

4. Rank candidates with semantic similarity plus non-semantic features: query length, language-detection confidence, answer-locale compatibility, freshness, distance to the next candidate, and a per-intent reuse policy. A multilingual cross-encoder or translation check can be a second-stage feature, but it must not bypass the hard filters.

5. Apply the reuse gate. A high-confidence match may return a verbatim answer only if answer_locale, data snapshot, authorization scope, and generation contract match. A match to a stable structured result may be rendered in the requested locale if the translation policy allows it. All other results are cache misses.

6. On a miss, make the normal retrieval, tool, and model call. Cache the new result only if it is eligible, successfully validated, and classified with an explicit TTL and invalidation dependencies.

Use at least three outcomes instead of a binary hit or miss:

The score threshold must be calibrated from your own data. A cosine score has no universal meaning across embedding models, languages, query lengths, and domains. Choose thresholds by intent family and language pair, and consider a margin requirement between the first and second candidates. A one-word query may have a high similarity score to many entries, which is a reason to miss, not a reason to reuse the first answer.

Translation equivalence and language-specific rendering

Translations can preserve the broad intent while changing an answer-relevant constraint. “Near,” “open,” “today,” “free,” “support,” and even a product name can be locale-dependent. A word may be ambiguous across dialects, and a machine translation can lose formality, legal qualification, or a negation. This is why similarity to a translated question is not enough.

Classify cached material by the strongest safe reuse mode:

If you do allow translation, cache the translation as its own answer-locale rendering with its own quality status. Do not silently return an English answer to an Italian question merely because it is “close enough.” If translation quality matters, use a human-reviewed terminology set for names, regulated terms, and support instructions, and test it per locale.

Example with multilingual travel support

Hypothetical example. A Rome travel assistant receives “parking” from a WhatsApp user with no established preference. The language detector is low confidence because the token is valid in multiple contexts. The semantic index finds entries for airport parking, hotel parking, and parking restrictions near the historic center in English, Italian, and Spanish.

The system does not choose the nearest English answer. It asks one low-friction clarification in the channel’s available languages, such as “Parking for a hotel, the historic center, or the airport? Reply with IT, EN, or ES for your preferred answer language.” It stores the user-selected it-IT answer locale after consent. The resolved intent, content locale, and current municipal parking-feed revision then become part of the exact key and semantic filters.

On a later Italian request equivalent to “Can I park near the Colosseum this Sunday?”, the semantic index may retrieve a canonical parking-restrictions intent. Because the restriction schedule is live, the safe semantic hit reuses only the intent and relevant place identifier. The assistant calls the current city data source, then renders the reply in Italian. The cache saves the new result with the feed version and a short TTL. The takeaway is that cross-lingual retrieval helped route the request, but it did not authorize reuse of a stale or wrong-language answer.

Invalidation and TTLs

TTL is a backstop, not an invalidation strategy. Give every entry a freshness class and attach dependencies that can invalidate it:

Invalidate or namespace-bump cache records whenever the embedding model, prompt, model, tool contract, safety policy, retrieval corpus, content locale, policy region, or authorization interpretation changes. Re-embedding may change which candidates are near each other, so an embedding-model upgrade is not only a performance change. Store its identifier in every record and run a shadow evaluation before serving its results.

HTTP caching is not a direct implementation blueprint for an LLM cache, but its conservatism is useful. RFC 9111 distinguishes freshness, explicit expiry, private responses, and authorized requests. Apply the same discipline to semantic outputs: do not serve a cached answer after the facts or the conditions that made it safe have changed. RFC 9111

Evaluate before serving and continuously afterward

Offline evaluation set

Build a labeled, versioned evaluation set from permitted historical traffic and deliberately created cases. Never use production prompts without applying the same privacy and retention rules as the cache itself. The unit of evaluation should be a proposed reuse decision, not simply a pair of similar sentences.

Include these groups for every supported language and important locale:

• Exact repeats, same-language paraphrases, and cross-language human translation pairs.

• Short queries, spelling variants, dialect variants, script variants, and code-switched messages.

• Near misses that look similar but have different intent, entity, negation, date, user role, or requested answer language.

• Locale-sensitive prompts involving units, currency, geography, opening times, policy jurisdiction, and culturally localized terms.

• Tenant and authorization boundary pairs that have identical wording but must never share an answer.

• Freshness cases where the old cached result became invalid after a source, policy, prompt, or tool update.

• High-risk categories that must bypass semantic response reuse, including personalized data, live tool calls, or content subject to a stricter safety policy.

For each (query, candidate) pair, label: intent match, answer-language compatibility, content-locale compatibility, authorization compatibility, freshness, safe reuse mode, and false-hit severity. A useful gold label set is verbatim_reuse, structured_reuse_then_render, fresh_generate, and never_cache. Review labels with native speakers and domain owners, not only with an automatic translation system.

Evaluate the embedding and the policy gate separately. Multilingual benchmarks can help select candidates and reveal language coverage, but they do not prove that your support answer is safe to reuse. MTEB supports selecting multilingual and cross-lingual tasks by language and retrieval type, which is useful for a reproducible model screen. Final selection still needs your domain set and false-hit analysis. MTEB documentation and MTEB repository

Production metrics

Measure cache value and cache harm together. A high hit rate with wrong answers is a regression.

Run new cache policies in shadow mode first. Record what would have been served, then compare it with a fresh answer and human review for a stratified sample. In production, sample semantic hits for review, monitor user corrections and re-asks, and alert immediately on a tenant, authorization, language, or freshness mismatch. Keep a kill switch that disables semantic answer reuse while leaving exact cache and fresh generation available.

Common failure modes and better alternatives

• One global similarity threshold. Calibrate by language pair, intent, query length, and cache class. A similarity score is a model-specific ranking signal, not a probability of safe reuse.

• Using detected language as the only locale field. Keep observed language, user-selected answer locale, content locale, and policy region distinct.

• Filtering language but not authorization. Language metadata prevents a visible mismatch, not a privacy breach. Apply identity and authorization filters before candidate retrieval.

• Caching polished prose when a structured result would do. Cache canonical IDs, facts, and source revisions separately from the localized rendering.

• Treating translation as a free cache hit. A translation can be fluent but lose conditions, terminology, or local applicability. Use explicit translation policy and evaluation.

• Applying TTL without dependencies. Attach source revisions and invalidation events. A timetable or policy update should purge relevant entries before their clock expires.

• Reporting hit rate alone. Pair it with safe-hit precision, false-hit severity, language adherence, and coverage by language.

If the organization cannot yet enforce authorization and freshness as hard filters, use an exact cache only. It is cheaper to add semantic caching later than to unwind a cache that has leaked data or normalized wrong answers. For many systems, an exact response cache plus a retrieval cache for public, versioned documents captures substantial latency and cost benefits with much lower reuse risk.