How should an offline multimodal RAG system index and retrieve documents, images, and audio?

The design principle

Offline means that document bytes, model weights, indexes, logs, and inference requests stay inside a boundary you control. It does not automatically mean secure. A local desktop application can still leak through backups, crash reports, a shared filesystem, a package updater, or a model runtime configured to pull weights on demand. Decide the boundary first: single-user encrypted workstation, a disconnected team server, or an internal network service. Pin and checksum models and parsing containers before the boundary is closed, disable telemetry and remote fetches, and record the version of every component that affects an answer.

Retrieve source records that support the answer, with their original locations. The primary key should identify the source asset and version. Each child record should identify an address in that version: a PDF page and normalized bounding box, a DOCX paragraph or table cell, an image region, or an audio start and end time. The answer model receives only permitted records plus their anchors. It must cite record IDs that the application resolves, rather than inventing page numbers from a flattened string.

This approach also avoids a common failure. Converting every modality into an image caption and indexing only captions is convenient, but it loses exact table values, reading order, figure relationships, and audio timing. Conversely, indexing only raw pixels makes exact product codes, names, and quoted wording unnecessarily hard to find. Preserve multiple representations and select the evidence that fits the question.

End-to-end flow

1. Register and preserve. On intake, calculate a content hash, capture original filename, MIME type, source system ID, owner, labels, ACL, and ingestion time. Write the source file to an encrypted content store or retain a read-only reference. Deduplicate only after confirming that the prospective duplicate has compatible retention and access rules.

2. Classify and parse locally. Detect the actual format from bytes, not only its extension. A general parser is useful as a routing layer. For example, Apache Tika documents supported parsers for PDF, Office formats, images, and audio, and can combine image extraction with Tesseract OCR (Tika 3.3.2 supported formats). Keep parser output, warnings, and the parser version so that a later reparse is explainable.

3. Create modality-native evidence. Extract born-digital text and structure from PDF and DOCX. Render only the pages that require OCR or visual retrieval. Run OCR on scans and embedded images, retaining OCR confidence and word boxes. Extract tables both as a structured grid and as a readable text serialization. Decode audio to a consistent local format, apply voice activity detection if useful, transcribe it, and retain segment and, where reliable, word timestamps. Whisper-derived local implementations can run CPU-only and have hardware-specific acceleration options; the project documents both quantization and CPU-only support (whisper.cpp).

4. Chunk without breaking the source. Form text chunks from headings, paragraphs, list items, and table units, with a small overlap only where the source structure requires it. Keep a table together if it fits; otherwise make row or row-group chunks that repeat headers. Make image records for the whole image and meaningful regions. Make audio chunks at natural transcript or speaker boundaries, normally with a modest overlap and a time span. Chunk IDs must retain their parent asset version and anchors.

5. Represent, index, and serve. Generate the representations described below, write them in one transaction or a recoverable job state, and mark the asset version searchable only after all mandatory records have succeeded. At query time, authenticate, enforce filters before ranking, retrieve in parallel, fuse, rerank, reconstruct citations from anchors, and ask the local LLM to answer only from the selected evidence.

Parsing and extraction choices

Documents and layout

For a PDF with a good text layer, preserve reading order, page number, font or style clues when available, headings, links, annotations if policy allows them, and coordinates. For a DOCX, use its native paragraph, heading, table, and media relationships before considering a rendered PDF. A universal parser can provide a fallback, but it is not proof that the recovered reading order or table structure is correct.

Run a page-quality gate before OCR. A simple first pass can record text density, image coverage, parser errors, rotation, and characters that look garbled. OCR a page if it has no usable text layer, is a scan, or fails quality checks. Keep the source image resolution used for OCR, language packs, engine version, and confidence. Tesseract's documentation makes clear that its current source and language data are separate artifacts, so pin both instead of treating OCR as a single opaque dependency (Tesseract documentation).

Layout data is not decorative metadata. It enables the user interface to highlight the exact evidence and lets retrieval distinguish a footnote from a heading, a figure caption from body text, and a table cell from surrounding prose. Some extraction libraries expose page number, coordinates, and HTML table representations; for example, Unstructured documents page number and coordinate metadata, and table HTML when available (document elements and metadata). Validate any layout extractor against your own documents, especially multi-column pages, forms, and tables without borders.

Images and visual pages

Store the original image, technical metadata, OCR text and boxes, plus one of two complementary semantic representations:

• A short local image caption is text-searchable and lets a text-only answer model describe why the image was retrieved. It is cheap to index but should be treated as a fallible derived annotation, not as the image's authority.
• A text-image embedding model maps image or page regions and text queries into a comparable space. It can discover a relevant diagram when its labels were missed by OCR. A document-vision retriever can go further by embedding rendered pages and preserving signals from layout, charts, and tables. Hugging Face's current visual document retrieval guide describes page-image and text-query similarity for this type of retrieval (visual document retrieval).

For reports, index both individual embedded images and full rendered pages. The image index handles a photograph or schematic; the page index handles a table whose meaning depends on column alignment and its caption. Save the rendering DPI, page image hash, crop coordinates, and model version. If a query hits a visual record, pass the actual permitted crop or page to a local vision-language model for grounding, and cite the page or region.

Audio

Decode audio locally, retain the original file, and record duration, channels, sample rate, language hypothesis, and any known speaker or recording context. Speech-to-text gives the main retrieval text, but do not concatenate an hour-long transcript into one document. Use timestamped segments with stable IDs. Whisper's transcription implementation produces segment details and supports optional word timestamps, which are useful anchors but should be tested on the relevant language and audio quality (Whisper transcription code).

Index transcript segments for lexical and text-semantic retrieval. If the product needs queries such as "where does the speaker sound frustrated?" or audio matching, add an audio embedding index separately. Speaker diarization is a different task from transcription. Store speaker labels as tentative unless an approved diarization process produced them, and never identify a person from voice merely because a speaker turn exists.

A shared schema for source records

Use an asset table for lifecycle and authorization, a record table for addressable evidence, and representation tables for indexes. The following logical schema works in a relational store, a local document store, or a combination. Arrays can be normalized into child tables when filtering or auditing needs it.

Keep raw source bytes, normalized extraction, and model-generated annotations distinct. A caption or a transcript correction may change without changing the original asset. Record each transformation as a directed derivation from an input hash to an output hash. That graph makes selective reprocessing possible when an OCR engine, embedding model, or policy changes.

Combine retrieval results while preserving source detail

Indexes to keep

One user-facing query endpoint should fan out to several indexes over the same record IDs:

The shared space option is a query text vector compared directly with image or page vectors. It is valuable for discovery and can reduce plumbing. Use it only when the model was trained for the pair you need, such as text to image or text to document page. Do not compare embeddings from unrelated models, even when their dimensions match. A model identifier, dimensionality, normalization rule, and distance metric belong with every vector. Current local embedding tooling likewise warns to use the same embedding model for indexing and querying (Ollama embeddings guidance).

Separate indexes remain worthwhile because the evidence and scoring mechanisms differ. A text vector has one semantic summary. A page-aware visual retriever may use many token or patch vectors and late interaction to score a query against page detail. The current ColPali documentation describes this page-as-image, multi-vector approach for layout, tables, charts, and visual elements (ColPali documentation). An audio transcript needs time-range metadata that is not encoded in a vector. Separate indexes can still return the same record_id and share authorization, lifecycle, fusion, and citation code.

Query processing and late fusion

Apply ACL, tenant, legal-hold, and retention predicates at the first possible stage. Do not retrieve an unauthorized record and rely on the LLM not to mention it. For approximate nearest-neighbor indexes, test filters carefully because candidate filtering can reduce recall. A relational implementation can co-locate metadata and vectors; pgvector documents exact search by default and HNSW or IVFFlat as approximate indexes that trade recall for speed (pgvector README). For small offline collections, exact search can be simpler and more predictable.

Launch the enabled legs in parallel. Use query routing as an optimization, not a secrecy boundary: a query containing a serial number should boost lexical search; "show the diagram" should enable visual legs; "what was said about the budget?" should prioritize transcript segments. Fetch a reasonably deep list from each leg, such as 30 to 100 candidates depending on corpus size, then collapse near-duplicate records from the same asset and page.

Fuse ranks rather than raw similarity scores. Text and image scores are generally not calibrated to one another. Reciprocal rank fusion is a robust starting point: for each result, add weight / (constant + rank) across lists, then tune weights against held-out questions. The constant prevents the top ranks from dominating entirely. SQLite FTS5, for example, exposes BM25 ranking and column weighting for a local lexical leg (SQLite FTS5). Keep feature values and per-leg ranks in the query log so poor answers can be diagnosed.

Rerank only the top fused, authorized candidates. A text cross-encoder can score a query with the extracted passage; a local vision-language scorer can evaluate a question with a page image or crop. Then assemble a small, diverse context: avoid returning five overlapping chunks from the same page when one passage, one table, and one figure give better coverage. The reranker selects evidence, not truth. The final answer model should be instructed to distinguish OCR or transcript uncertainty, quote only what the retrieved record says, and say when evidence is insufficient.

Example

Suppose a user asks, "Which warning label is next to the pressure chart, and what did the maintenance call say about it?" The system first routes the query to lexical, text-semantic, visual-page, and transcript legs. The lexical leg can find the words "pressure" and the transcript phrase; visual-page retrieval can find the chart and adjacent label even when the label is small; transcript retrieval returns the relevant 15-second segment.

It then fuses the allowed page and audio candidates, reranks the page image against the question, and returns a context containing the rendered page crop, its OCR text with a box, and the timestamped transcript segment. The answer cites "manual.pdf, page 18, chart-adjacent label" and "maintenance-call.wav, 12:22 to 12:37." One answer can join modalities without pretending that the original image and audio are ordinary text chunks.

Local model selection

Choose model families by measured task fit, license, hardware footprint, language support, and reproducibility, not leaderboard rank alone. Use a compact local embedding model for most text and transcript chunks, an OCR engine plus a layout extractor for documents, a text-image or visual-document retriever only if evaluation has visual questions, a speech-to-text model that provides stable segments, and an instruction-tuned local LLM that can follow a strict citation format. Keep model selection behind versioned adapters so a model replacement creates a new representation version rather than silently mixing incompatible vectors.

For a constrained CPU-only machine, favor efficient text embeddings, a modest quantized transcription model, lexical retrieval, and a small quantized answer model. Defer full-page visual retrieval or run it in a background job. For a workstation with a capable GPU, batch OCR and page embedding, use a stronger reranker, and reserve memory for the answer model. GPU acceleration lowers latency only after model load, preprocessing, and storage access stop dominating. A very large model that causes queueing can make the user experience worse than a smaller model with a better retrieval and citation path.

Measure four latencies separately: ingestion time per page or audio minute, cold-start model load, search fan-out and fusion, and answer generation. Index build time and storage matter too. Approximate indexes save query time at scale but require recall testing. Quantization and lower precision can reduce memory pressure, yet every change must be measured for retrieval quality and for the ability to reproduce citations.

Security, permissions, updates, and deletion

Authorization belongs in the retrieval service and the storage layer. Authenticate the caller, evaluate current authorization against the asset or a server-side policy reference, and propagate allowed record IDs to every lexical, vector, cache, reranker, preview, and generation step. Do not embed ACL text as a substitute for enforcement. NIST's zero trust guidance centers protection on resources rather than network location, a useful principle even for an offline intranet (NIST SP 800-207).

Encrypt source blobs, databases, vector indexes, model-cache policy material, and backups at rest with keys separated from the data where feasible. Encrypt local service connections when they cross a machine boundary. Minimize plaintext temp files produced by rendering, OCR, audio decoding, and LLM context assembly; give them short lifetimes and protected directories. Log record IDs, model versions, policy decision IDs, and outcome metrics, but do not log raw sensitive content or full prompts by default.

For an update, ingest a new asset_version, create replacement children and representations, then atomically switch the active version after quality checks. Existing answers can continue to resolve their cited historical version only if policy permits retention. For deletion, locate not just the source file but all derivatives: extracted text, OCR boxes, thumbnails, rendered pages, captions, transcripts, vectors, FTS rows, caches, queues, backups, and logs containing content. Tombstone first to block serving, purge synchronously where possible, and track delayed backup expiry. A delete statement is not automatically media sanitization; NIST defines sanitization as making target data infeasible to access for a defined effort and provides media-specific guidance (NIST SP 800-88 Rev. 2).

Evaluation before rollout

Build a private, permission-safe evaluation set from documents representative of the actual corpus. Each test item should include a natural question, an expected answer or acceptance rubric, one or more gold evidence anchors, modality label, access policy, and known hard negatives. Never use only synthetic text questions if the product promises image or audio retrieval.

Include at least these groups:

• Text questions: exact names, paraphrases, multi-section synthesis, and answers that require refusing because evidence is absent.
• Table questions: a value from a row and column, a comparison across rows, a header-dependent value, and a table embedded in a scan.
• Image and layout questions: chart trend, figure label, form field, diagram relationship, and a query where a caption alone is misleading.
• Audio questions: a quoted phrase, a topic stated across two segments, a timestamp lookup, noisy speech, and a question that must not infer speaker identity.
• Security and lifecycle questions: two near-identical assets with different ACLs, a deleted asset, an updated document, and a prompt that attempts to make the answer model disclose an excluded record.

Measure retrieval recall at several cutoffs for the gold anchor, not only whether the final prose seems plausible. Report lexical, text-vector, visual, audio, and fused recall separately. Then measure reranker uplift, citation-anchor accuracy, answer faithfulness to retrieved evidence, latency percentiles, cost in local compute time, and unauthorized-result rate, which should be zero. Review failures by modality and by parser version. A better answer model cannot compensate for an index that never retrieved the relevant table or audio segment.