How can teams reliably validate structured JSON extracted by AI from long industrial catalogs?

Treat each extracted field as a claim

The right unit of validation is not the JSON document. It is an individual field associated with a specific product identity and a source location. A product record becomes trustworthy only when the system can answer five questions for each material value:

1. Which exact catalog release and file produced this value?
2. Which page, visual region, table cell, heading, caption, or footnote supports it?
3. Which product family, model code, option code, operating condition, and unit does it apply to?
4. What transformations converted the printed value into the normalized value?
5. Did any source in the same catalog or a newer approved release contradict it?

This approach avoids a dangerous shorthand: “valid JSON” is a syntactic result. A JSON parser can confirm braces, types, required keys, and enum membership. It cannot establish that 500 belongs to the requested model rather than the next row, that mm/s was not confused with mm/min, or that a footnote restricts the value. JSON Schema supports type, enum, conditional, and composition rules that make a data contract clearer, but semantic verification must happen outside the schema (JSON Schema reference).

A layered pipeline for catalog validation

flowchart LR
  A[Immutable catalog PDF and manifest] --> B[Native text OCR and page images]
  B --> C[Page regions tables headings footnotes]
  C --> D[Schema-constrained extraction]
  D --> E[Normalize units model codes variants]
  E --> F[Deterministic and cross-field validation]
  F --> G[Independent evidence verification]
  G --> H[Accept review abstain or conflict]
  H --> I[Golden-set evaluation and monitored release]
  I --> D

The ordering matters. Do not ask one model to read an entire catalog, infer all table structure, select a product identity, normalize units, and decide whether it is correct in one opaque step. Preserve intermediate artifacts so a reviewer can see whether the root cause was OCR, reading order, table segmentation, extraction, normalization, or a rule.

Preserve the original and create a document manifest

Before extraction, store the original file unchanged and record its supplier, catalog title, publication or revision date if shown, acquisition time, locale, page count, URL or supplier reference, and a cryptographic file hash. Treat a revised PDF as a new source, even when the filename is unchanged.

Create a catalog_version_id from the file hash and use it in every source reference. Keep page images at a stable DPI as well as native text. If a supplier provides an authorized, versioned product feed or configuration export, use that as the preferred structured source and retain the PDF as human-readable evidence. Do not scrape undocumented endpoints or silently combine different supplier releases without a documented source-precedence rule.

Make parsing and segmentation independently reviewable

Industrial PDFs are often born digital, scanned, or mixed. The same page can contain a dense table, a figure, a model-code grammar, a sidebar, and a footnote that changes the meaning of the table. Parse the document into layers rather than treating extracted plain text as the source of truth.

Keep two document representations

1. Text and structure layer: native text runs, PDF coordinates, headings, tagged structure where available, links, and candidate table cells.
2. Visual layer: rendered page image plus cropped regions for tables, diagrams, captions, footnotes, and model-code blocks.

Use the text layer for search, exact quotations, and machine checks. Use the visual layer whenever an extractor reports a table, diagram, superscript, footnote marker, or reading-order ambiguity. Tagged PDFs can help, but they are not proof of table structure. Adobe notes that automatic tagging can misinterpret complex layouts, closely spaced columns, irregular alignment, and borderless tables, producing incorrectly combined or out-of-sequence elements (Adobe Acrobat guidance).

Build page packets, not one giant prompt

Make a packet the smallest source unit that lets a reviewer reconstruct meaning. A useful table packet contains the page image crop, table caption, repeated header rows, row and column coordinates, surrounding heading, associated footnotes, and the prior or next page when the table continues. A prose packet contains a heading path, paragraphs, page coordinates, and any referenced note or figure.

The packet must not lose relationships at a page boundary. If a table’s header appears on page 37 and its continuation rows on page 38, extraction from page 38 alone is not acceptable. Likewise, include the footnote block when a table marker appears in any selected row.

Assign stable IDs such as cat_9f31:p038:table_02:r07:c04 and keep the coordinate system documented. A bounding box without its page image, page number convention, and rendering scale is weak evidence.

Use a schema for shape, not for truth

Schema-constrained extraction is worth using because it eliminates many downstream handling errors: missing required fields, accidental free-form prose, unexpected types, and undeclared keys. With OpenAI, Structured Outputs via json_schema is intended to adhere to the supplied schema, whereas JSON mode only ensures valid JSON and does not guarantee a particular schema (OpenAI Structured Outputs). Other providers offer comparable mechanisms, but the rest of this design does not depend on one.

The schema should make uncertainty explicit. Do not encode an unknown speed as 0, an empty string, or an invented default. Require a status and source object for every material attribute. In strict OpenAI function schemas, all properties are required and optional values can be represented with null; objects must also disallow additional properties. Check the current provider’s supported JSON Schema subset before adopting a schema design (OpenAI function strict mode).

Compact schema fragment

This provider-neutral fragment uses a list of attributes rather than arbitrary keys. It keeps an original rendering, a normalized value, a status, and a source span together.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["catalog_version_id", "models"],
  "properties": {
    "catalog_version_id": { "type": "string" },
    "models": {
      "type": "array",
      "items": {
        "type": "object",
        "additionalProperties": false,
        "required": ["model_code", "attributes"],
        "properties": {
          "model_code": { "type": "string" },
          "attributes": {
            "type": "array",
            "items": {
              "type": "object",
              "additionalProperties": false,
              "required": [
                "key", "raw_value", "normalized_value", "normalized_unit",
                "verification_status", "source"
              ],
              "properties": {
                "key": { "type": "string" },
                "raw_value": { "type": ["string", "null"] },
                "normalized_value": { "type": ["number", "null"] },
                "normalized_unit": { "type": ["string", "null"] },
                "verification_status": {
                  "type": "string",
                  "enum": ["verified", "needs_review", "unsupported", "conflicting"]
                },
                "source": {
                  "type": "object",
                  "additionalProperties": false,
                  "required": ["page", "bbox", "quoted_text", "packet_id"],
                  "properties": {
                    "page": { "type": "integer", "minimum": 1 },
                    "bbox": { "type": "array", "items": { "type": "number" } },
                    "quoted_text": { "type": "string" },
                    "packet_id": { "type": "string" }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

In a production schema, add source_document_sha256, the page-image checksum, source type such as table_cell or footnote, row and column labels, parser version, extraction-run ID, and a field-specific precision or tolerance rule. Preserve raw_value even after normalization. It gives a human enough evidence to correct a unit parser or later change a conversion policy without rerunning the entire catalog.

Normalize without erasing meaning

Normalization makes data searchable and comparable. It can also create errors when it hides the condition under which a value applies. Keep a three-part representation for quantities:

• Printed value: exact string from the source, such as 500 mm/s or 0.2 to 0.8 MPa.
• Parsed quantity: numerical value or lower and upper bounds plus printed unit.
• Canonical quantity: value and unit after a controlled conversion, plus conversion method and rounding policy.

Store ranges as ranges, not a single maximum. Store tolerances separately from nominal values. Preserve qualifiers such as at 25 °C, vertical mounting, with option H, and except high-load type; they are conditions, not decoration.

Units require deterministic conversion, controlled rounding, and an explicit unit on every value. NIST advises that a complete quantity declaration include both value and unit, and that software used for critical conversion be validated with correct factors and appropriate rounding (NIST metrication practices). Do not convert torque, pressure, mass, force, length, or speed with model-generated arithmetic when a controlled unit library can perform the operation and retain the conversion audit trail.

Model codes and variants need their own layer

Do not flatten a product family and every option into one record. Use a hierarchy such as family, base model, option code, configuration, and condition. Record the source that establishes each relationship.

For example, if LX20-200-H is formed from a family, 200 mm stroke, and high-load option H, preserve those components and their source spans. A speed limit in a high-load footnote belongs to the H configuration, not automatically to every LX20-200 model. Model-code parsing should produce a candidate structure, then deterministic grammar and catalog rules should check it.

Validate in layers

Each layer should return a machine-readable finding with severity, field ID, catalog version, evidence IDs, rule version, and recommended disposition. This makes a failed field explainable instead of merely “low confidence.”

Deterministic checks should carry the heaviest load

Use rules that do not require a model wherever the catalog grammar or engineering data model permits it:

• Validate model-code syntax and option positions against documented grammar.
• Verify that every speed, stroke, pressure, voltage, and torque value has an allowed dimension and unit.
• Reject impossible ranges, inverted bounds, negative values where not permitted, and excess precision beyond the printed source.
• Require a footnote reference when the source cell contains a marker. Resolve the marker before the field can be verified.
• Check that a selected variant contains all required option codes and that its attribute values satisfy variant-specific rules.
• Require explicit, authoritative compatibility evidence for a compatibility edge. Do not infer that two components are compatible merely because their dimensions or connector names look similar.
• Compare values from repeated tables or annexes only after confirming that product identity, condition, locale, and revision match.

Cross-field rules are where many plausible extractions fail. A single value can be syntactically valid and physically plausible while wrong for the selected configuration. Put these rules in versioned code or a reviewable declarative ruleset, not only in a prompt.

Require independent evidence verification

Model consensus is a useful signal for triage, but it is not validation. Two models can use the same OCR text, share a layout blind spot, or make the same attractive assumption. The verifier should be independent in its evidence search and decision inputs, not merely a different model name.

Use this pattern:

1. The extractor emits a proposed field, product identity, source packet, and raw evidence.
2. The verifier receives the catalog version, product identity, field definition, and permitted source corpus, but not the extractor’s proposed value or citation at first.
3. The verifier searches page and table packets, returns the best direct evidence, parses it independently, and records its own source span.
4. A deterministic reconciler compares product identity, normalized value, unit, condition, and source version.
5. Agreement with direct evidence permits the field to move to verified. Disagreement, missing evidence, or contradictory evidence moves it to review or conflict.

Blind verification is especially useful for values that would influence component selection, compatibility, load limits, electrical ratings, or safety margins. A verifier may use an LLM for evidence finding, but the acceptance decision must still require literal source support and applicable rules.

Confidence is a routing signal, not proof

Do not publish a model’s self-reported 0.93 confidence as if it were a probability of correctness. Build confidence from observable conditions, for example:

Define acceptance as policy, not a magic threshold. A baseline policy might automatically accept only fields with an exact source span, schema pass, unit parse, product-variant match, cross-field pass, no unresolved footnote, no conflict, and a verified catalog version. All other material fields go to review. The severity of a wrong field should set the policy. A decorative dimension and a value used to select a load-bearing component should not share a threshold.

Example with units variants footnotes and a conflict

This is a hypothetical linear-actuator catalog. It illustrates the process, not a real manufacturer specification.

The catalog’s table on page 42 lists base family LX20, 200 mm stroke, and maximum speed 500 mm/s. A footnote marked * says “For high-load option H, maximum speed is 300 mm/s.” The model-code guide on page 12 identifies LX20-200-H as the 200 mm, high-load configuration. A compatibility chart on page 61 lists driver D-24 for LX20, while a later page 64 footnote says that high-load option H requires D-48.

The extractor should create two conditional records, not one flattened field:

{
  "model_code": "LX20-200",
  "attributes": [
    {
      "key": "max_speed",
      "raw_value": "500 mm/s",
      "normalized_value": 500,
      "normalized_unit": "mm/s",
      "verification_status": "verified"
    }
  ]
}

{
  "model_code": "LX20-200-H",
  "attributes": [
    {
      "key": "max_speed",
      "raw_value": "300 mm/s",
      "normalized_value": 300,
      "normalized_unit": "mm/s",
      "verification_status": "verified"
    },
    {
      "key": "compatible_driver",
      "raw_value": "D-24 on p.61; D-48 required for H on p.64",
      "normalized_value": null,
      "normalized_unit": null,
      "verification_status": "conflicting"
    }
  ]
}

The action is not to choose D-24 or D-48 by majority vote or by which page appears first. Preserve both source spans, identify whether page 64 is a later correction or a conditional exception, and route the compatibility relationship to a reviewer or manufacturer confirmation. The takeaway is that a value can be perfectly valid JSON and still be operationally unsafe if its condition, footnote, or source conflict is lost.

Handle duplicates and conflicts as first-class data

Duplicates occur when a catalog repeats a specification in a family table, an ordering table, a drawing, a feature list, and a translated annex. Start by generating a candidate duplicate key from manufacturer, catalog version, product family, fully parsed model code, attribute key, condition, normalized value, and normalized unit. Use text similarity or embeddings only to propose candidates for reconciliation, never as the final merge decision.

Merge only after deterministic identity checks pass. Keep aliases and every evidence span. If two values differ, do not average them, choose the most common one, or overwrite the old value without a documented source-precedence rule. Mark the field conflicting until a reviewer resolves the source versions, locale, operating condition, or supplier correction. A catalog update should expire derived records from the earlier version rather than mutate their provenance in place.

Review the right records and sample the rest

Human review should not be a random final inspection of whatever happened to be extracted. Use it to resolve ambiguity and to estimate the false-accept rate of the automated system.

Send 100 percent of these cases to review:

• Fields used in compatibility, sizing, electrical, motion, pressure, load, temperature, or safety-related decisions.
• Conflicts, unresolved footnotes, no-source fields, unknown units, OCR uncertainty, diagrams, multi-page tables, and unfamiliar templates.
• New supplier, new language, new catalog layout, parser revision, schema revision, or model-code grammar.
• Any field where independent verifier evidence does not match the extractor.

For automatically accepted lower-risk fields, draw a stratified random sample. Stratify by supplier, catalog template, document quality, table complexity, field type, language, and confidence band. Reviewers should label both the field value and its evidence applicability. Report the observed false-accept rate with uncertainty, not only the proportion accepted automatically. If a sample reveals a material failure class, stop auto-acceptance for that slice until the rule or extractor is fixed.

Measure the dataset rather than the model

Create a golden dataset from catalog packets that qualified engineers or trained technical reviewers have labeled at field and evidence-span level. Include intentionally difficult cases: split tables, repeated headers, footnotes, diagrams, variants, dual units, missing data, contradictions, and no-answer conditions. Lock one holdout set for release decisions and add reviewed production failures to future dataset versions.

Track metrics at field level and by risk slice:

Break every metric down by catalog quality and task type. An overall 99 percent field score can conceal failure on scanned Japanese tables, dense multi-page tables, high-load variants, or electrical compatibility charts. NIST’s AI Risk Management Framework similarly calls for documented test sets and methods, evaluation under conditions similar to deployment, and monitoring of system behavior in production (NIST AI RMF).

Use a durable error taxonomy

Every failed validation or human correction should have one primary cause and optional secondary causes. Suggested primary classes are:

The taxonomy turns corrections into engineering work. It tells the team whether to repair OCR, table segmentation, schema design, data rules, retrieval, source management, or human workflow.

Batch for cost and latency without losing context

Long catalogs require batching, but batching should follow document structure rather than arbitrary token counts. First perform cheap, deterministic work such as file hashing, page rendering, native text extraction, OCR detection, language identification, heading detection, and table-region proposal. Then send only evidence packets that need interpretation to a model.

Use a two-pass schedule:

1. Run independent page or table-packet extraction in parallel. Include packet metadata, table headers, footnotes, and a constrained schema. Cache packet results by catalog hash and parser version.
2. Run a reconciliation pass for product families and variants. It receives normalized candidates plus source IDs, not the full catalog. It resolves duplicate candidates, runs cross-field rules, and emits only reviewable discrepancies.

Keep one packet within the deployed context budget, but do not cut a logical table, its header, or its footnotes merely to fit a token target. If a complete context is too large, use an explicit continuation relationship and route the result to review rather than assuming omitted context is irrelevant. Measure tokens, request count, model latency, OCR latency, retry rate, and human-review minutes per accepted field and per catalog. This identifies whether cost is coming from vision processing, repeated context, failed retries, or a low-quality source template.

Handle incomplete, refusal, timeout, and schema-validation failure as explicit processing states. Do not retry a partial output by appending it to another prompt and treating the combined text as evidence. The official Structured Outputs guide notes that applications still need to handle refusals and incomplete responses such as outputs limited by maximum tokens (OpenAI Structured Outputs).

Acceptance policy and release gates

Put the publishing rule in code and make it auditable. A field eligible for automatic publication should meet all of these requirements:

• The catalog version, product identity, and field definition are known.
• The field passes JSON Schema and type validation.
• The raw value, unit, range, qualifier, and source span are present.
• Unit conversion and rounding pass controlled tests where normalization occurred.
• Variant, model-code, and cross-field rules pass.
• Independent evidence verification found matching direct support.
• No unresolved footnote, duplicate collision, source conflict, or high-risk policy rule applies.

Everything else remains in a non-published queue or is published only with a visible needs_review, unsupported, or conflicting status, depending on the product’s user interface and risk policy. Never turn a nonverified value into a silent default to make a search index look complete.

Before deploying a change to OCR, segmentation, extraction prompt, model, schema, unit library, ruleset, or reconciliation logic, rerun the golden dataset and compare the same catalog snapshots. Block release if evidence precision, variant association, conflict detection, or any critical slice regresses beyond its approved tolerance. Review all new critical failures, store their packets and human adjudications, and add representative cases to the next golden-set version.

Failure modes and viable alternatives

The most useful alternative is often to reduce extraction rather than improve it. Ask suppliers for versioned product-master feeds, configuration rules, CAD metadata, selection-tool exports, or officially licensed datasets. Where that is not possible, scope the PDF pipeline to the exact attributes and product families needed for a defined engineering workflow. This makes the rules, evidence requirements, and review burden manageable.

Practical implementation checklist

1. Define the engineering decisions the dataset may inform and classify fields by consequence.
2. Prefer authorized, versioned structured supplier data where available. Preserve the PDF as a source artifact.
3. Store immutable catalog files, hashes, page images, native text, OCR output, and parser versions.
4. Segment by pages, headings, tables, cells, diagrams, captions, and footnotes. Create stable packet IDs and retain coordinates.
5. Design a schema with raw values, normalized values, units, conditions, verification state, and source spans. Use null for unknown values.
6. Extract packets with schema-constrained output, then parse model codes and normalize with controlled conversion code.
7. Run deterministic field, unit, grammar, footnote, variant, duplicate, and compatibility-evidence rules.
8. Independently verify material claims from source packets without revealing the first extractor’s proposed value.
9. Send conflicts, unsupported fields, unfamiliar layouts, high-consequence fields, and random accepted samples to trained reviewers.
10. Maintain a golden dataset, track field and evidence metrics by slice, gate changes, and add adjudicated failures to future tests.

Limits and engineering boundaries

This pipeline increases traceability and reduces silent error. It does not turn a catalog PDF into an engineering authority beyond what the manufacturer states, and it does not replace a qualified engineer’s design review, applicable standards, site conditions, or supplier confirmation. Compatibility may depend on configurations, regional versions, lifecycle state, wiring, firmware, mechanical load, operating environment, and safety requirements that the catalog does not capture.

Use explicit human escalation for safety-critical, legal, contractual, medical, financial, or regulated decisions. Confirm licensing, redistribution rights, confidentiality, and retention rules before storing supplier documents or sharing derived data. A source span makes an assertion auditable; it does not itself grant a right to reuse the underlying catalog content.