Essential cookies keep Instavar working. Optional analytics help us understand how the site is used. Cookie Policy
Manage Cookie Preferences
Service reliability telemetry, including Sentry error monitoring and Vercel Speed Insights, stays enabled so we can secure the product and diagnose failures.
Failure ownerTest layout and table claims with narrower evidence.
Table OCR needs numeric-token, row, column, and cell-position checks because visually plausible Markdown can still move or change values.
Do this first
Check numeric tokens and cell positions against the source instead of accepting visually plausible Markdown.
Keep this boundary in view
The fixtures do not establish NuExtract3 as a universal table OCR winner across layouts or domains.
Inspect the evidence and what would change the conclusion
Evidence layer
Public-safe NuExtract3 and FireRed table comparison
Observed
28 Jul 2026
Sample
1,040 scored cells in a bounded fixture set
What supports it
A bounded public-safe comparison scored 1,040 cells and exposed differences that visual inspection alone could miss.
Change our mind if
Visual review alone catches every consequential number and cell-structure error on independent table sets.
Claim OCR-TABLE-NUEXTRACT3-V1 / v1, checked 6 Aug 2026
A table can look correct in Markdown while its rows, columns, or numbers are wrong. That makes table OCR a validation problem, not only a transcription problem.
We tested NuExtract3 and FireRed-OCR on the same 40 public-safe, scan-like table crops derived from US Bureau of Labor Statistics data. NuExtract3 preserved all 1,040 truth cells in their expected positions. FireRed remained the better general OCR workflow, but four outputs inserted a title row into the HTML table and shifted the cell alignment.
The result supports a specialist fallback pattern. It does not establish that NuExtract3 is the best table OCR model in general.
The short version
Both models completed all 40 table crops with no failed or empty outputs.
NuExtract3 strict numeric F1 was 0.963462; FireRed scored 0.957854.
NuExtract3 matched 1,040/1,040 truth cells and all 480/480 numeric cells after normalization.
FireRed matched 1,036 cells and 432/480 normalized numeric cells in their expected positions.
The main FireRed structural failure was placing a title inside the table, shifting later row and column positions.
A validator routed 23 risky crops to NuExtract3 and kept FireRed for 17 crops.
The fallback pipeline matched NuExtract3's aggregate score on this sample, but it reused existing outputs and therefore did not measure runtime savings.
What NuExtract3 is
NuExtract3 is a 4B vision-language model for document understanding. Its model card describes structured extraction from text or images, document-to-Markdown conversion, multilingual documents, and HTML table output. The weights are published under Apache-2.0.
That is upstream capability information. Our local result covers only public-safe table crops on one RTX 3090 Ti workflow. It is not evidence for receipts, contracts, forms, handwriting, or arbitrary financial reports.
Character error rate can tell you that text differs from a reference. It cannot tell you whether the value 7.4 stayed in the correct year and series column.
A useful table evaluation needs at least four views:
Check
Failure it catches
Local test / OCR-TABLE-NUEXTRACT3-V1
Test this claim on your documents.
Copy a read-only experiment brief that asks your coding agent to preserve raw evidence, test a negative case, and return a verdict that can disagree with this page.
Use Codex to test this bounded OCR claim in my local environment.
Source: https://instavar.com/research/ocr/nuextract3-table-ocr-number-cell-validation
Claim ID: OCR-TABLE-NUEXTRACT3-V1
Claim version: v1, checked 6 Aug 2026
Claim: Table OCR needs numeric-token, row, column, and cell-position checks because visually plausible Markdown can still move or change values.
Observed evidence: A bounded public-safe comparison scored 1,040 cells and exposed differences that visual inspection alone could miss.
Evidence layer: Public-safe NuExtract3 and FireRed table comparison.
Sample: 1,040 scored cells in a bounded fixture set.
Boundary: The fixtures do not establish NuExtract3 as a universal table OCR winner across layouts or domains.
Potential falsifier: Visual review alone catches every consequential number and cell-structure error on independent table sets.
Safety and privacy:
- Work read-only by default and keep all documents and outputs local.
- Do not upload documents, reveal credentials, or send document contents to an external service.
- Do not install software, download large models, start paid jobs, or change my environment without asking first.
- If the available tools cannot run the comparison safely, stop with a concrete setup plan instead of pretending the claim was tested.
Test method:
1. Inspect the available OCR workflows and record their exact versions, runtime, hardware, and configuration.
2. Help me choose a small representative sample. Include a known difficult page and a negative control with expected text or structure when available.
3. Before running anything, state the primary failure we are testing and the measurement that would expose it.
4. Preserve raw outputs. Compare text fidelity, structure, blank-page behavior, grounding correctness, runtime, and cleanup effort only where those properties can actually be measured.
5. Treat anchor count as an unverified quantity until sampled anchors are checked against the page.
6. Report the result as support, contradict, mixed, or inconclusive. Explain what observation produced that verdict.
7. Name the smallest follow-up test that could change the verdict.
Strict numeric-token precision and recall
missing or invented numbers anywhere in the output
Row and column counts
inserted, removed, or merged table structure
Cell-position matching
values shifted into the wrong coordinates
Numeric-cell normalized accuracy
wrong numbers after harmless formatting differences are removed
These scores answer different questions. Keep them separate rather than blending them into one table-accuracy number.
The public-safe test
The sample contained 40 unique table crops rendered from official BLS public data. It covered four data series across 2015 through 2024. Every row had explicit cell truth, numeric-token truth, legal metadata, and public-reporting status.
The crops were intentionally scan-like rather than pristine HTML. Both models received the same images and were scored with the same code.
This was a table-crop diagnostic:
no Paddle detector was run
no full-page PDF benchmark was run
no chemistry documents were included
no production routing change was made
Results on 40 table crops
Metric
FireRed-OCR
NuExtract3
Successful outputs
40/40
40/40
Failed or empty outputs
0
0
Truth numeric tokens
520
520
Matched numeric tokens
500
501
Missed numeric tokens
20
19
Hallucinated numeric tokens
24
19
Strict numeric precision
0.954198
0.963462
Strict numeric recall
0.961538
0.963462
Strict numeric F1
0.957854
0.963462
Truth cells
1,040
1,040
Matched cells
1,036
1,040
Extra cells
8
0
Row-count matches
36/40
40/40
Column-count matches
36/40
40/40
Cell normalized accuracy
0.900000
1.000000
Numeric-cell normalized accuracy
0.900000
1.000000
The strict numeric F1 difference was small. The cell-position difference was operationally larger.
The failure a visual review can miss
In four FireRed outputs, a title or caption was placed inside the HTML table. The visible table values still appeared afterward, so a quick visual scan could call the output acceptable.
Structurally, every subsequent row was displaced. A downstream CSV import, database loader, or calculation would associate values with the wrong row coordinates.
This is why table evaluation cannot stop at "the numbers are present." It must ask:
Are the expected rows present?
Are the expected columns present?
Did non-table text enter the table?
Is each number in the expected cell?
NuExtract3 kept all 1,040 cells aligned on this sample. FireRed's four title-row insertions reduced normalized cell and numeric-cell accuracy to 0.9.
Why the strict numeric score tells a different story
The mismatch audit reviewed the 30 highest-severity candidates:
Primary cause
Count
Title or caption numeric error
26
Row or column shift
4
Some title years were misread outside the table body. Those errors correctly reduce strict numeric-token precision or recall, but they do not necessarily corrupt table cells.
That explains the two score views:
strict numeric F1 checks numeric continuity across the whole output
numeric-cell accuracy checks values inside the explicit table grid
Neither should replace the other. If titles and captions matter to the application, use both. If only the table body drives calculations, cell-position scoring should control the main acceptance gate.
A validate-then-fallback pipeline
We tested a simple policy that kept FireRed as the general OCR lane and selected NuExtract3 only when validation failed.
The validator requested fallback if any of these occurred:
row-count mismatch
column-count mismatch
numeric-cell normalized accuracy below 0.99
strict numeric F1 below 0.99
no parseable table cells
empty output
missing cell-level score
On the 40 crops:
FireRed was accepted for 17 crops
23 crops were assigned to NuExtract3
no crop lacked a usable output
the selected outputs matched NuExtract3's aggregate scores
Metric
FireRed alone
NuExtract3 alone
Validator-selected output
Strict numeric F1
0.957854
0.963462
0.963462
Cell normalized accuracy
0.900000
1.000000
1.000000
Numeric-cell normalized accuracy
0.900000
1.000000
1.000000
Row-count matches
36/40
40/40
40/40
Column-count matches
36/40
40/40
40/40
This result demonstrates selection logic, not performance savings. The pipeline reused outputs that had already been generated by both models. It did not measure the latency or GPU cost of invoking NuExtract3 only after a FireRed validation failure.
The validator was deliberately conservative
All 23 fallbacks were triggered by strict numeric F1 below 0.99. Only four also had row, column, and numeric-cell failures.
That means the policy can over-route when a title year is wrong but the table body is intact. A production validator should offer at least two modes:
Table-cell-critical mode
Use when calculations or database imports depend on the grid:
row and column counts
parseable table cells
cell-position matching
numeric-cell accuracy
Broad numeric-continuity mode
Use when titles, dates, captions, footnotes, and table cells all matter:
all table-cell-critical checks
strict numeric-token precision and recall across the complete output
The acceptance contract should reflect the downstream risk, not an arbitrary universal threshold.
A practical table OCR checklist
Before accepting extracted table data:
Separate page detection, crop selection, transcription, and parsing.
Store the raw model output before normalization.
Confirm that a table was actually parsed.
Compare expected and predicted row and column counts.
Keep captions and titles outside the table grid.
Normalize harmless formatting differences before cell comparison.
Compare numeric values by cell position.
Count missing and hallucinated numeric tokens separately.
Escalate structural failures to a specialist model or human review.
Preserve the validation result with the extracted data.
For unlabelled production documents, some checks need business rules rather than ground truth. Examples include expected column names, numeric ranges, totals, date order, checksum relationships, or reconciliation against a source system.
When to use NuExtract3
NuExtract3 is worth a focused test when:
table structure is more important than general page transcription
documents contain forms, invoices, receipts, contracts, or repeated schemas
HTML or structured extraction is useful downstream
row shifts and numeric-cell errors are expensive
a general OCR model already handles ordinary pages
Our result does not justify replacing FireRed for every page. It supports treating NuExtract3 as a table specialist behind an explicit validation gate.
Limitations
The sample contains 40 BLS-derived scan-like table crops, not complete PDFs.
It represents four series and ten years, not arbitrary table diversity.
The comparison does not cover handwriting, merged cells, multi-page tables, rotated pages, or multilingual tables.
The fallback experiment reused existing outputs and did not measure incremental latency.
NuExtract3 required a local vLLM 0.17.1 path and tokenizer override in this experiment.
The result should not be blended with DocLayNet because the DocLayNet slice lacked explicit table text and cell-level numeric truth.
Final verdict
NuExtract3 was the stronger table-cell specialist on this bounded BLS sample. FireRed remained the broader general OCR workflow.
The important result was not the small strict numeric F1 gap. It was the structural difference: NuExtract3 preserved all expected rows, columns, and cell positions, while four FireRed outputs shifted the table by inserting title content.
Use model routing only after defining what "correct table extraction" means for the downstream system. If a wrong number in the right-looking table can trigger a payment, calculation, or report, visual plausibility is not an acceptance test.