AI / Technology
ExtractBench: schema-guided extraction that scores accuracy, completeness, grounding, and cost
Enterprise document extraction is not just about getting the right value once. It also has to find every record, point back to the source, and stay affordable when documents get long and messy.
Why this problem matters
A schema-guided extractor takes a document plus a user-defined schema and returns structured data. In enterprise work, the output must be useful for review, audit, and downstream systems, so the extracted values and their source evidence both matter.
The paper says existing benchmarks usually cover only part of this problem. Some focus on fixed field sets, some ignore grounding, and some do not measure cost or handle the mix of scans, handwriting, long lists, and dense tables that real enterprise documents contain.
- Real workflows need structured output plus evidence, not just a text answer.
- The benchmark is built to compare systems across document length, perception noise, table structure, and business domain.
Fixed-ontology extraction benchmarks
In fixed-ontology settings, the field list is predetermined. That works for narrow tasks, but it does not match enterprise workflows where users supply different schemas at inference time.
The paper treats these benchmarks as too rigid for schema-guided extraction. They cannot test how a system handles a new schema, a new document family, or a field layout the benchmark designer did not foresee.
Benchmark design and inputs
ExtractBench gives the model a document and a schema, then asks for structured JSON that matches that schema. The benchmark spans 370 documents, 4,869 pages, 67 document types, and 8 business domains: finance, energy, government, automotive, supply chain, healthcare, legal, and real estate.
The corpus mixes real born-digital documents, synthetic long lists, and scanned forms. The challenge tags cover task, perception, table structure, length, and domain so the same document can be examined from several angles.
- Input: document plus user-defined JSON Schema.
- Output: structured JSON with evidence, not just values.
How the benchmark is scored
The main value score is order-insensitive value F1 over JSON values. That means row order does not matter when the benchmark checks whether the right information was extracted.
Grounding is scored separately using word-level F1 and page-level F1. The paper uses human-verified boxes where available, so the benchmark can check whether a correct value is actually tied to the right source location.
- Value correctness and grounding are measured separately.
- Completeness matters because long lists can lose rows even when the early values are right.
How far the evidence really carries the claim
ExtractBench is a benchmark for schema-guided enterprise document extraction, and the paper claims it is the first to score value accuracy, record completeness at scale, grounding, and measured cost together.
The benchmark covers 4,869 pages from 370 enterprise documents across 8 business domains and 67 document types.
Evaluation uses order-insensitive value F1 over JSON values, plus word-level and page-level grounding F1 where source boxes are available.
Long documents expose truncation and missing-record failures.
Product interpretation
A production system built for this benchmark would need three things: schema-guided output, source evidence, and protection against truncating long lists. The paper’s comparison suggests that value accuracy alone is not enough for enterprise adoption.
The cautious product reading is that auditable extraction should treat completeness and grounding as first-class requirements, especially when documents are long, scanned, or table-heavy.
- If the product handles long records, it needs safeguards against early stopping.
- If reviewers must trust the output, the system should preserve provenance at the page or box level.
Research appendixTerms, sources, and open questions
Paper evidence
ExtractBench is a benchmark for schema-guided enterprise document extraction, and the paper claims it is the first to score value accuracy, record completeness at scale, grounding, and measured cost together.
Abstract; sourceLabel: arXiv metadata.
The benchmark covers 4,869 pages from 370 enterprise documents across 8 business domains and 67 document types.
Method notes; sourceLabel: arXiv HTML full text / Paper section 3.
Evaluation uses order-insensitive value F1 over JSON values, plus word-level and page-level grounding F1 where source boxes are available.
Method notes; Table 3 in Paper section 3 and sourceLabel: Paper section 3.
The curation pipeline combines frontier-model ensembles for real documents, programmatic generation for synthetic long lists, and human verification for scanned forms.
Method notes; sourceLabel: Paper section 3 and Paper section 7.
Commercial VLMs perform well on short documents but often truncate record lists on long ones.
Abstract and later notes; sourceLabel: arXiv metadata / Paper section 5.
Coding agents retain higher accuracy on these workloads but at much higher cost, while LlamaExtract Agentic Plus is reported as strongest overall and cheaper than coding-agent approaches.
Abstract and Paper section 5; sourceLabel: arXiv metadata / Paper section 5.
The benchmark taxonomy tags documents along task challenge, perception challenge, table structure, length, and business domain.
Paper section 6 and section 3 notes; sourceLabel: Paper section 6.
Limits and caveats
- Long documents expose truncation and missing-record failures.
- Grounding remains weak even for systems that return source boxes.
- Benchmark ground truth is labor-intensive and only covers the sampled corpus.
- Cost comparisons are reported, but implementation details for some systems are not fully specified in the notes.
- Section-level results do not fully explain which failure mode dominates in every domain or document type.
How it compares
What remains uncertain
The exact evaluation setup for grounding beyond word-level and page-level F1 is not fully specified in the provided notes.
The notes mention the metrics but not every scoring detail or threshold.
Check Table 3 and Appendix B / Section A.3 in the paper PDF.
The full experimental table for all models, datasets, and challenge slices is not included in the provided notes.
Only selected headline results and qualitative comparisons are available here.
Inspect the results section and appendix tables in the paper PDF.
The exact value behind 'word-level grounding F1 below 50%' is not stated in the provided notes.
The notes report the inequality, not the precise number.
Find the relevant results table for grounding metrics in the paper PDF.
How much each benchmark tag changes difficulty is not quantified in the provided notes.
Tags are described, but per-tag causal impact is not isolated here.
Review ablations or per-slice results in the paper's experiments section.
Generalization beyond the 370-document benchmark corpus remains unproven.
The notes describe benchmark composition but not external replication.
Look for external evaluations, released code, or follow-up benchmarks.
Glossary
- Schema-guided extraction
- Extracting data into a user-specified structure instead of a fixed field set.
- Value F1
- A score for how many extracted values are correct, with order ignored.
- Record completeness
- How well the system returns all rows or items that should be present, not just some of them.
- Grounding
- Linking an extracted value back to the source page, word, or box in the document.
- Page-level F1
- A grounding score that checks whether the value points to the correct source page.
- Word-level F1
- A stricter grounding score that checks whether the value points to the correct source words or box.
- VLM
- Vision-language model, a model that reads images and text together.
- Coding agent
- A system that uses code or programmatic steps to extract and validate document data.
Sources
Source grounding
Abstract: The paper positions ExtractBench as the first benchmark to jointly measure value accuracy, record completeness at scale, grounding, and cost. arXiv metadata
Abstract: LlamaExtract Agentic Plus ranks first on all three reported metrics in the abstract. arXiv metadata
Paper section 3: The benchmark spans 370 documents, 4,869 pages, 67 document types, and 8 business domains. Paper section 3
Paper section 3: Scoring uses order-insensitive value F1 and grounding F1 at word and page level. Paper section 3
Paper section 3: The curation pipeline uses frontier-model ensembles, programmatic generation, and human verification. Paper section 3 / Paper section 7
Paper section 5: Direct VLM extraction is inexpensive but often truncates long record lists, while coding agents are more robust but more expensive. Paper section 5
Paper section 6: The benchmark uses challenge axes for task, perception, table structure, length, and business domain. Paper section 6
Three key questions about this paper
What problem does ExtractBench: schema-guided extraction that scores accuracy, completeness, grounding, and cost address?
A schema-guided extractor takes a document plus a user-defined schema and returns structured data. In enterprise work, the output must be useful for review, audit, and downstream systems, so the extracted values and their source evidence both matter.
What evidence supports the main claim in ExtractBench: schema-guided extraction that scores accuracy, completeness, grounding, and cost?
ExtractBench is a benchmark for schema-guided enterprise document extraction, and the paper claims it is the first to score value accuracy, record completeness at scale, grounding, and measured cost together. Abstract; sourceLabel: arXiv metadata.
What limitation should readers know about ExtractBench: schema-guided extraction that scores accuracy, completeness, grounding, and cost?
Long documents expose truncation and missing-record failures.