We think a lot about the distribution of all real-world documents, particularly for real-world use cases. We think about the distribution in…

We think a lot about the distribution of all real-world documents, particularly for real-world use cases. We think about the distribution in terms of page length, field length, domain, and task type. Some documents have a single page but can have anywhere from 1-10 fields to 500-1000 fields to be extracted. Other documents are hundreds of pages long and can have 10k-100k fields. On long documents (e.g. a bankruptcy creditor matrix), commercial VLMs lose attention and create large swaths of missing or hallucinated values. The number of fields to extract per page also depends on the extraction task. 1. Needle-in-a-haystack: You find very specific information within a general document corpus. Generally 1-10 fields per page 2. Dense documents: typically extract from filled forms. Quite dense per page, though not super long in terms of total # pages. 3. Long-lists: Documents with both massive #'s of values per page (e.g. 100-1k fields per page) and massive numbers of pages. We'd love for you to explore our overall data taxonomy within ExtractBench, our comprehensive benchmark for real-world document extraction tasks. I love the diagram we show below in Figure 6, that offers one slice of the diverse set of data that we curate! Come check out the ArXiv paper: https://arxiv.org/pdf/2607.29677 ExtractBench: https://extractbench.ai/ Introducing ExtractBench, the most comprehensive benchmark for information extraction from complex enterprise documents. The latest models are pushing the frontier of coding and knowledge work, but surprisingly they still struggle on complex doc extraction tasks in production. A …

Source: Jerry Liu (X) | 2026-08-15

Related