Identify, Locate, Link: End-to-End Key-Value Extraction from Document Images

arXiv:2608.20868v1 Announce Type: cross Abstract: Document processing pipelines traditionally cascade optical character recognition (OCR) engines with downstream models for structured information extraction, leading to multi-stage error propagation. We fine-tune SmolDocling, a compact 256M-parameter vision-language model (VLM), to perform end-to-end key-value extraction directly from document images, jointly solving identification, localization, and association in a single pass without OCR preprocessing. We extend DocTags with specialized key, value, region, and link tags, enabling many-to-many relationships in a unified output sequence. To address data limitations, we design an augmentation pipeline combining synthetic form filling and graph-based crops that preserve complete key-value subgraphs. We further introduce a layout-aware evaluation framework extending text matching with spatial bounding box verification. On FUNSD, XFUND, and a large-scale private dataset, our model outperforms larger zero-shot VLM baselines under layout-aware evaluation, while being 27 times smaller than Qwen2.5-VL (7B) and over 5 times faster at inference. The model weights will be released publicly after publication.

Related

• CC-OCR V2: Benchmarking Large Multimodal Models for Literacy in Real-world Document Processing
• Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction
• RaV-IDP: A Reconstruction-as-Validation Framework for Faithful Intelligent Document Processing
• UNIKIE-BENCH: Benchmarking Large Multimodal Models for Key Information Extraction in Visual Documents
• Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images

Source: arXiv cs.CL | 2026-08-24