Safety
TDDN: Text-aligned Diffused DINO Network for Puzzle Understanding
arXiv:2609.07937v1 Announce Type: new Abstract: Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs
arXiv:2609.07937v1 Announce Type: new Abstract: Structured visual reasoning, such as image puzzles, demands fine-grained visual perception, an ability current Vision Language Models (VLMs) lack. VLMs built on CLIP-based ViT backbones trade fine-grained detail for high-level semantics, and we show this loss propagates downstream. To recover it, we fuse DINOv3 and CleanDIFT representations into a perception encoder (DiffusedDINO) and align it with RoBERTa-L, yielding a text-aligned model TDDN that preserves this perceptual advantage: with frozen backbones and only sim590K alignment pairs, TDDN matches CLIP on image-text retrieval, surpassing it on three of four settings. It does so while more than tripling CLIP's dense-prediction accuracy (ADE20K 5.20 o 18.11 mIoU, COCO-Stuff 7.35 o 24.44), despite CLIP's massive training corpus. TDDN leads on segmentation benchmarks among general-purpose contrastive encoders, including SigLIP,2. We further introduce Puzzle Perception, a segmentation and visual question answering dataset that probes fine-grained spatial understanding, on which TDDN doubles CLIP's segmentation accuracy (11.04 o 22.51 mIoU).
Related
- Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models
- Cosine Misleads: Auxiliary Losses Reshape Vision Language Models, Not Their Latents
- Improving Reasoning in Vision-Language Models via Perception Verified Self-Training
Source: arXiv cs.CV | 2026-09-09