SignDino: Self-Supervised Sign Language Representation Learning via Temporal-Axis Self-Distillation

arXiv:2609.06296v1 Announce Type: new Abstract: Self-supervised sign language representation learning must model two properties not central to natural-image SSL: signs are produced by a small set of anatomically distinct articulators, and their meaning depends on the temporal organisation of those articulators. We introduce SignDino, a self-supervised sign-video encoder that moves the DINOv3 student--teacher recipe from the spatial domain of image crops to the temporal domain of tracked sign streams. Each video is decomposed into left-hand, right-hand, and face streams by a detector-first YOLOv8n+ByteTrack pipeline. A frozen DINOv3 ViT-B/16 embeds each per-frame anatomical crop, while lightweight temporal Transformers, not the image backbone, form the student and EMA teacher. They are trained by temporal DINO self-distillation, frame-level masked-token prediction in the style of iBOT, KoLeo feature spreading, and Gram anchoring of the frame-to-frame similarity structure. This design keeps strong image-level visual primitives fixed and learns only how articulator states evolve across time. We evaluate on sign-to-English translation, isolated sign recognition, and fingerspelling detection benchmarks. Across these tasks, SignDino provides a strong public self-supervised representation and shows competitive or state-of-the-art performance under matched downstream evaluation.

Related

• Temporal Self-Distillation: Learning Visual State Tracking in Videos Without Supervision
• Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
• SCAR: Self-Supervised Continuous Action Representation Learning
• SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition
• SIGNET: Motion-Level Knowledge Transfer for Cross-Language Sign Language Translation

Source: arXiv cs.CV | 2026-09-09