Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction

arXiv:2608.28078v1 Announce Type: new Abstract: Vision foundation backbones provide strong representations for dense prediction, yet a single shared feature still needs to support tasks with different, image-dependent adaptation requirements. We propose MemMTL, a multi-task dense prediction framework that estimates a compact task state from global visual context and refines it through a learnable task-state prototype memory. The refined state is converted into task-conditioned expert logits and combined with token-level logits before sparse top-k selection over a local expert bank shared by all tasks. A separate task-agnostic residual bank provides a common adaptation path, and both paths are added once to the backbone feature before task-specific prediction. We specify a matched evaluation protocol on NYUD-v2 and PASCAL-Context with SAM 3 and ViT-L backbones to measure predictive quality, computational cost, and the contributions of task-state conditioning, prototype retrieval, and sparse routing. The numerical record in the present working draft predates this canonical implementation and must be regenerated before it can support empirical claims.

Related

• Learning to Orchestrate Vision Foundation Models for Multi-Task Dense Prediction
• EdgeCrafter: Compact ViTs for Edge Dense Prediction via Task-Specialized Distillation
• Selective, Regularized, and Calibrated: Harnessing Vision Foundation Models for Cross-Domain Few-Shot Semantic Segmentation
• A3-FPN: Asymptotic Content-Aware Pyramid Attention Network for Dense Visual Prediction

Source: arXiv cs.CV | 2026-08-31