NEW paper from Apple. Interesting idea: 'Attention to Mamba'. The paper introduces a two-stage recipe for cross-architecture distillation fr…

NEW paper from Apple. Interesting idea: "Attention to Mamba". The paper introduces a two-stage recipe for cross-architecture distillation from Transformers into Mamba. Naive distillation collapses teacher performance. Their trick: first distill the transformer into a linearized-attention student using a kernel adaptation, then transfer that student into a pure Mamba with no attention blocks. On a 1B model trained on 10B tokens, the Mamba student hits 14.11 perplexity against a 13.86 Pythia-1B teacher, nearly matching quality at linear-time inference cost. If you can reliably convert trained transformers into state-space models without retraining from scratch, the entire open-weights ecosystem becomes cheaper to serve at long context. This is the kind of quiet infrastructure work that decides which architectures actually get deployed in agent stacks. Paper: https://arxiv.org/abs/2604.14191 Learn to build effective AI agents in our academy: https://academy.dair.ai/

Related

• cool new paper on self-improving agents
• Another banger paper from Microsoft. Why it's a big deal: It teaches reasoning models to compress their own chain-of-thought mid-generation.…
• Small models are cheap to run, but expensive to adapt. The hard part is not only fine-tuning. It is the surrounding loop that involves colle…
• // Scaling Coding Agents via Atomic Skills // Most coding agents train end-to-end on full tasks like resolving GitHub issues. But complex so…

Source: DAIR.AI (X) | 2026-04-18