Model Releases
Mapping the Post-Transformer Architecture Landscape in 2026 [D]
Went down the rabbit hole to map the different Post-Transformer architectures. Various directions include Dragon Hatchling (BDH), Mamba, RWKV, xLSTM, Titans, and a bunch of hybrid models coming from d
Went down the rabbit hole to map the different Post-Transformer architectures. Various directions include Dragon Hatchling (BDH), Mamba, RWKV, xLSTM, Titans, and a bunch of hybrid models coming from different AI Labs. My interest areas are architectural changes that can improve reasoning, memory, and adaptation within practical compute and energy budgets. IMO, the next frontier might be less about scaling alone and more about getting more intelligence out of limited resources. That's what got me looking into these architectures and putting together this comparison table. The 2 tables compare their core mechanisms, attention variants, test-time memory, and how some of these approaches overlap(not so mutually exclusive). Post-Transformer architectures Architecture / Lab Core mechanism Mamba-2 / Mamba-3 - Gu, Dao et al. Selective SSM; Mamba-3 adds complex-valued state + MIMO RWKV-7 "Goose" - RWKV Foundation Parallelizable RNN, constant memory, no KV cache BDH (Dragon Hatchling) - Pathway Graph of neurons, attention emerges from local interactions. GPU form = ReLU-low-rank + linear attention xLSTM - NXAI Exponential gating + matrix memory (mLSTM) Titans - Google Research Neural long-term memory module updated by "surprise" at test time Qwen3-Next - Alibaba Qwen Gated DeltaNet: attention, 3:1 Kimi Linear - Moonshot AI Kimi Delta Attention: MLA, 3:1 MiniMax-01 / M1 - MiniMax Lightning Attention: softmax, 7:1, + MoE Jamba / Jamba 2 - AI21 Labs Mamba : attention ≈ 7:1, + MoE Nemotron-H - NVIDIA Most attention layers swapped for Mamba-2 Falcon-H1 - TII Attention and Mamba-2 heads run in parallel inside each block Zamba2 / Zamba2-VL - Zyphra Mamba-2 backbone + shared-weight attention blocks every N layers LFM2 / LFM2.5 - Liquid AI 10 gated short-conv blocks + 6 GQA blocks, found by hardware-in-the-loop search StripedHyena 2 (Evo 2) - Together AI / Arc Institute Three Hyena conv operators (short/medium / long implicit) + attention HRM / HRM-Text - Sapient Intelligence Two coupled recurrent modules (slow planner, fast worker), single forward pass Where they overlap Architecture Linear attn. Softmax attn. Test-time / fast weights Brain-inspired Mamba-2 / Mamba-3 ✓ (via SSD) - - - RWKV-7 "Goose" ✓ - ✓ (dynamic state evolution) ? (RNN lineage) BDH (Dragon Hatchling) ✓ - ✓ (Hebbian synapses) ✓ xLSTM ✓ (covariance update) - ✓ ? (LSTM lineage) Titans - ✓ (MAC / MAG variants) ✓ ✓ (memory framing) Qwen3-Next ✓ ✓ (1 in 4) ✓ (delta rule) - Kimi Linear ✓ ✓ (1 in 4) ✓ (delta rule) - MiniMax-01 / M1 ✓ ✓ (1 in 8) - - Jamba / Jamba 2 ✓ (inherited) ✓ (1 in 8) - - Nemotron-H ✓ (inherited) ✓ - - Falcon-H1 ✓ (inherited) ✓ - - Zamba2 / Zamba2-VL ✓ (inherited) ✓ (shared blocks) - - LFM2 / LFM2.5 - ✓ - - StripedHyena 2 (Evo 2) - ✓ - - HRM / HRM-Text - ? (inside modules) - ✓ For Ref: ? = uncertain or variant-dependent. “Inherited” = attributed to a constituent component. Ratios = linear-attention/SSM layers: softmax-attention layers. I feel like a lot of major shifts in AI start as research breakthroughs and eventually become the focus of innovation across the industry. Lmk if I've missed any architectures worth adding or if there are any disagreements. Thoughts? submitted by /u/Public-Feedback-6705 [link] [comments]
Related
- ParaRNN: Parallel Training for Non-Linear RNNs (sLSTM, RWKV-7, CfC, Titans) via Triton Newton Scans [P]
- Going from 3B/7B dense to Nemotron 3 Nano (hybrid Mamba-MoE) for multi-task reasoning — what changes in the fine-tuning playbook? [D]
- Pathway’s brain-inspired architecture development on Amazon SageMaker HyperPod
Source: r/MachineLearning | 2026-10-14