Model grafting: turning Qwen3.5-4B into a causal encoder-decoder after the fact

Recently, the new DeepSeek-V4.1-Flash architecture showed how a causal encoder-decoder can work, but it was trained from scratch. Model Grafting does it to an existing model: cut at some depth, let the lower layers read the prompt, and use the upper layers get for encoder's residual stream as prefix KV via identity-init adapters, then heal with self-distillation from the unmodified parent. Decoding part stays the same, this method was described in this blog post https://latentnode.pages.dev/articles/model-grafting I applied the same recipe to Qwen3.5-4B to create two graft variants - https://huggingface.co/LocalLLaMA/Qwen3.5-4B-graft8 and https://huggingface.co/LocalLLaMA/Qwen3.5-4B-graft16 The graft8 variant shows speedup of ~3.7x at 128K prompt with some loss in accuracy. The graft16 is much more closer to the parent model while showing 2.0x speedup in prompt processing with minimum loss in accuracy. submitted by /u/asankhs [link] [comments]

Related

• DeepSeek Flash 4.1 is already being tested via API and rolling out.
• DeepSeek debuts DeepSeek-V4.1-Flash, its smallest model built on its new Causal Encoder-Decoder architecture, with 552B backbone parameters and 1M-token context (Reuters)
• 3 experiments running dsv4-flash-0731 q4+ quants on 128GB RAM + ~60 GB VRAM (with a quite bad pcie infra) with an acceptable tgs and relatively acceptable pp speed
• DeepSeek-V4-Flash on SM89 4x48gb 4090s with DSpark

Source: r/LocalLLaMA | 2026-09-22