Model Releases
KIV: 1M token context window on a RTX 4070 (12GB VRAM), no retraining, drop-in HuggingFace cache replacement - Works with any model that uses DynamicCache [P]
KIV is a project shared on r/MachineLearning presenting a drop-in replacement for HuggingFace's `DynamicCache` that enables up to 1 million token context windows on consumer hardware with only 12GB of
KIV is a project shared on r/MachineLearning presenting a drop-in replacement for HuggingFace's DynamicCache that enables up to 1 million token context windows on consumer hardware with only 12GB of VRAM (e.g., an RTX 4070), requiring no model retraining or architectural changes. It works by offloading KV cache layers to CPU RAM rather than keeping them entirely in GPU memory, exploiting the fact that the KV cache is accessed sequentially by layer index during the forward pass — allowing efficient prefetching and eviction. The solution is compatible with any model that uses HuggingFace's DynamicCache, making it broadly applicable across a wide range of transformer-based LLMs without any code changes beyond swapping the cache class.
Related
- HiCI: Hierarchical Construction-Integration for Long-Context Attention
- Stacked from One: Multi-Scale Self-Injection for Context Window Extension
- Small Vision-Language Models are Smart Compressors for Long Video Understanding
- how to create .md files and set context window more than 64k for ollama and claude running locally.
- I reduced my token usage by 178x in Claude Code!!
Source: r/MachineLearning | 2026-04-12