Research
INT3 compression+fused metal kernels [R]
INT3 compression with fused Metal kernels enables large language models to compute attention operations directly on compressed (INT3/INT4) key-value cache representations using custom GPU kernels that
INT3 compression with fused Metal kernels enables large language models to compute attention operations directly on compressed (INT3/INT4) key-value cache representations using custom GPU kernels that eliminate intermediate dequantization steps. This approach achieves significant speedups (up to 48x) and memory savings (3.2x compression) compared to standard dequantize-then-attend pipelines, enabling long-context inference on consumer hardware.
Related
- DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing
- [[d-will-googles-turboquant-algorithm-hurt-ai-demand-for-memor|[D] Will Google’s TurboQuant algorithm hurt AI demand for memory chips? [D]]]
- YOCO++: Enhancing YOCO with KV Residual Connections for Efficient LLM Inference
- Latent-Condensed Transformer for Efficient Long Context Modeling
- EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling
Source: r/MachineLearning | 2026-04-22