110+ TPS through 100K tokens with 27b on a single 3090, with 400k ctx

Hello everyone! As always, thank you for sharing you previous results (and bugs) on the last version. I am happy to report that LlamAmpere v0.6 is now out. The headline items for you (at temperature 1): a ~13% jump in TPS, with a 112 TPS average across 3 task types in the 100-105K token interval a +20% increase on prompt processing, now starting around 1800 TPS and holding 1000+ through the first 100k The average prefill over the first 100K went from ~1,080 to ~1,300 TPS A new file format (Ampere Integer+) designed to be as fast, smaller, and higher precision (lower KLD) than XS formats. Vs EXL3, it is approximately the same KLD, with a significantly faster speed. The recommended model (now available on HF) is about 10% better KLD and 1.2 GB smaller. This results in slighter faster speed, and of course more room for context. I will be releasing more quants at smaller sizes in this format later this week as I continue to tweak the design at 2 and 3 bits. I have also upgraded EXL3's kernels, and in my testing it was running ~8% faster than in the original EXL fork. But, at 4 bits this configuration shows ~17% faster decode and 25% faster prefill on AI+, so I will be focused more on those kernels moving forward (EXL had a non-stat sign <2% lower KLD in testing) As in the last release, these numbers are built around SJKVaRN, which enables a maximum total context of over 400K ctx with 23.5GB total usage (across multiple predictions or context scaling). I will note that these prefill numbers were performed with the recently create 3090/ti GPU unlock; if you haven't implemented it, I recommend doing so (repo here: https://github.com/Zagitta/gpu-unlock) Github for LlamAmpere here: https://github.com/JakeATX/llamAmpere new model here: https://huggingface.co/jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI4 There are also upgrades relevant for 3.8 FlashNext included, but at present testing is "only" even with the other best options like Strata. I have a couple pending commits that I hope to have out by the end of the day, that should materially boost speed for MOEs models further; I will publish results on that separately. And for 12 GB cards, I will have some new quants (and possibly a distillation) for 3.8, hopefully by EOW. Please share your results, and feedback, as usual. Enjoy! https://preview.redd.it/i1zngu27y7yh1.png?width=1017&format=png&auto=webp&s=fb0275d70ad81f6ee29db2131f1660d85ffbf624 submitted by /u/Brief-Tap-6616 [link] [comments]

Source: r/LocalLLaMA | 2026-10-28

Related