DeepSeek v4 Flash vs. Qwen3.6-27B, 3.5-122B, and Gemma 4 31B Benchmark

Just wanted to share my agentic coding benchmark run of DSv4F 0731 at both High and Low reasoning efforts (not Max)... I ran a 109-question subset of Aider Polyglot (the JS/C++/Python languages), based on the coding that I do most often. This benchmark measures file-editing and diffs in a harness, and gives the model two tries to accomplish the task -- one try blind, then if it fails, it gets another shot after seeing the results of its first attempt. I tracked first try pass rate, second try pass rate, well-formed diffs, # malformed, # of context overflows (60k token limit) and timeouts (1-hour limit including retry) [first image]. I also tracked cost, based on seconds/case, prompt tokens, completion tokens, prefill and decode tokens, and kilotokens/solved question [second image] Finally, I broke down the success rate by language for each model [3rd image]. Overall, as expected, Flash on High was the overall winner. However, it wasn't that far ahead, and it outspent the next-best 122B over 5X in tokens to get there. Also, the Qwen models solved a LOT more of the cases on their first try than DS4, which is a surprising find. Overall, I was shocked how well 122B performed. I used the excellent ThinkingCap fine-tune of 27B because I didn't want to die of old age before base 27B finished the benchmark -- in previous coding and knowledge benchmark runs I did to choose my daily driver, 27B and ThinkingCap always performed within the statistical margin of error of each other, but ThinkingCap completed the same task using 20-30% of the total tokens. I highly recommend trying it out if you feel like 27B overthinks excessively. Or, if you can fit it, just run 122B -- it consistently overdelivers in all my testing. Setup: M5 Max 128GB DeepSeek v4 Flash: antirez mixed Q2-Q4 imatrix K-quant, Dwarfstar inference engine. Qwen3.5-122B: Unsloth Q5_K_XL, llama.cpp [n=4] Qwen3.6-27B-ThinkingCap: Unsloth Q8_0, llama.cpp [n=4] Gemma4-31B-QAT: Unsloth Q4_K_XL (QAT uncompressed) [n=4] Notes: I ran both High and Low reasoning modes in DSv4 because of a quirk in the way reasoning effort is sent in the current build of Dwarfstar, the inference engine I use for DSv4. Basically, Deepseek changed the encoding of reasoning effort between preview and 0731, so the string used to trigger Max effort on preview now triggers High effort on 0731, a new string triggers Max, and if no string is passed, instead of defaulting to High 0731 defaults to Low effort. There is currently no way to call Max effort in Dwarfstar without editing the code, and after seeing the token use of Low/High, I decided I wasn't likely to use Max in actual use anyways, so I didn't make the edits required to do a Max run. I'd already burnt several days of GPU time on this anyways... The JS/C++/Python set of Aider Polyglot is 109 questions, not 107. However, 2 questions triggered a linter bug in several runs before I caught it, resulting in uncontrolled generation as the linter fed back an empty error message. DS4F in particular generated 60k tokens trying to find a non-existent error, which is what led me to catch the issue, as I thought the run was hung. Out of fairness, I have excluded these 2 questions from all the metrics. For the llama.cpp models, I ran n=4 (4 simultaneous threads). This allowed me to complete the benchmarks ~2.5x as fast. Dwarfstar doesn't allow this, so in the interest of fairness, I measured both aggregate decode and single-stream decode for each model on the same prompt/output, then scaled all the relevant metrics by that proportion to ensure the numbers are comparable. Please share any benchmarks or comparisons you've done! submitted by /u/returnity [link] [comments]

Source: r/LocalLLaMA | 2026-08-04