Nine models, five coding tasks, one 16GB GPU

I ran five coding tasks against nine models on one RX 9060 XT (16 GB). Each run gets a single spec file in an empty directory and has to produce solution.py and test_solution.py, run its own tests, and stop. Grading happens with a checker the model never sees: fixed cases from the spec, randomized cases against a brute-force oracle, and a performance gate where the spec names one. A cell is capped at 1750 seconds; if the model is still working when the clock runs out, whatever is on disk gets graded. The tasks: quoting rules for one CSV record (c1), an arithmetic evaluator with pinned exception behavior (c2), a POSIX-ish shell tokenizer (c3), max-profit interval scheduling with two tie-breaks and 200k intervals in under 3 seconds (c4), and shortest-prefix pattern matching with 20 stars against a 4,000-character text in under 10 seconds (c5). model c1 c2 c3 c4 c5 score out tokens deepseek-v4.1-flash (cloud) P P P P P 5/5 43k Qwen3.8-27B UD-IQ4_XS + MTP (kvstream fork) P P P P P 5/5 185k Qwen3.8-27B IQ3_XXS P P P F P 4/5 125k Swift-1.5-Qwen3.8-27B P P P P F 4/5 113k Ornith-1.5-35B-A3B Q4_K_M P P P P F 4/5 57k Qwen3.6-35B-A3B UD-Q4_K_M P P P P F 4/5 126k Nemotron-3.5-Bolt-74B-A9B IQ2_XXS P F F F F 1/5 51k Mellum2.1-12B-A2.5B F P F F F 1/5 167k LFM2.5-24B-A2B Q6_K F F F F F 0/5 187k c1 to c3 mostly sort models into "writes code and iterates" and "does not". Everyone who reached c3 wrote a single-pass scanner of 76 to 97 lines and passed, and nobody was tempted into shlex, which would have failed the comment rule. c4 and c5 carry the signal. On c4 every model that produced a file used the same shape: sort the intervals, then a DP with bisect for the predecessor. The losses were the tie-break rule, the cost of carrying interval lists through the DP, and duplicate-heavy input. On c5 the split is clean. All three passes came from an automaton stepping a set of live states through a closure (both Qwen3.8-27B variants built a Thompson NFA, the cloud run used a big-integer bitset). The failures came from parse-then-match routines that missed one rule each, or from handing the matching to Python's re, which backtracks on star piles and eats memory while it does. Per model: deepseek-v4.1-flash: 5/5 in 236 s and 43k tokens. bisect DP at 0.25 s on c4, bitset NFA at 0.00 s on c5. Writes a test file, runs it, stops. kvstream-fork (Qwen3.8-27B UD-IQ4_XS with MTP): also 5/5, but 185k tokens over 94 minutes. Thompson NFA with epsilon closure on c5, 0.03 s. Both of its capped cells were self-inflicted: on c5 it reported "all tests pass" (the grader agreed) and then hung a 9,000-case fuzz whose oracle translated patterns to re. Qwen3.8-27B IQ3_XXS: 4/5, clean on c1 to c3. c4 produced no file at all: 46k tokens of ls, pwd and re-reading the spec, with no code in its reasoning either. Solved c5 with the same NFA design. Swift-1.5-Qwen3.8-27B: 4/5. Writes early and iterates hard, 8 edits and 22 bash calls on c1. c4 passes at 0.27 s; c5 never rejects nested groups. Ornith-1.5-35B-A3B Q4_K_M: 4/5 and the cheapest local, 57k tokens and about 16 minutes for all five cells, c1 done in 37 s. c5 fails one randomized case where a group and a star interact. Qwen3.6-35B-A3B UD-Q4_K_M: 4/5 at 126k tokens, fastest local cells on c2 and c3. c4 is correct but measured 3.03 s against the 3.0 s gate, and re-grading the same file gives 2.97 to 3.23 s. c5 misses the nested-group rule too. Nemotron-3.5-Bolt-74B-A9B IQ2_XXS: 1/5. Three cells spend two turns, make one tool call, and then report files that do not exist; c3's report still had the agent template's placeholder "What was done." in it. c1 is a real pass, and its reasoning for c3 did contain code that passes the grader. Mellum2.1-12B-A2.5B: 1/5. c1 imports csv and declares BLOCKED, leaving a file named BLOCKED with the reason in it. c3 never returns on tokenize("ab cd"). c5 converts the pattern to a Python regex; grading that file needs a memory cap, since uncapped it reached 7.3 GB. LFM2.5-24B-A2B Q6_K: 0/5, 187k tokens, three of five cells using the entire cap. 88 turns on c1 ending in an empty-field off-by-one. c4 is correct and slow: 6.98 s against the 3.0 s gate because it copies the chosen interval list inside the DP. The same 5/5 shows up twice at very different prices: 43k tokens in 236 seconds against 185k in 94 minutes. The models reach the same algorithms, so the cost is in the path, not the destination. It was a pretty fun experiment, mostly did it to settle which local model to use as a coding workhorse. Based on this it's probably going to be the Q4 Qwen 3.8 27b. submitted by /u/Ok_Substance2327 [link] [comments]

Source: r/LocalLLaMA | 2026-10-18

Related