Qwen3.8-27B IQ3_XXS wrote a correct multilayer TMM on a 16 GB Quadro — after 100 minutes, 3 compactions, and 108k output tokens

https://preview.redd.it/i8rjx0ar5mlh1.png?width=2160&format=png&auto=webp&s=c2588bc7b2519ea71b176ca73faf566dfc585496 I wanted to see whether a heavily quantized 27B model running entirely on an older 16 GB workstation GPU could do more than the usual coding demos. FFT felt too easy, so I asked it to implement the coherent optical transfer-matrix method (TMM) for absorbing multilayer films from scratch. TL;DR: the implementation was correct long before the model believed it. It then spent most of the remaining hour debugging its own broken validators. Hardware NVIDIA Quadro RTX 5000, 16 GB, Turing / SM 7.5 Intel Xeon Silver 4116, 12 cores / 24 threads 256 GB system RAM Ubuntu 24.04.4 LTS NVIDIA driver 580.173.02 Model and serving stack Target: unsloth/Qwen3.8-27B-GGUF Quant: Unsloth Dynamic V3 Qwen3.8-27B-UD-IQ3_XXS.gguf (10.93 GB on disk) Speculative draft: incoai/Qwen3.8-27B-DFlash2-GGUF, Qwen3.8-27B-DFlash2-Q4_K_M.gguf (1.14 GB) Custom experimental DFlash2 llama.cpp PR #27342 build for SM75: 0.1.2-dev, build 32, commit f5a7ec1 -ngl all, Flash Attention on 100,352-token context K and V cache both q4_0 Batch 256, physical microbatch 64 DFlash maximum draft length 3 One inference slot Roughly 0.8–1.1 GB VRAM left after loading everything, depending on desktop use Real Harness responses are usually around 29–35 tok/s; very predictable output such as counting reaches about 42 tok/s The agent UI was DeepSeek Harness 0.1.1-rc.2, connected to llama.cpp through its OpenAI-compatible endpoint. Both the model server and Harness ran on the Ubuntu workstation; my Windows laptop was only the browser client over an SSH tunnel. Harness provided the file and shell tools and displayed the reasoning live. The session used Qwen's native low reasoning effort, but I had removed the small output cap and allowed the request to use the whole 100,352-token context. The task The prompt required a pure Python implementation using only math and cmath—no optics library and no NumPy for the actual solver. It had to support: complex refractive indices using the convention N = n + ik complex Snell angles with the physically correct square-root branch TE/s and TM/p polarization characteristic matrices for arbitrary absorbing films power reflectance and transmittance the test stack Air | SiO2 100 nm | Au 30 nm | Si at 600 nm and 45 degrees The 100-minute reasoning saga The complete agent turn lasted 99 minutes 53 seconds. The session log contains 22 model steps, 21 tool calls, and 107,594 model output tokens. The first model call alone produced 43,033 output tokens and took about 27 minutes before it finally wrote tmm.py. There were three context-compaction attempts. The first two each spent almost six minutes generating a checkpoint and then failed because the summary itself hit its token cap. Only the third compaction succeeded. Total time spent compacting was almost 18 minutes. Watching the chain of thought was honestly painful. Qwen repeatedly re-derived the same field conventions, lost track of which direction its matrices propagated, found a contradiction, announced that everything was verified, and then invented another validation method that contradicted it again. The branch/sign problem was interesting. For the requested passive n+ik convention and a forward field proportional to exp(ik_zz), it eventually reduced the complex-angle choice to this: sin_j = (N0 / Nj) * sin_theta0 cos_j = cmath.sqrt(1.0 - sin_j * sin_j) if (Nj * cos_j).imag < 0.0: cos_j = -cos_j In other words: calculate one square-root branch and simply flip it if the normal wavevector would grow instead of decay in the forward direction. No complex-arcsine heroics. That is correct for the passive positive-index materials in this test. It is not a completely general branch selector: it lacks a tolerance and a forward-Poynting tie-break when Im(k_z) is zero, and it should reject or specially handle gain media and absorbing incident media. So there is a genuine domain limitation, but it does not affect Air/SiO2/Au/Si or the passive dielectric tests below. The other funny part was the apparently "reversed" algorithm. The implementation uses a characteristic matrix whose +isin(delta) signs propagate the field state in the opposite direction from the common textbook presentation. It therefore left-multiplies each new layer: M = matmul(Mj, M) giving M_N ... M_2 M_1, rather than the conventional-looking M_1 M_2 ... M_N. At first this looked like Qwen had simply turned the algorithm around until the numbers worked. But its boundary equations were derived for exactly that propagation direction, so the reversed product is internally consistent. Ironically, Qwen later forgot its own convention in an external validator, used M = M * Mj, and obtained the wrong reflectance (0.842122 instead of 0.464845). Reversing the validator back to the implementation's order made it agree. It spent most of the run debugging its referees Two supposedly independent validation approaches were repeatedly broken by the model: The recursive reflection validator had an off-by-one base case, then an index error, and finally used exp(2j1jdelta). In Python, 2j1j == -2, so it calculated exp(-2delta) instead of exp(2idelta). The successful third compaction actually identified this exact typo, but after resuming, the model walked past the fix and eventually abandoned that validator. The forward field-march initially omitted the propagation factors exp(+/-idelta) entirely. After fixing that, the single-layer case worked, but the multilayer case still failed because of the matrix-order mistake described above. It also briefly rejected a quarter-wave antireflection test because its validator used the wrong expected refractive index. It corrected that to n_layer = sqrt(n0n_sub) and got reflectance around 1e-32. The definitive internal check was a completely independent 6x6 linear solve for all forward and backward amplitudes in the two-film stack. That matched the TMM result to about 1e-16 for both polarizations. External result I then compared the generated solver against Steven Byrnes' established tmm implementation over 400–800 nm. For Air | SiO2 100 nm | Au 30 nm | Si: maximum |delta R|, s polarization: 1.22e-15 maximum |delta R|, p polarization: 1.33e-15 I also tested three thick, weakly absorbing dielectric films to generate lots of Fabry-Perot fringes: n=1.37 + 0.0002i, thickness 6.20 um n=2.05 + 0.0010i, thickness 4.70 um n=1.68 + 0.0030i, thickness 7.35 um glass substrate, 45-degree incidence, 400–800 nm That spectrum contained 38 resolved maxima for s and 29 for p. Maximum disagreement with Byrnes was 1.25e-14 for s and 3.50e-15 for p. The plotted curves lie on top of each other. So yes: a roughly 3-bit 27B model on an older 16 GB Turing GPU produced a genuinely correct niche numerical-physics implementation from scratch. The quantization did not prevent it from getting the final mathematics right. But the agent behavior was equally instructive. More autonomous validation was not monotonically better. Once the initial implementation and analytical limits were correct, the model spent over an hour creating bugs in its own "independent" checks and then reasoning in circles about the contradictions. The implementation was right long before the model believed it. It spent most of the hour debugging its own referees. Has anyone else seen this pattern with local reasoning models—good primary solution, followed by a validation death spiral? I would also be interested in comparing the same task across IQ3, Q4, and higher reasoning-effort settings, although I am not eager to watch another 108k-token derivation immediately. EDIT: Formatting submitted by /u/1000_bucks_a_month [link] [comments]

Source: r/LocalLLaMA | 2026-08-26

Related