I built an LLM benchmark harness that lets you browse and compare how models answered each question

Hello everyone. I felt current LLM benchmark harnesses hand you headline numbers but offer no tooling to see how models actually answered each question (they dump everything to a JSONL or Parquet file, so you end up writing custom code just to read the answers). So I built lm-eval-ledger: a benchmark harness that runs the benchmarks, writes everything down, and provides a web app that lets you inspect and compare how each model answered each question. For the demo I benchmarked three models on a single 5090: Qwen3.5-9B, NVIDIA-Nemotron-3.5-Lightning-30B-A3B (UD-Q4_K_XL GGUF), and Gemma-4-12B-it (QAT w4a16). It looks like Qwen thinks far longer than the other two. GPQA Diamond: Qwen 0.717 vs Nemotron 0.657 vs Gemma 0.601 — but 2h26m vs 1h34m vs 1h19m LiveCodeBench: Qwen took 22h57m (vs 9h51m / 6h13m), with 0.713 vs 0.837 / 0.820 accuracy Here are the full results served by lm-eval-ledger on Hugging Face Spaces: https://huggingface.co/spaces/jayminbhan/lm-eval-ledger What lm-eval-ledger records and displays Per question: system prompt, model generation, extracted answer, ground truth, stop reason, generation character count Per benchmark: accuracy, tok/s, time to completion, sample count, no-answer count Extras: pairwise comparison of two models on the same task/questions, always-wrong / always-right questions across benchmark runs How it works: everything is written to a single SQLite DB, and a Flask app displays it. No more custom bash scripts for multi-model x multi-task runs either. lm-eval-ledger is YAML-driven: copy template.yaml, list N models and M tasks, run one command. That’s it. Usage pip install lm-eval-ledger # add a backend: pip install "lm-eval-ledger[vllm]" lm-eval-ledger init # writes template.yaml, creates results/ and logs/ directories lm-eval-ledger -c bench.yaml # run benchmarks lm-eval-ledger serve # browse at http://localhost:8090 Details and the full task list are in the GitHub repo: https://github.com/jayminbhan/lm-eval-ledger I verified the vLLM / SGLang / HF / server (llama.cpp) backends on Linux, and HF / server (llama.cpp) on Windows. This started as a script to look at one wrong GSM8K answer. It got out of hand. Any feedback, ideas, or pull requests are greatly appreciated! submitted by /u/jayminban [link] [comments]

Source: r/LocalLLaMA | 2026-09-06

Related