Search: “benchmark”

This r/MachineLearning discussion post details a hands-on benchmark study in which TranslateGemma — Google's open translation model suite built on Gemma 3, available in 4B, 12B, and 27B sizes and cove

I was using: weights https://huggingface.co/RadixArk/Qwen3.8-Flash-Next-NVFP4 with the optimized SGLANG (patched) from https://old.reddit.com/r/BlackwellPerformance/comments/1w04xb7/qwen38_flashnext_o

Alright, I posted that I had my agent test a bunch of different workflows for over 12 hours and got the 'Bro just wasted 12 hours of credits'. It was obvious the proof should come from the visual data

Model: unsloth/GLM-5.3-Flash-GGUF UD-Q2_K_XL (4 shards, 101 GiB) + mmproj-BF16. Reference: llama.cpp PR #27754 (glm5next), CUDA 12.8, SM 120. Throughput Both engines back to back in one session, flash

Hey guys, Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFl

The pelican on a bicycle is sooo outdated, so I came up with a new, improved version. Qwen3.8-27b medium (UD-Q4_K_XL) vs. Sol 5.6 high vs. Qwen3.6-35B (UD-Q6_K_XL) Prompt (only real with typo!): 'Crea

Finally llama.cpp now officially supports Ling-3.0! (Starting from build b10472+) If you want to run them locally, bartowski has already released the GGUF imatrix quantizations for both models: - Ling

With all the new models released recently one important upgrade went unnoticed: Llama.cpp bumped ROCm from 7.2 to 7.14. I was waiting for that because in 7.14 support for gfx1103 (Radeon 780m) was int

Disclaimer, this was kinda written with AI (Gemma 4 again) but it also did really well here, it outputted what I wanted, when I asked it to refine stuff or improve on certain areas it did that without

A while ago, I posted here getting OSS-120B and GLM-4.6 playing full games of Civilization V. Since then, models have moved pretty far, and we wanted a better understanding about models' capabilities

Hello everyone. I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts. First of all, and as a summary

I had Astra run a multi-hour investigation into speeding up local MoE inference while keeping the model weights and quantization unchanged. The experiments covered Qwen3.6-35B-A3B and Qwen3.8 Flash-Ne

TL;DR: The result looks TOO GOOD TO BE TRUE — that's exactly why I'm posting it. I ran Ox alpha on the entire SWE-bench Verified-Mini set with the official mini-swe-agent scaffold — the same agent use

When Qwen3.8-27B dropped on Thursday the obvious question came up for us : does a 27B dense model actually fit on the 24GB Mac Mini machines? Ran it properly over the weekend on an M4 Pro (24GB unifie

In medium reasoning mode, it both scores higher than the 3.6 version, AND is very much more efficient (almost half requests needed, and a third less tokens generated) - at DeepSeek v4 Flash 3107 MXFP4

LabyrinthBench measures the thing that actually kills long agent runs — whether a model can still use what it learned twenty turns ago — deterministically, with no LLM judge, on your own hardware, wit

I've been out of these forums for awhile but I figured I would provide a formal update on how this has been going now that it has some operation time under its belt, just to put the information out th

We developed a leaderboard ranking frontier model security. There's no shortage of model capability rankings, but we didn't find anything comparable for model security. Yet security is becoming increa

Recently I bought an Ollama Cloud sub and accidently spent my whole 5h quota upon using DeepSeek V4 Pro... but why? isnt it supposed to be a cheap model? Youd think there would be a correlation betwee

Someone modelled sugarcane farming as an integer program. See, sugarcane only grows next to water. Water costs one tile and can feed at most four cane tiles. The layout therefore becomes a coverage pr

I've been building a C99 inference engine from scratch (no Python, no BLAS, just gcc and make) that runs BitNet's ternary models on CPU. A few weeks ago I got obsessed with the matmul kernel - wrote a

This r/ollama post describes a streamlined method for performance-testing locally-running large language models using the Ollama framework, achievable with just three terminal commands. It likely intr

Purpose-built to stress-test computer vision models, depth cameras, and spatial AI against severe specular glare and geometric reflections. This 425-asset production archive features a custom faceted

Nonobench measures how well LLMs solve nonograms (picross). Each model gets the row and column clues once and returns the full grid. No tools, one attempt per puzzle. Method: - Standard mode: 30 puzzl

A while back I posted here about a CLI agent I'm building that parses tool calls out of a model's plain-text output instead of trusting native function-calling to work. Finally ran the actual numbers.

With the release of ThinkingCap-Qwen3.8-27B, I thought it would be worthwhile to do a comparison between the original Qwen3.8-27B, the new ThinkingCap, and Swift-Qwen3.8-27B. Both Swift which I alread

Suppose I create two machine learning models suppose tree and neural network for a task let's suppose regression problem, now suppose I am sending both of this paper to two different journals, now the

I kept picking local models badly. Parameter count, quantization, VRAM. I could read all of it and still not know whether a model would be any good on my machine. So I stopped guessing and wrote somet

How it works. The model being tested is given a server capable of running it's weights and full context. That server is placed in a median priced apartment. The AI is given a bank account with for ren

Was considering choosing a 4xRadeon AI Pro R9700 build for a office AI server. Curious to know what results you guys are getting? Mainly considering DeepSeek V4 flash and Qwen 3.8 Flash with offloadin

Hi all! I've been building a local agent harness, but before writing the code, I wanted to get some intuition around whether (and how) structuring agent execution as a graph helps vs. standard tool-ca

My test is running DeepSeek v4 Flash 0731 at native size on DGX Spark machine (GB10, 128 GB unified memory). The model size is bigger than RAM, so weights will be loaded many times when running. To im

My hobby project skill-issue is an arena FPS in a single C file, about 34k lines, everything procedural. My test for local setups is to have an agent port it to a single file HTML version with three.j

https://omlx.ai/benchmarks/performance/2pko3m1k - you can expand the raw args, but I have full annotations of what worked and what didn't. I'm now testing the model on acutal coding and haven't seen a

https://preview.redd.it/uvdb7g4o7djh1.png?width=1600&format=png&auto=webp&s=1db6176144bcf8f40efb15cc04e8370ad52bfe2f https://preview.redd.it/rj4l66eq7djh1.png?width=1257&format=png&auto=webp&s=692b5f7

Needs a lot of requests compared to Qwen (almost twice) and Gemma (almost x3). Final score is fine, even though it is 'not a coding model' https://wonderrico.github.io/local_llm_benchmark/benchmark-ma

152 generations, deterministic grading (exact number/string/JSON/regex), no LLM judge on my 16GB laptop. task type | deepseek-r1:1.5b (1.1GB) | llama3.2:3b (2.0GB) | gemma:2b | codellama 7b arithmetic

Google Cloud published OKF (Open Knowledge Format) on June 12th — a spec for storing curated knowledge as a directory of markdown files with YAML frontmatter. One concept per file, linked to each othe

I keep seeing the 'use a big model via API as the architect, run local small/mid models as workers' pattern recommended for people with modest local hardware. I've been running it myself (orchestrator

Hello guys. I found some problems in current frontier models. And want to share. # math_code_hallucination > Record of a failure caused by combining mathematics and code in a single prompt. --- ## Cas

I wanted to know how cheap you can go and still run local models, so I ran Ollama CPU-only on a Youyeetoo X1S. It's a single-board x86 machine with a Celeron N5095 (Jasper Lake, 4C/4T, 15W), 16GB of R

'Will Smith eating spaghetti' became a shorthand for early-stage AI video generation limitations , with the original 2023 clip generated with ModelScope showing distorted faces, morphed hands, and unn

This work presents an explainability library for transformer models that provides tools for understanding transformer behavior through attributions and concept-based explanations . The resource likely

Someone prompted different LLMs to generate CAD code for a bridge under fixed constraints (2-foot span, under 500g filament, 18-hour print limit), printed them, and load-tested them to failure. The re

Hey r/LocalLLaMA, We’ve released our full ShapeLearn GGUFs for Qwen 3.8 27B. Blog / Download models TL;DR 3.84 bpw (GPU-5) reaches 99.63% of BF16’s aggregate score of 8 benchmarks, being the most accu

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local. LLM Test Scores Feature De