Search: “evaluation”

Since OpenAI retired SWE-bench Verified in February (every frontier model tested could reproduce reference fixes for some tasks; underspecified tests rewarded knowing the intended fix), I've been tryi

I wanted a quick calories counter for myself, using LLMs to evaluate the calories from pictures of meals + descriptions. I needed to pick a model so I made a quick benchmark. The setup was: - Nutritio

Fizgig, a tool for the Klein 9B model, has introduced 'LoRA Royale,' a new evaluation mode featuring epoch travel via morphing slider, seed travel, prompt travel, and LoRA strength travel capabilities

TuneForge is an MCP (Model Context Protocol) server that enables coding agents like Claude and Cursor to perform machine learning operations directly within chat interfaces, including dataset generati

Just watched a video from three years ago where Matt Berman tested Bard (yes, remember Bard?) and most of the tests were like summarization or logic tests. If it ever came to coding, it was the snake

We used HFlow to evaluate the latest open weights VLMs for processing egocentric data. This was based on Build AI's Egocentric-10k evaluation, which used Gemini 2.5 Flash to measure hand visibility an

GoBench evaluates LLMs on 9x9 Go games against a ladder of KataGo opponents, from random to superhuman. It measures general reasoning ability, strongly correlates with ARC-AGI 2 (r=0.83 correlation),

In February OpenAI stopped reporting SWE-bench Verified and recommended other labs stop too. Every frontier model they tested could reproduce the human-written reference fix, or verbatim details of th

I'm a retired platform engineer, coding mainly in Rust, and involved with a ML study group. I developed a ML programming language (alternative to Python, Colab) to help me learn (and teach) ML concept

Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showin

Hello fellow local AI people! I took 'you must create your own benchmarks' literally, and built a website for this. How does the end result look like Let's say I want to know which model has most comm

Hi everyone! We’ve just released a major update to the leaderboard! We are expanding beyond Python with a new multilingual slice featuring real-world software engineering tasks across 5 languages. Ope

This post was made with AI. I tried to remove as much slop as possible and keep it straight to the point to save your time as I know how annoying AI slop posts can be, but I still wanted to retain all

Generated with H3: Mock H3 version of the ending to Wargames(83) as a test of a custom node I've been working on w/ both Claude(Opus/Fable) and OAI's Sol/Astra. One click generation using only 4 movie

Hey r/LocalLLaMA, Prism-LM recently released its Bonsai 2 QAT models based on Qwen3.8, and they quickly gained traction. In our evaluation, the models strike a strong balance between throughput and qu

Live results, coding benchmarks included, agentic benchmarks included, domains and other classifications filters included. (Spoiler: DeepSeek-V4-Vision-Exp rules, but other models have their rule area

One thing that has bothered me about LLM benchmarks for a while is that most of them are essentially snapshots. A model is evaluated, a score is published, and we tend to talk about that score as if i

If you run local agentic coding harnesses (Aider, Claude Code, etc.), prompt evaluation usually eats up most of your execution time. Every turn re-evaluates thousands of identical prefix tokens_system

Introducing Open SLM Evaluations! Open SLM Evaluations is a dataset containing benchmarks for 106 SLMs(150M parameters and below). It has benchmark scores for each model, links, parameter counts, and

AQuA's preprint uses 'recursive self-improvement' for a bounded research loop. It does not say the research-agent LM rewrites its own weights. The paper separates three objects: The language model dri

This r/MachineLearning post discusses the landscape of frameworks and tools used to support benchmarking of LLMs and agentic AI systems, covering how to systematically evaluate model capabilities beyo

Lunara introduces a novel Diffusion Mixture Transformer architecture with fewer than 10B active parameters for modeling artistic intelligence in image generation. Its CAT training algorithm iterativel

Disclosure: I’m one of the authors of VeriLoop E2, a 27B model post-trained from Qwen3.8-27B. The weights are Apache-2.0; the harness we evaluate it in is not open source (details at the bottom). This

Like many of you, I have seen many posts and tweets in the last weeks complaining about Artificial Analysis being 'broken', 'meaningless', and 'bought out.' People who say this have done no research a

Suppose you’re recording a human plugging a cable into a socket to collect demonstrations for robot learning. The hand tracker captures the approach accurately. Then occlusion causes the hand estimate

I wanted to find out whether a huge text-only MoE could be given basic vision without retraining the language model itself. The short answer is yes. I froze DeepSeek V4 Flash and a 417M-parameter Moon

I've ran Kimi-k3 through 34 oneshot prompts and evaluated the generated htmls, screenshots and gifs using sonnet 4.6. It came out to be better than opus4.8 from the evals. Kimi K3: https://oneshotlm.c

Signals V2 is an LLM-free analyzer tool that evaluates live agent execution trajectories by scoring them as OpenTelemetry spans, enabling developers to monitor and trace AI agent behavior without rely

This r/StableDiffusion thread discusses community comparisons of video generation models (such as those built on Stable Diffusion or FLUX architectures) evaluated specifically for how well they preser

Figure 1b from our paper Disclosure: I'm one of the authors (Microsoft). The paper, code, dataset are public and ThinkingBox is on Hugging Face OpenEnv as well. Raw evaluation trajectories are not rel

I've recently been working part-time on an AutoResearch-style project. The setup is roughly: humans take recent work from top-tier ML/AI conferences, turn part of it into a well-defined task with an e

Hi, Im gonna go all in on this, best finetune iv got my hands on for agentic coding, period. Test it yourself, im not gonna give you any benchmarks or evaluations, use your own tests and pracices, giv

'TypeSafe AI sells Jev as a frontier-class reasoner that cannot hallucinate, built by the co-inventor of ChatGPT - fast, and almost free. We ran it live on 16,379 benchmark requests, measured its late

Ok, so the Mac M5 Ultra (256GB) hit the market, but the only publicly available benchmarks material are flashy YouTube 'clown influencers' videos. We need serious numbers to evaluate whether Apple’s s

Baseten launched a new safety infrastructure standard alongside its Base Labs research arm on Wednesday, partnering with Hugging Face and Goodfire AI to build safety evaluation and monitoring infrastr

Hi Ollama Team, I’m writing regarding the announced retirement of GLM-5.1 from Ollama Cloud on 25 September 2026. I wanted to provide some fairly detailed technical feedback because GLM-5.1 is not sim

Style transfer capabilities of different open-source methods 2026 Update This is the updated version of the study published at 2025.09.12. Link: https://www.reddit.com/r/StableDiffusion/comments/1nfoz

I’ve been building Catalyst around a capability I want engineering agents to have: An AI agent can read a program and make a good guess about what matters. Catalyst lets it measure what actually matte

Hi everyone, I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-

A few weeks ago, I saw this post where the author trained a SIREN MLP to implicitly memorize Bad Apple as a coordinate function: (t, y, x) to pixel. That got me curious about a slightly different form

I'm testing a new approach for reducing the cost of image-based LLM inference. I evaluated it on the MOMA Graph benchmark, using 1,315 questions. Compared with using GPT-4o to process the original ima

Krea 2 Turbo — 4-Step Distillation LoRA (work in progress) A LoRA for Krea 2 Turbo that reduces the minimum usable step count from 8 to 4. This is an update release, following up from my previous post

Share what your favorite models are right now and why. Given the nature of the beast in evaluating VLMs (untrustworthiness of benchmarks, immature tooling, intrinsic stochasticity), please be as detai

I trained three LLMs from scratch in raw PyTorch then post-trained each one with SFT and then GRPO. Same process every time: same synthetic arithmetic curriculum, same reward function, same hyperparam

Including the rationalisation for the data below - this is a more robust version of an earlier post I did similar to this - explaining below: How I chose the comparisons The basic question I’m trying

I've been building an open-source tool for diagnosing world models, the kind that predict future frames from a starting context and a sequence of actions. It compares a rollout against ground truth an

I am working on an AI for a small single-player merge puzzle and would appreciate pointers to related algorithms, papers, or existing implementations. It resembles 2048 in its action -> afterstate ->