How to evaluate LLMs before production
Mariko Wakabayashi is a Principal Applied Scientist at Microsoft who leads the development of agentic AI workflows for cybersecurity operations. She specializes in evaluating LLM‑powered systems and a
Knowledge catalogue
Mariko Wakabayashi is a Principal Applied Scientist at Microsoft who leads the development of agentic AI workflows for cybersecurity operations. She specializes in evaluating LLM‑powered systems and a
Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency, while maintaining flexibility to choose among more than 20 models. The p
We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post
In controlled offline evaluations, HydraFusion’s selective coding workflows matched or exceeded the evaluated Opus 5 baseline while reducing estimated workflow cost. Now available as a research previe
musa : define CUDA_ARCH for device passes (#29508) The MUSA vendor header never defined CUDA_ARCH, so every architecture test in the shared ggml-cuda sources evaluated to 0. Kernel bodies gated on the
CUDA: fix CCCL version guard breaking on major version rollover (#29453) CUDA: fix CCCL version guard breaking on major version rollover The guard compared the major and minor components independently
ggml-openvino: update to 2026.4.1, optimize performance, expand ops, improve device listing. (#29852) ggml-openvino : Qwen3.5 MoE perf (#312) Squash of ravi9#312: ggml-openvino: add detailed inference
HIP: broaden MoE ncols_opt tile heuristic on RDNA3.5 architecture (#28935) It's found the MoE ncols_opt tile heuristic needs to be broadened to include the RDNA3.5 architecture. The code change is imp
vulkan : fuse UNARY(GELU|SIGMOID|SILU|SOFTPLUS) + MUL (#27220) vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL vulkan : fuse UNARY(SIGMOID|SILU|SOFTPLUS) + MUL implement fusion in unary.comp behind U
model: add Qwen3.8-Flash-Next (qwen4exp) (#27742) gguf: add qwen4exp (Qwen3.8-Flash-Next) arch and converter Adds the GGUF-side plumbing for HF model_type qwen4_exp: MODEL_ARCH.QWEN4EXP plus tensors f
What's Changed Models that don't set a repeat_penalty now default to 1.0 (off) instead of 1.1, matching other engines and speeding up speculative decoding; set a per-model parameter if an older model
model : fix SWA not being enabled for EXAONE 4.5 (#26848) model : fix SWA not being enabled for EXAONE 4.5 load_arch_hparams tests hparams.n_layer() == 64 before LLM_KV_NEXTN_PREDICT_LAYERS has been r
ci : onboard AMD ROCm CI with gfx1151 fixes (#26544) ci: prepare for amd rocm ci Signed-off-by: Aaron Teo aaron.teo1@ibm.com ci: fix editorconfig-checker Signed-off-by: Aaron Teo aaron.teo1@ibm.com ci