Search: “benchmark”

Hi all! After all the jev hype I wanted to try by myself and got qwen3.8 27b to mix-and-match open implementations on top of ninfer, and to be honest I'm quite impressed with the results, latency is n

I want to try using AI to find possibly overlooked interventions for irritable bowel syndrome (IBS) and create a cocktail of interventions that AI thinks would give the highest expected benefit, attac

https://preview.redd.it/d7brwofkhuqh1.png?width=1400&format=png&auto=webp&s=9a0079ebc2710faff9a73378dac58c6dd67e3cec link: https://huggingface.co/Altworld/Hemmingway-1 submitted by /u/paf1138 [link] [

Agent evaluation is moving from scoring individual function calls to judging full task completion within executable environments that track state over multi‑step tool use. Scoring occurs at two levels

Hey everyone, I wanted to share a project I’ve been working on called Jayce. The whole thing started because I was watching a toddler named learn the names of stuff He didn't need to completely rewire

arXiv:2609.21626v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed for enterprise information extraction (IE), where the same document must be reorganized different

arXiv:2609.21953v1 Announce Type: new Abstract: Learning to defer asks a predictive system when to act autonomously and when to defer to a human expert. Population-adaptive deferral extends this probl

arXiv:2609.22000v1 Announce Type: new Abstract: Computer-use agents (CUAs) have advanced along two separate lines: graphical interaction and software development through code and the command line. Rea

arXiv:2609.21223v1 Announce Type: new Abstract: Vision-language-conditioned robot policies integrate perception, language understanding, and control for general-purpose manipulation. However, existing

arXiv:2609.21983v1 Announce Type: new Abstract: Reusing manipulation experience across robot embodiments is important for scaling robot learning and reducing repeated task-specific data collection. Ho

arXiv:2511.13254v2 Announce Type: replace Abstract: Large Language Models (LLMs) have displayed remarkable capabilities across diverse domains, but their training remains resource- and time-intensive,

I forked llama.cpp to implement Declarative Attention (arXiv:2609.02737, Google DeepMind and KAIST AI). The model declares in its own output which context chunks it needs <focus magic_chunks='N'>), an

I’ve spent basically the last 8 hours testing different models on the exact same web-development prompt, and I finally finished. The whole point of this nine-hour test was that which local model match

Typesafe's new model, Jev is all over the internet, this sub etc as a new classifier model. Then an AI dev posted here how they'd done something very similar, and made a new model, Laya. https://www.r

First time trying something like this. I asked chatgpt and it told me there are mcp server plugins for both ghidra and ida pro as well as RE-MCP which has support for both. I could run that on my low

I needed a relatively simple but acceptable level of AI for working on one project. I didn't have any heavy requests, I just needed to give the AI access to the project files so it could search throug

AI tools have given us a lot of slop research, but I'm not talking about that. I'm talking about real productivity acceleration due to AI tools (E.g. iterating ideas that would've taken multiple days

Screenshot of heavy load. I did offload vision to a spare 2060 instead of local ram. The control panel was generated with qwen3.8-27b davidau q4k_m max mtp. Made the panel from scratch over a week, ba

I did a small test of the new hyped quantisation of Qwen3.8 vs the biggest quant which fits into my limited 16GB VRAM with decent context. The results are interesting. Of course, the smaller file give

I improved the TPS of Gemma 4 31B. Improving TPS and performing optimisations requires understanding of the model architecture, and I had to fork VLLM and apply my own patch to break into making a con

Given the recent interest in tools like Jev, I wanted to share ProgramAsWeights (PAW), an open-source research project I'm working on at the University of Waterloo. You describe a text function in Eng

Note: This writeup was put together with the help of AI. So I've got dual AMD MI50 32GB cards. If you know these cards, you know the story - AMD dropped official ROCm support for gfx906 after ROCm 5.7

Oh My Pi + vLLM on two 3090s. Average wait per turn went from 28s to 7s, mostly from changing omp settings: explicit effort level on every role (unset ones defaulted to xhigh) thinking_token_budget of

arXiv:2505.17640v3 Announce Type: replace Abstract: Time series segmentation assigns a label to each part of a sequence. We formulate dense univariate segmentation as node classification on a graph wh

arXiv:2609.20541v1 Announce Type: new Abstract: Large language models can report a numerical confidence together with generated content, but it is unclear whether this report is more than calibrated r

arXiv:2602.03128v2 Announce Type: replace Abstract: Multi-agent LLM frameworks are data-intensive systems that govern how agents orchestrate tasks, manage state, and coordinate decisions. These archit

arXiv:2609.19991v1 Announce Type: cross Abstract: Omni models can describe video content, but can they locate events in time, preserve event order, and judge audio-visual synchronization? We introduce

bonsai claim 98.2% intelligent retained, but their own documents show Ternary Bonsai 2 27B reaches 52.8 and 60.8, respectively, compared with 69.7 and 80.6 for Qwen3.5-27B, retaining roughly three qua

arXiv:2609.20625v1 Announce Type: cross Abstract: Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitw

arXiv:2609.19425v1 Announce Type: new Abstract: Tool-augmented large language model (LLM) agents fail in a way no tool-selection or tool-security method addresses: they call tools that do not exist an

arXiv:2609.20586v1 Announce Type: cross Abstract: Referring scene understanding for embodied robots requires grounding object- and relation-centric language queries from a designated viewpoint. While

arXiv:2609.20441v1 Announce Type: new Abstract: Geospatial foundation models can provide strong flood-segmentation performance, but their size limits deployment on memory-constrained edge hardware. We

arXiv:2609.19885v1 Announce Type: new Abstract: Interpreter agents built on machine translation (MT) increasingly mediate live conversation between people who do not share a language, yet we still eva

arXiv:2609.19827v1 Announce Type: new Abstract: With the widespread industrial deployment of Large Language Models (LLMs), DeepSearch has emerged as the dominant paradigm for resolving complex user qu

arXiv:2609.19680v1 Announce Type: new Abstract: Financial QA systems are typically improved before deployment through better retrieval, prompting, or agent coordination, leaving their reliability beha

arXiv:2609.19716v1 Announce Type: new Abstract: Pre-trained 3D vision models have substantially advanced point cloud analysis, yet adapting them to downstream tasks via full fine-tuning is computation

arXiv:2609.20508v1 Announce Type: new Abstract: E-commerce livestreams have emerged as an important channel for presenting products to online consumers, containing multiple products whose information

arXiv:2609.19356v1 Announce Type: cross Abstract: We introduce a new method to guide flow matching models. Our approach, which we call probe guidance, uses the frozen internal states of an existing di

arXiv:2609.20277v1 Announce Type: new Abstract: World Action Models (WAMs) have demonstrated strong robotic manipulation capabilities by augmenting pretrained video generative models with action exper

arXiv:2609.19417v1 Announce Type: new Abstract: Graph-based retrieval-augmented generation (RAG) is widely used for multimodal, cross-document question answering. However, building corpus-level graphs

arXiv:2609.19377v1 Announce Type: cross Abstract: Recovering numerical series from line plots requires accurate axis calibration and reliable curve extraction. We present LinePilot Digitizer (LinePilo

arXiv:2603.21785v2 Announce Type: replace Abstract: Visual odometry (VO) frontends are typically tuned offline by domain experts on pre-recorded datasets and then deployed with fixed hyperparameters.

arXiv:2609.20794v1 Announce Type: new Abstract: Generative models are increasingly used to solve scientific inverse problems, but existing evaluations still focus primarily on whether a method can pro

arXiv:2609.19657v1 Announce Type: cross Abstract: Repeated prompt prefixes are increasingly common in LLM serving workloads, appearing in system prompts, templated retrieval-augmented generation pipel

arXiv:2609.20754v1 Announce Type: new Abstract: Effective troubleshooting agents in enterprise customer support depend on retrieving actionable guidance from similar historical cases, yet existing ret

arXiv:2505.16180v3 Announce Type: replace-cross Abstract: Evaluating image captions requires cohesive assessment of both visual semantics and language pragmatics, which is often not entirely captured