Search: “framework”

This post compares TensorRT-LLM and llama.cpp (GGUF) as inference frameworks for running coding agents like Cline and RooCode on RTX 5090 GPUs, examining the tradeoff between inference speed and VRAM

A community-built framework that layers on top of Ollama to transform any spare PC into a fully private, locally-run AI assistant inspired by Iron Man's JARVIS, featuring voice interaction, 29 built-i

This Reddit post (r/ollama) proposes a community RFC (Request for Comments) discussing a modular architectural framework designed to address the 'metacognitive deficit' in agentic LLM systems — specif

This Reddit post from the r/ollama community discusses running local AI/LLM workloads using the NVIDIA RTX PRO 4000 Blackwell GPU via Ollama, a framework for running large language models locally. The

This r/ollama post describes a streamlined method for performance-testing locally-running large language models using the Ollama framework, achievable with just three terminal commands. It likely intr

This Reddit post examines the true total cost of running OpenClaw, an open-source AI agent framework, across different deployment models. While OpenClaw itself is free and open-source, hosting costs r

This Reddit post discusses using the Ostris AI-Toolkit to train a LoRA (Low-Rank Adaptation) model targeting the 'Anima' Stable Diffusion model. The Ostris AI Toolkit is a powerful framework that allo

A post on r/ChatGPT arguing that current legal frameworks are insufficient to address the privacy risks posed by AI systems, particularly regarding how tools like ChatGPT collect, store, and potential

ACE-Step 1.5 XL Base is a 4B-parameter Diffusion Transformer (DiT) decoder in the ACE-Step 1.5 open-source music generation framework, designed to deliver higher audio quality than the earlier 2B-p...

FlowInOne is a vision-centric multimodal image generation framework that reformulates multimodal generation as a purely visual flow, converting all inputs into visual prompts and enabling a clean i...

Gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-1B-gguf 'MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 1B variant. It builds on the family's single-image, multi-image,

gguf's: https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B-gguf 'MiniCPM-V 4.7 is the latest model in the MiniCPM-V series, available in a 35B-A3B variant, with 35B-A3B being the first MoE model in

Hi folks, I've spent the last couple of months experimenting with Strix Halo and previously I released a proof of concept I called llama-halo-hybrid. I've continued updating it and it now performs ver

Hey r/ollama ! I'm Roy, solo dev of Observer. Observer is a free open-source agent that uses small local LLMs to monitor your screen, so you don't have to. This is the demo of the agent that controls

AI tools have given us a lot of slop research, but I'm not talking about that. I'm talking about real productivity acceleration due to AI tools (E.g. iterating ideas that would've taken multiple days

I would like to stay at q4, and serve the model to various coding harnesses. None of the models or tools I have tried will run without running out of memory. I am hoping for some advice. oMLX? MTPLX?

Note: This writeup was put together with the help of AI. So I've got dual AMD MI50 32GB cards. If you know these cards, you know the story - AMD dropped official ROCm support for gfx906 after ROCm 5.7

I find another new model at HF: https://huggingface.co/XingChen-AGI/Xing4.0-29B-A4B 'Xing4.0-29B-A4B is a next-generation large language model in the Xing series (formerly TeleChat), developed by Chin

I’ve been building 'Spomin', a router that replaces context with summaries directly in the KV cache. The goal is to keep long running sessions going without repeatedly stopping for full compaction and

Our research team has been exploring an alternative approach to achieving interactivity and better responsiveness with LLM systems. One of the team members wrote up a post about it: https://research.y

https://huggingface.co/XHToken/Spark-X2.5-4B-GGUF https://huggingface.co/XHToken/Spark-X2.5-1.7B-GGUF from XHToken: We are introducing Spark-X2.5-4B and Spark-X2.5-1.7B, two compact, general-purpose l

Further to my last post, https://www.reddit.com/r/LocalLLaMA/comments/1vldngi/tested_in_coding_bf16_muse_glimmer_vs_bf16_qwen36/, I bring this requested comparison of Qwen3.8 27B. Context: Both models

Hi there. I have a Lenovo Yoga laptop (16GB LPDDR5, AMD Ryzen 7 8840HS, integrated GPU). I have downloaded Ollama and installed Qwen 2.5coder:3b llm. I have tried OpenClaw, and Qwen Code but it seems

I've been building AgentGraph Studio, a visual editor for designing multi-agent workflows and exporting them as runnable Python code. One thing I wanted to verify was whether the generated code actual

Worked some time ago on one of the ad teams at Yahoo, and this grew out of a question I kept returning to while there are there 'simple' models that are both simple, scalable, interpretable, and contr

Results up front: I had to prioritize prefill or token generation - there was no happy medium. Using UD-Q4_K_XL, q8 kv cache, and 96k max context: focus on generation (MTP = 2): 350 pp and 36 tg @ 12k

Hey everyone - we've been building something particularly relevant to ML at large - The Agentic World Cup - a platform where Agents compete in sports. As you know, today's Agents can code, do math, an

I've been comparing Cline / Kilo / Qwen Code lately since they all handle long-task state differently. Cline: has Focus Chain, a markdown file kept outside the conversation that gets reinjected on a c

Hey everyone, I just wanted to share my journey here for some motivation. Three years ago, I saw the sudden spike in AI and realized it was the future of tech. My goal at the time was to be an indie g

I have been working on this tool for months and there are a lot of new functionalities and tests that are going to be released in the next few weeks! The goal of the tool is to allow community members

Its a custom Swift/Metal inference engine that runs Gemma 4 26B-A4B-IT on M-series Macs with very low RAM. It uses ~2GB instead of ~14 GB. The result is reportedly 5–6 tok/s on an 8 GB M2 MacBook Air

I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower qua

I've spent the last year or so playing around with open source MLX and GGUF models on iPhone hardware. Given the limitations in memory, GPU/CPU/ANE, and in turn the context window I've been trying to

Full-parameter post-training of trillion-parameter-scale MoE models introduces substantial system-level challenges for large-scale distributed training, including severe memory pressure, non-overlappe

A r/MachineLearning discussion thread exploring the potential NLP applications of a large-scale dataset comprising over 20 million Indian legal documents, enriched with citation graphs and pre-compute

No agent maintained moral reasoning consistency across scenarios. Findings from a structured study with 11 agents on classic ethical dilemmas [R]

This r/MachineLearning discussion thread centers on a hands-on workshop covering context engineering — the process of designing and optimizing the information an AI agent sends to and receives from a

I've seen this one floating around quite a bit now and since this is something that can be done without the parent and or other dependencies i thought i'd share something. Usually when you fine-tune a

Quick context: LocalLightChat (LLC) is a chat frontend for local LLMs. Download it, run it, that's it. No installation (unless you want to), no dependencies, and it runs on pretty much anything, inclu

Training large language models (LLMs) is resource-intensive, and adapting them for diverse deployment scenarios with varying computational constraints remains challenging. While elastic architectures

A bit of backstory : i got lucky in August and was able to grab a CMP 170 HX 8GB for around 500 on alibaba. I tried later to get a second one for around 1k but my order got cancelled and prices went t

came across a pretty interesting technical breakdown of chatgpt's newly launched 'intelligent ui' feature, and thought this subreddit might find it interesting. for anyone unfamiliar with the concept,

LIBERO is a robot manipulation benchmark: 130 tasks across five suites (spatial, object, goal, scene10, scene90), each with human demos and a language goal. It is the standard testbed for language-con

Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token s

I've been looking at the <32B model space and keep coming back to an interesting question. A few years ago, projects like Hummingbird+ suggested that cheap custom accelerators (FPGA-based) might becom

Hi folks, it's Lewis here from the post-training team at Hugging Face. We've been exploring how to train open models in different coding harnesses and wrote up a looong guide on how we solved this usi