Search: “pretraining”

AntLing has released the full six-checkpoint matrix for the Ling-3.0 base model. tiny: pretrained, mid-trained, WSM-merged flash: pretrained, mid-trained, WSM-merged The concrete artifact is six separ

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free

DeepSeek released on a paper on pretrained engrams in January. I’m surprised ds didn’t release engrams with v4 pro. When will ds or another lab release a fixed engram model? Fixed engrams will be the

It's an encoder-only transformer that consumes past(blood glucose + carbs + insulin) and future(carbs + insulin) and predicts future blood glucose for the next 2 hours. Announced meals and boluses/bas

mii-llm, an open source AI lab, released Zagreus-0.4B-por, a compact bilingual Portuguese–English language model pretrained entirely from scratch. The model has approximately 400 million parameters an

For the past few weeks I've been trying out converting existing dense models to sparse Mixture-of-Experts models, with no pretraining from scratch. The Idea If you can turn a dense model into an MoE t

https://huggingface.co/inclusionAI/Ling-3.0-tiny-base https://huggingface.co/inclusionAI/Ling-3.0-flash-base-midtrain https://huggingface.co/inclusionAI/Ling-3.0-flash-base-30T https://huggingface.co/

SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers SCoPE adds camera sightlines as positional coordinates to a pretrained video diffusion transformer. Given a first frame

A few months ago, I had the idea of making a LLM from scratch as a personal project (for learning and partly for improving my resume). Since I learned a lot from other posts on here over the past year

openPangu-2.0-Pro is an MoE model trained on Ascend. The model has 505B total parameters and 18B activated parameters. Its context length is 512k. The total pretraining data contains 34T tokens. Durin

Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a

TSAuditor is a contamination auditing framework for time series foundation models that uses probe adaptation dynamics to detect when evaluation datasets may have been exposed during pretraining, valid

Chroma1 is an 8.9B parameter text-to-image foundational model , with different versions including V48 (512x raw pretrained checkpoint) and variants like Chroma1-HD (mixed resolution finetune) . Chroma

Learning from data as we observe it is easy for humans, but most machine learning models have limited ability to learn from new data that they have not seen during training. Prior-fitted networks (the

First of all, thank you for reading. I trained a small MoE model completely from scratch (no external base weights) and wanted to share the results + a couple of lessons. Apex-2 - Architecture: Decode

Hi, I use hermes as a harness, and I am pleased with it, but sometimes Hermes's context size is a bit too much for my system, so I wanted to delegate or straight up use a lighter option for coding tas

Standard constrained RL assumes consequences are immediate and attributable to the current action. This breaks down whenever violations are delayed and stochastic, which is most real-world settings yo

*SupraLabs released a new experimental model!* SupraElegans-500K is a ~500,000-parameter causal language model built around a sparse, signed, recurrent neural graph. No Transformer, no attention mecha

As the title suggests I'm currently trying to make a modal to identify which app is being used. The thing is I don't have labelled data and generating it is out of question since that is a lot of work

Hi ML Community, I am excited to share our latest framework, AFP-GIC, officially published in IEEE Access (2026). We have released the deployment codebase and hosted an interactive visual playground.

Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this In the past 2 months Ive been training a new model

Hey there folks! Aritra here from Hugging Face. I wanted to update you all about the latest changes in `transformers`. We now natively support GGUFs (llama cpp quants). You can use it like so: from tr

Having a blast with Qwen Image 2.1 on my M2 Macbook Pro with 32GB RAM, even though it takes 16 minutes to complete the recommended number of iterations with the resolution reduced to 1024x1024. And th

What if you don’t need a new model to get Jev-style decisions? I built hearim(헤아림. hea-a-rim. not Here, I am.. lol), an open-source gateway that turns ordinary local or hosted LLMs into a Jev-compatib

According to people familiar with the matter, Naive AI, an AI startup founded in February this year by Tsinghua University professor Dai Jifeng, has completed three funding rounds totaling 400 million

I posted something about a 9b model a few days ago. The real problem was 2 fold. Someone suggested the size was too big to prove it all out. It's a fair argument. I needed the model depth though. The

from internlm: We introduce Intern-S2-397B, our most capable multimodal foundation model for scientific intelligence and long-horizon agents. Intern-S2-397B scales along three critical dimensions: pre

I have a full model, it's ready to train. It's ~9b parameters. 9.4b to be more exact. That includes a 1/2/3 Engram table, Moonshot's AttnRes modeling, and RoPE / NoPE layering at 3:1 as more or less v

Hey gang, im wondering if you in theory could use ngrams as seen with Qwen 3.8 Flash or DS4.1 in order to dynamically train the model? Normally the ngram embeddings behave similar to a lookup table of

A few posts tagged with 'new model' present models that are finetunes. My opinion : I'd rather have the 'new model' tag reserved for new 'major' releases, like a new Qwen model, Deepseek V4 -> Deepsee

I was reading up on the recent controversy around Tristan Buckmaster, Levent Alpöge, OpenAI, and the Navier–Stokes result, and it got me thinking about something broader than this particular dispute.

I don't think anyone posted about this here, but Qwen released a finetuned version of 3.5 4 for driving. The full Bf16 checkpoint is 9B. This is a very interesting development of Chinese AI labs tackl

TimesFM-3 is the third generation of Google Research's zero-shot forecasting model, and the main change from 2.5 is that it handles multivariate inputs natively instead of being limited to a single se

***This screenshot is an app i made for creating a dataset from scratch, this is not a real chat with the model*** First of all i want to shout out everyone that actually tested our work .. we got 500

Quote: ... developing a new A.I. model intended to be as powerful as Anthropic’s cutting-edge models. ... In July, while developing Watermelon, Meta paused and later resumed a stage of A.I. developmen

Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter

We introduce Mobius-v0, an architecture that comprises a globally shared Memory (FFN) that stores knowledge vectors and multiple Reasoners (Self-Attn) that iteratively achieve compositional reasoning.

Sometimes you have tasks that are outside of your expertise and the idea is this workflow automation could be leveraged to manage to have local AI figure it out using research from his workflow gather

Questions & Responses(in BOLD) below. Favorite question(s) moved to end of the thread with combined responses(removed duplicates). Be optimistic folks. I'm sure we're getting other models too apart fr

Hello guys. I found some problems in current frontier models. And want to share. # math_code_hallucination > Record of a failure caused by combining mathematics and code in a single prompt. --- ## Cas

https://huggingface.co/swiss-ai/Apertus-v1.5-70B https://huggingface.co/swiss-ai/Apertus-v1.5-8B Apertus 1.5 is a family of 8B and 70B parameter language models designed to advance the state of multil

LLMs handle speech well once you run speech-to-text. They don't hear the rest: a bird outside, a glass breaking two rooms away, a smoke alarm two floors down. I've been working on an experimental open

Disclosure: I work at Pluralis Research, the lab that built this. Code is open, and I'm happy to answer questions. TL;DR: As far as we can tell, this is the first RL post-training run whose entire rol

A Reddit post on r/MachineLearning sharing an open-source project and accompanying book by Sebastian Raschka that walks through implementing GPT-2, Llama 3, and DeepSeek from scratch using PyTorch, wi

SenseNova's NEO-unify is an encoder-free unified multimodal model built on a Mixture-of-Transformers (MoT) backbone that eliminates the need for traditional VAE encoders in image generation. The 2B NE

A Reddit user exploring Ollama Cloud noted that, at the time of their post, only the Kimi model supported image (vision/multimodal) inputs among the available cloud models. Kimi K2.5 is a native multi