Search: “pretraining”

What if you don’t need a new model to get Jev-style decisions? I built hearim(헤아림. hea-a-rim. not Here, I am.. lol), an open-source gateway that turns ordinary local or hosted LLMs into a Jev-compatib

I’ve been experimenting with whether Qwen3.8-Flash-Next’s pretrained PLE n-gram memory can improve a much smaller Qwen3.5-0.8B model. I trained the 0.8B setup with limited resources, mostly using free

Automated lint: 17 errors, 59669 warnings, 3 info

Automated lint: 19 errors, 64915 warnings, 3 info

arXiv:2610.02418v1 Announce Type: cross Abstract: Modern large language models (LLMs) are trained on massive, largely unfiltered datasets, including content scraped from nearly every accessible websit

arXiv:2610.07385v1 Announce Type: new Abstract: Fine-tuning vision-language models (VLMs) is typically evaluated at a single downstream checkpoint, obscuring whether a semantic capability was never ac

arXiv:2610.07764v1 Announce Type: cross Abstract: Natural language processing (NLP) models can detect depression-related language in text written near the time symptoms are measured, but whether pretr

🚨 @AnthropicAI: AI “MAY BE” capable of self-replicating! Now read the actual research: • No fully autonomous rogue replicator demonstrated. • Cyber exploitation deliberately simplified. • Real-world r

Automated lint: 17 errors, 59669 warnings, 3 info

arXiv:2609.30667v1 Announce Type: new Abstract: A robust world model must strike the balance between faithfully capturing environmental dynamics and abstracting away from irrelevant content. While rec

arXiv:2609.37499v1 Announce Type: new Abstract: Historical archives are an under-used source for extending the instrumental climate record backward in time, and LLMs offer a way to extract the indices

arXiv:2609.37310v1 Announce Type: cross Abstract: With LLM watermarking being deployed commercially and now required by regulations, improving its reliability and effectiveness has become crucial. Yet

Auto-generated synthesis of 18054 entries about arxiv-cs-cv

Automated lint: 19 errors, 64915 warnings, 3 info

arXiv:2609.32717v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) alignment is intended to ensure that models remain helpful and safe, but its stability under input distributional s

arXiv:2610.19420v1 Announce Type: cross Abstract: Simulated environments enable scalable training data generation for embodied agents, but more data does not necessarily provide what a particular agen

arXiv:2610.14921v1 Announce Type: new Abstract: A language model that keeps learning online must not slowly erode its own base capabilities. We ask which part of a modern LLM should absorb continual u

arXiv:2610.17637v1 Announce Type: new Abstract: Since scaling laws were introduced for modern LLMs (Kaplan et al., 2020), several works have extended them to account for quantization, tracking the eme

arXiv:2610.15781v1 Announce Type: new Abstract: Training-free pruning removes weights from pretrained large language models to reduce their deployment cost. However, existing approaches typically cons

arXiv:2610.19232v1 Announce Type: new Abstract: Large language models impose substantial memory demands and arithmetic costs during inference. Dense-to-mixture-of-experts (MoE) conversion is a model o

arXiv:2610.17093v1 Announce Type: new Abstract: A model diff credits post-training with whatever separates a base checkpoint from one post-trained descendant, presuming a rerun would land close by. On

Microsoft AI: Microsoft unveils MAI-Base-2, a MoE model with 2.8T total and 80B active parameters, claiming 4x-12x more pretraining FLOPs-efficiency than DeepSeek-V4-Pro-Base — In June, we announced M

arXiv:2610.17124v1 Announce Type: cross Abstract: Full-duplex spoken language models (FD-SLMs) support simultaneous listening and speaking, but often lag behind half-duplex spoken language models (HD-

arXiv:2311.04902v3 Announce Type: replace Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting per

Hello guys. I found some problems in current frontier models. And want to share. # math_code_hallucination > Record of a failure caused by combining mathematics and code in a single prompt. --- ## Cas

arXiv:2608.20418v1 Announce Type: cross Abstract: We introduce Malaria-Instruct, a curated instruction-following dataset derived from the ChEMBL Legacy Malaria corpus for Malaria virtual screening, an

arXiv:2606.21803v2 Announce Type: replace Abstract: Next-token prediction is the self-supervised signal that trains language models, and every observed prompt token provides the same signal at test ti

arXiv:2609.03967v1 Announce Type: cross Abstract: Several studies have evaluated the ability of Large Language Models (LLMs) for meal planning, yielding positive outcomes. These models can process nat

arXiv:2601.04260v2 Announce Type: replace Abstract: Understanding how Large Language Models (LLMs) perform logical reasoning internally remains a fundamental challenge. While prior mechanistic studies

arXiv:2609.21039v1 Announce Type: new Abstract: A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form W = BA in which two parameter matrices

arXiv:2609.34866v1 Announce Type: new Abstract: Most post-training quantization pipelines fit each weight matrix to its pretrained counterpart, one matrix at a time. Whether that proxy tracks what an

arXiv:2603.26164v2 Announce Type: replace-cross Abstract: Data-centric training has emerged as a promising direction for improving large language models (LLMs) by optimizing not only model parameters

arXiv:2608.07436v1 Announce Type: new Abstract: Under the standard split, Muon gets hidden matrices and AdamW embeddings/output head. Muon groks modular addition faster, but its solutions do not hold.

arXiv:2608.17923v1 Announce Type: new Abstract: Appendicitis is one of the most common abdominal emergencies worldwide and requires prompt diagnosis and treatment to prevent life-threatening condition

Automated lint: 12 errors, 53990 warnings, 3 info

A significant fraction of humanity read this thread. Is anyone prepared to do something about it? Fretting doesn’t count. I resigned from Anthropic today. I spent the last three years doing pretrainin

Automated lint: 12 errors, 54839 warnings, 3 info

Automated lint: 14 errors, 54840 warnings, 3 info

Automated lint: 17 errors, 59669 warnings, 3 info

Automated lint: 19 errors, 64915 warnings, 3 info

arXiv:2609.37499v1 Announce Type: new Abstract: Historical archives are an under-used source for extending the instrumental climate record backward in time, and LLMs offer a way to extract the indices

arXiv:2610.00888v1 Announce Type: cross Abstract: Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per for

Reasoning from scratch, round number 6! An introduction (and implementation) of Reinforcement Learning with Verifiable Rewards (RLVR) and Group Relative Policy Optimization (GRPO). 00:00 Introduction

Last time when I posted on this subReddit to share my work, the response almost made me cry because a tiny demo got so many people talking about this In the past 2 months Ive been training a new model

arXiv:2610.12610v1 Announce Type: cross Abstract: Ethnic and cultural identities rarely align with political borders, a tension made sharper by migration and long-standing cross-border ties. Communiti

Microsoft AI: Microsoft unveils MAI-Base-2, a MoE model with 2.8T total and 80B active parameters, claiming 4x-12x more pretraining FLOPs-efficiency than DeepSeek-V4-Pro-Base — In June, we announced M

arXiv:2610.14705v1 Announce Type: cross Abstract: Large language models have advanced medical reasoning, but clinically grounded reasoning in traditional Chinese medicine (TCM) remains challenging bec

arXiv:2311.04902v3 Announce Type: replace Abstract: Large Language Models (LLMs) with billions of parameters are prime targets for network pruning, which removes some model weights without hurting per

NVIDIA’s Seattle Robotics Lab and Isaac engineering team created reusable robotic systems to automate GB300 tester‑tray assembly, achieving over 95 % success on busbar tasks with a 160‑second cycle ti

arXiv:2610.08108v1 Announce Type: new Abstract: Diffusion language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) language models, offering flexible token-update orders

arXiv:2610.15560v1 Announce Type: new Abstract: Existing navigation systems commonly rely on dedicated depth sensors or explicit depth reconstruction, while learning to generate trajectories directly

arXiv:2610.14726v1 Announce Type: new Abstract: Latent diffusion models enable high-quality video generation, but video VAE encoding and decoding remain computationally expensive. Existing methods use

arXiv:2610.15781v1 Announce Type: new Abstract: Training-free pruning removes weights from pretrained large language models to reduce their deployment cost. However, existing approaches typically cons

arXiv:2610.15373v1 Announce Type: cross Abstract: We adapted Nemotron-3-Nano-30B-A3B for eight Indian languages without changing its backbone architecture. We expanded the vocabulary in place from 131

arXiv:2610.20086v1 Announce Type: new Abstract: Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usuall

arXiv:2610.15371v1 Announce Type: cross Abstract: Gnani-Evon-v3.3-30B-A3B is a long-context hybrid Mamba-Transformer mixture-of-experts model for reasoning, instruction following, and tool use in Engl