Search: “pretraining”

I trained a 1B-parameter LLM from scratch on 20B tokens for about $200

model-releasesr-localllama

A few months ago, I had the idea of making a LLM from scratch as a personal project (for learning and partly for improving my resume). Since I learned a lot from other posts on here over the past year

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Impressive new paper from Meta. (bookmark it) Scaling laws assume model size and training data act on loss independently. This work introduc…

DGX agent

Impressive new paper from Meta. (bookmark it) Scaling laws assume model size and training data act on loss independently. This work introduces Skaling law, which couples capacity and data through a si

researchdair-ai--x

10 Aug 2026

Same physical state, different collective dynamics: state encodings select synchronization outcomes in language-model agents

arXiv:2608.06968v1 Announce Type: cross Abstract: Language-model agents act on state encodings of their environment, yet these are treated as interchangeable interfaces. Using pretrained language mode

WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN

arXiv:2608.07267v1 Announce Type: new Abstract: Recent vision-language navigation (VLN) systems increasingly adapt pretrained vision-language models (VLMs) into vision-language-action (VLA) policies t

safetyarxiv-cs-ai

Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

arXiv:2608.07341v1 Announce Type: cross Abstract: Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. extbf{Contamination mitigation

Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation

arXiv:2608.05999v1 Announce Type: new Abstract: Vision-language-action (VLA) models have demonstrated remarkable capabilities in robotic manipulation by leveraging pretrained vision-language models. H

researcharxiv-cs-ro

Simulator-Grounded Large Language Models for Industrial Causal Reasoning: Tool-Use, Structured Injection, and Plant-Portable Retrieval for Wastewater Treatment Decision Support

arXiv:2608.05151v1 Announce Type: cross Abstract: Wastewater operators need answers grounded in how their plant's variables interact and how fast effects propagate, not in generic pretraining text, wh

Toward Deployable Bangla Sign Language Recognition with Expert-Validated Data and a Lightweight Attention-Based Model

arXiv:2608.06252v1 Announce Type: cross Abstract: Deaf and hard-of-hearing people in Bangladesh communicate mainly through Bangla Sign Language (BdSL). Automatic BdSL recognition on personal devices c

Agents

When Do Corrective Features Help? An Agent for Corrective Feature Discovery on Black-Box Forecasters

arXiv:2608.05207v1 Announce Type: new Abstract: Frozen pretrained forecasters often fail in structured, recurring ways that are costly to repair through fine-tuning. We study corrective feature discov

agentsarxiv-cs-lg

RegisterBridgeMM: A Register-Centric Framework for RGB-Infrared Object Detection

arXiv:2608.04833v1 Announce Type: new Abstract: RGB-infrared (RGB-IR) object detection benefits from complementary visible and thermal cues, but effective fusion remains challenging under illumination

6 Aug 2026

SEAR: Simple and Efficient Adaptation of Visual Geometric Transformers for Unpaired RGB+Thermal 3D Reconstruction

arXiv:2603.18774v2 Announce Type: replace Abstract: Foundational feed-forward visual geometry models enable accurate and efficient camera pose estimation and scene reconstruction by learning strong sc

6 Aug 2026

Seeking Physics in Diffusion Noise

arXiv:2603.14294v3 Announce Type: replace-cross Abstract: Do video diffusion models encode signals predictive of physical plausibility? We probe intermediate denoising representations of pretrained Di

6 Aug 2026

Local Ai

A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learning

arXiv:2608.03015v1 Announce Type: cross Abstract: Pretrained large language models (LLMs) have demonstrated in-context learning (ICL) capabilities for numerical inference over sequences serialized as

local-aiarxiv-cs-ai

Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension

arXiv:2608.03494v1 Announce Type: new Abstract: Vocabulary extension is an efficient way to adapt pretrained large language models (LLMs) to new languages, but the initialization of newly added token

Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation

arXiv:2608.03791v1 Announce Type: new Abstract: Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpo

Tutorials

GVCCTurbo: Rate-Compute Quality Scheduling for Codebook Driven Generative Compression

arXiv:2608.03517v1 Announce Type: new Abstract: Codebook-driven generative compression uses a pretrained image or video generator as a zero-shot visual prior and transmits compact codebook indices to

tutorialsarxiv-cs-cv

Wiring Beats Blending: What Transfers Between Transformer Sizes -- and What Doesn't

model-releasesarxiv-cs-lg

arXiv:2608.02829v1 Announce Type: new Abstract: Model families train every size from scratch. Can a pretrained large model be converted into a smaller sibling? We characterize the 1.4B->410M conversio

CENTILE: A Telemetry Foundation Model Evaluated by the Decisions It Drives

arXiv:2608.01725v1 Announce Type: cross Abstract: Modern computing and networking infrastructure emits telemetry continuously, yet operators convert it into decisions with a separate predictor per tas

researcharxiv-cs-lg

Few-Shot Concept Prompt Learning for Segmentation Foundation Models via Visual Grounding

arXiv:2608.01663v1 Announce Type: new Abstract: Promptable segmentation foundation models (FMs) such as SAM3 and Medical SAM3 promise few-shot, interactively-specified segmentation for medical imaging

HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models

arXiv:2608.02124v1 Announce Type: cross Abstract: Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence. We identify a previously overlooked cause: spec

researcharxiv-cs-cl

RL Bootstrapping of OpenVLA-OFT for a Novel Robot Embodiment

arXiv:2608.01013v1 Announce Type: new Abstract: Adapting a pretrained vision-language-action (VLA) policy to a new robot usually assumes embodiment-specific demonstrations. This assumption is especial

safetyarxiv-cs-ro

Writing-System-Level Tokenizer Adaptation for Byte-Level BPE

arXiv:2608.00582v1 Announce Type: new Abstract: Pretrained byte-level BPE tokenizers can segment underrepresented languages inefficiently. Replacing a tokenizer changes the meaning of nearly every tok

Tutorials

DualDiT: A Conditional Dual-Output Diffusion Transformer for Joint OCT Image and Segmentation Mask Generation

arXiv:2607.29337v1 Announce Type: cross Abstract: Background and Objective: Generating realistic medical images with anatomically accurate segmentation masks helps address the shortage of annotated da

tutorialsarxiv-cs-ai

3 Aug 2026

Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers

arXiv:2607.28611v1 Announce Type: new Abstract: Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibi

Huawei opensouced openPangu-2.0-Pro, 505B-A18B

openPangu-2.0-Pro is an MoE model trained on Ascend. The model has 505B total parameters and 18B activated parameters. Its context length is 512k. The total pretraining data contains 34T tokens. Durin

safetyr-localllama

Predict before you train: Scaling Laws for particle physics foundation models

arXiv:2607.23377v1 Announce Type: cross Abstract: The largest machine learning models in particle physics are also the most expensive to train, yet the return on scaling a given architecture cannot be

Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

arXiv:2607.27680v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only parti

ViP-Rig: Visual-Prompted Controllable Rigging

arXiv:2607.27982v1 Announce Type: new Abstract: Rigging is inherently task-dependent because the same mesh may require different skeletons and deformation behaviors across animation tasks. In practice

Between Gradient and Natural Gradient: A Continuum of LoRA Initializations

model-releasesarxiv-cs-lg

arXiv:2607.26247v1 Announce Type: new Abstract: Low-rank adaptation (LoRA) fine-tunes large pretrained models at a fraction of the cost of full fine-tuning, but its performance depends strongly on how

LumaGuide: Distribution Shaping for Training-Free HDR Generation in Diffusion Models

arXiv:2607.26237v1 Announce Type: new Abstract: Pretrained diffusion models generate realistic images but are constrained by the statistical biases of their training data, limiting their ability to pr

SPROUT: A Scalable Diffusion Foundation Model for Agricultural Vision

arXiv:2603.27519v2 Announce Type: replace Abstract: Image-based plant phenotyping depends on dense structural understanding of crops, yet pixel-level annotation remains expensive across species, organ

TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Plans

model-releasesarxiv-cs-ro

arXiv:2603.09971v2 Announce Type: replace Abstract: We present TiPToP, a modular manipulation system that integrates pretrained foundation models with a GPU-accelerated Task and Motion Planner to solv

Architectural Backdoors in Vision-Language Model Supply Chains via Representation Steering

arXiv:2607.25479v1 Announce Type: cross Abstract: Vision--Language Models (VLMs) are increasingly deployed through a model supply chain in which pretrained checkpoints, architecture definitions, text

safetyarxiv-cs-ai

Beyond Predictive Accuracy: A Reliability-Aware Audit of Molecular Representations for Human Olfaction

arXiv:2607.24848v1 Announce Type: cross Abstract: Pretrained molecular encoders are commonly evaluated through downstream prediction, but predictive accuracy alone does not establish that a learned re

safetyarxiv-cs-ai

CFR-Net:Collaborative Feature Refinement Network for Medical Image Anomaly Detection

arXiv:2607.11509v2 Announce Type: replace Abstract: Medical image anomaly detection remains challenging because networks pretrained on natural images often exhibit limited adaptability to medical imag

CondPSE: A Polynomial-Filtered Structural Encoder with Conditional Modulation for Graphs

arXiv:2607.25169v1 Announce Type: cross Abstract: Message-passing graph neural networks are bounded by the 1-WL test and can miss topological structure that distinguishes non-isomorphic graphs. Positi

Instruction-Tuned Models Locally Reuse Human Syntax More Than Humans Do

arXiv:2607.26015v1 Announce Type: new Abstract: Syntactic convergence (the tendency of speakers to adapt in language towards the grammatical profiles of their interlocutors) is a well-documented featu

REPREC: Representation Driven Parameter-Efficient Recommendation System

arXiv:2607.24845v1 Announce Type: cross Abstract: Large language models (LLMs) have been applied to sequential recommendation by formulating it as a natural language task. Previous work has improved p

RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection

arXiv:2607.24771v1 Announce Type: new Abstract: Knowledge injection updates pretrained MLLMs with new factual or domain-specific knowledge, but fitting full authoritative answers can cause drift in no

Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation

arXiv:2607.23860v1 Announce Type: cross Abstract: Deep ensembles provide the most reliable uncertainty estimates in deep learning, but their cost grows linearly with the number of members. Implicit en

EgoPlay: Event-Triggered Video Editing for Egocentric Streams

arXiv:2607.24560v1 Announce Type: cross Abstract: We introduce EgoPlay, an event-triggered video-to-video editor for egocentric streams, obtained by fine-tuning a pretrained V2V diffusion transformer

MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

arXiv:2607.24424v1 Announce Type: new Abstract: Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features ca

microsoft/Mage-VL · Hugging Face - An Efficient Codec-Native Streaming Multimodal Foundation Model

model-releasesr-localllama

Mage-VL is a codec-native, proactive-streaming multimodal foundation model for image and video understanding, whose visual encoder is trained entirely from scratch at a compact 4B scale. It targets a

Industry

Great to see this out, proud of the team!!! @AMD Instella 16B MoE foundation model, fully open with checkpoint after each stage from pretrai…

Great to see this out, proud of the team!!! @AMD Instella 16B MoE foundation model, fully open with checkpoint after each stage from pretraining, long-context, SFT, DPO and RL. Dataset detail, trainin

industryemad-mostaque--x

25 Jul 2026

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging

arXiv:2607.20561v1 Announce Type: cross Abstract: LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter

Tutorials

ElasticTTT: Prior-Preserving Test-Time Tuning for Video Editing

arXiv:2607.21529v1 Announce Type: cross Abstract: Test-Time Tuning (TTT) on pretrained diffusion models has emerged as a powerful paradigm for video editing. However, there exists a foundational misma

tutorialsarxiv-cs-ai

GigaPath-Flash and GigaTIME-Flash: Efficient Pathology Foundation Models for Whole-Slide and Tumor Microenvironment Analysis

arXiv:2607.18218v2 Announce Type: replace-cross Abstract: Foundation models have emerged as a driving force in computational pathology, with the potential to transform cancer diagnosis, prognosis, and

Learning causality from internet videos in latent space first, and then using RL to teach the foundation model how to act. This approach is …

model-releasesyann-lecun--x

Learning causality from internet videos in latent space first, and then using RL to teach the foundation model how to act. This approach is 30× cheaper than Gemini 3.1 Flash on pretraining and achieve