Search: “pretraining”

DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

arXiv:2607.24516v1 Announce Type: cross Abstract: While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely he

researcharxiv-cs-ai

28 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

Scaling GUI Agents with Visual State Transitions

DGX agent

arXiv:2607.24112v1 Announce Type: new Abstract: We introduce State Transition Pretraining (STP) as a new scaling axis for GUI agents. During the STP stage, we continually pretrain a unified multimodal

agentsarxiv-cs-ai

28 Jul 2026

Stress-Testing EEG Foundation Models for Clinical Decoding: Dataset Identity and Targeted Negative Controls

arXiv:2607.24519v1 Announce Type: cross Abstract: Pretrained EEG foundation models are increasingly proposed for clinical decoding, but their transfer across populations and robustness to negative con

28 Jul 2026

A Unified Framework for In-Context Learning with Causal and Masked Language Models

arXiv:2607.04081v1 Announce Type: new Abstract: In-context learning (ICL) has emerged as a central capability of pretrained language models, yet its theoretical analysis has focused primarily on causa

researcharxiv-cs-lg

7 Jul 2026

InternVLA-A1.5: Unifying Understanding, Latent Foresight, and Action for Compositional Generalization

arXiv:2607.04988v1 Announce Type: new Abstract: Unified models for robot manipulation aim to equip one policy with both the semantic priors of pretrained VLMs and the physical dynamics learned through

safetyarxiv-cs-ro

7 Jul 2026

MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training

arXiv:2602.06285v2 Announce Type: replace Abstract: Recent research in geospatial machine learning has demonstrated that models pretrained with self-supervised learning on Earth observation data can p

7 Jul 2026

AFFMAE: Scalable Vision Pre-Training for High-Resolution Microscopy Segmentation on Desktop Hardware

arXiv:2602.16249v2 Announce Type: replace Abstract: Self-supervised pretraining has transformed computer vision by enabling data-efficient fine-tuning, yet high-resolution pretraining typically requir

2 Jul 2026

The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail

arXiv:2606.04010v1 Announce Type: cross Abstract: Brain foundation models (BFMs) are self-supervised Transformers pretrained on fMRI data. We posit that these models should capture each subject's cogn

4 Jun 2026

Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization

arXiv:2605.17379v1 Announce Type: cross Abstract: Large language models pretrained on general-domain corpora often exhibit tokenization inefficiencies when applied to specialized domains. Although con

19 May 2026

Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models

arXiv:2605.10759v1 Announce Type: cross Abstract: Diffusion and flow-matching models scale because pretraining is supervised regression: a clean sample is noised analytically, and a model regresses ag

12 May 2026

Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications

arXiv:2604.27987v1 Announce Type: new Abstract: Fine-tuning pretrained models has become a standard approach to adapting pretrained knowledge to improve the accuracy on new sparse, imbalance datasets.

1 May 2026

Polychromic Objectives for Reinforcement Learning

arXiv:2509.25424v5 Announce Type: replace-cross Abstract: Reinforcement learning fine-tuning (RLFT) is a dominant paradigm for improving pretrained policies for downstream tasks. These pretrained poli

safetyarxiv-cs-ai

28 Apr 2026

Are Natural-Domain Foundation Models Effective for Accelerated Cardiac MRI Reconstruction?

arXiv:2604.22557v1 Announce Type: cross Abstract: The emergence of large-scale pretrained foundation models has transformed computer vision, enabling strong performance across diverse downstream tasks

tutorialsarxiv-cs-cv

27 Apr 2026

SS3D: End2End Self-Supervised 3D from Web Videos

arXiv:2604.22686v1 Announce Type: new Abstract: We present SS3D, a web-scale SfM-based self-supervision pretraining pipeline for feed-forward 3D estimation from monocular video. Our model jointly pred

27 Apr 2026

Towards Brain MRI Foundation Models for the Clinic: Findings from the FOMO25 Challenge

arXiv:2604.11679v1 Announce Type: new Abstract: Clinical deployment of automated brain MRI analysis faces a fundamental challenge: clinical data is heterogeneous and noisy, and high-quality labels are

14 Apr 2026

Applications

PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation

arXiv:2610.16355v1 Announce Type: new Abstract: Generalist robot manipulation can benefit from pretrained vision-language models (VLMs) for semantic reasoning and video generation models (VGMs) for wo

applicationsarxiv-cs-ro

14 Oct 2026

GRACE: Generation-aware latent compression for efficient video generation

arXiv:2610.10524v1 Announce Type: new Abstract: Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far few

tutorialsarxiv-cs-cv

8 Oct 2026

Compact Robot Policies Need Fine-Grained Visual Representations

arXiv:2610.08183v1 Announce Type: cross Abstract: Multi-task manipulation policies differ in architecture, scale, and pretrained priors all at once, so published comparisons cannot attribute performan

7 Oct 2026

Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

arXiv:2610.07074v1 Announce Type: cross Abstract: Estimating a decision threshold requires locating observations near an unknown boundary. We study how gradient-based pretraining learns this statistic

7 Oct 2026

Scalable next-scale autoregression for medical image generation across anatomical regions

arXiv:2602.14512v3 Announce Type: replace Abstract: Autoregressive pretraining has been key to the scalability of large language models, yet medical generative foundation models remain predominantly b

7 Oct 2026

Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs

arXiv:2610.02657v1 Announce Type: new Abstract: Converting a pretrained autoregressive (AR) model to a diffusion language model (dLLM) enables parallel generation without pretraining a new model. Publ

5 Oct 2026

MintFlow: Minimal Trajectory Intervention for Constrained Flow Matching

arXiv:2610.02260v1 Announce Type: new Abstract: Flow matching models excel at generative modeling, and many downstream applications require their samples to satisfy prescribed constraints, such as obs

researcharxiv-cs-ai

5 Oct 2026

Applications

Beyond Simulation: Retain-and-Repair Neural Operators for Real-World Adaptation

arXiv:2609.39387v1 Announce Type: new Abstract: Neural operators increasingly benefit from pretraining on numerical simulations, yet adapting them for real-world prediction remains challenging. We int

applicationsarxiv-cs-lg

1 Oct 2026

FluxLite: Inference-Time Proposal Control for Discrete Diffusion Models

arXiv:2609.35947v1 Announce Type: new Abstract: Many inference-time tasks for pretrained discrete diffusion models and diffusion language models reduce to drawing samples from a tilted version of the

30 Sept 2026

V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents

arXiv:2609.37250v1 Announce Type: cross Abstract: World-action models (WAMs) couple future visual-state prediction with action generation. By adapting video generators or image-editing models pretrain

safetyarxiv-cs-ai

30 Sept 2026

From Data to Program: Fast & Direct Generative Program Inference from Empirical Data

arXiv:2609.35348v1 Announce Type: new Abstract: Estimating probability densities from a finite set of samples typically requires dataset-specific model fitting. We introduce PRODiGI, a pretrained data

Quantifying the Effect of Test Set Contamination on Generative Evaluations

arXiv:2601.04301v3 Announce Type: replace Abstract: As frontier AI systems are pretrained on web-scale data, test set contamination has become a critical concern for accurately assessing their capabil

Reduce, Then Encode: Multiscale Volumetric Reduction for 2D Foundation Models in Brain MRI

arXiv:2609.35405v1 Announce Type: new Abstract: Pretrained 2D foundation models offer a practical alternative to dedicated 3D pretraining for brain structural magnetic resonance imaging (sMRI), but th

Scalable In-Domain Self-Supervised Foundation Model for Dense Representation Transfer in High-Resolution Plant Imaging

arXiv:2609.32183v1 Announce Type: new Abstract: High-resolution plant imaging enables detailed characterization of plant morphology, but dense scientific analysis remains limited by costly pixel-level

Anchor to Expand: Semantic Anchoring for Personalized Text-to-Image Diffusion Models

arXiv:2511.22245v2 Announce Type: replace Abstract: Personalizing text-to-image diffusion models extends pretrained models to represent novel user-specific concepts from only a few reference images. H

safetyarxiv-cs-cv

From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning

arXiv:2609.30391v1 Announce Type: new Abstract: Existing in-context reinforcement learning methods mainly pretrain Transformers with supervised behavior-prediction objectives. This enables task infere

safetyarxiv-cs-lg

InternW0-Delta: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data

arXiv:2609.31394v1 Announce Type: cross Abstract: World Action Models (WAMs) jointly model visual dynamics and action generation for generalist robot manipulation. A central challenge is to integrate

PALM: Point-in-Time Adaptation for Financial Language Models

model-releasesarxiv-cs-cl

arXiv:2609.30316v1 Announce Type: cross Abstract: Language models used in financial backtests suffer from look-ahead bias, as a model trained on text published after the study period has already obser

TimeBraid: Unifying Time Series and Language for Understanding and Forecasting

arXiv:2609.29792v1 Announce Type: cross Abstract: We present TimeBraid, a series of unified time-series and language models that align pretrained language models and pretrained time-series foundation

tutorialsarxiv-cs-ai

25 Sept 2026

Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning

arXiv:2609.29060v1 Announce Type: cross Abstract: Transformers achieve remarkable performance by jointly learning broad families of tasks during pretraining and adapting to unseen tasks from only a sh

researcharxiv-cs-lg

25 Sept 2026

Enabling Vision and Cross-Modal Learning for Multimodal Stroke Recurrence Prediction: An Interpretable Two-Step Framework

arXiv:2609.22271v1 Announce Type: new Abstract: Multimodal stroke recurrence prediction requires effective integration of heterogeneous clinical and imaging data, yet modality imbalance often causes m

22 Sept 2026

GRRR: The Geometry of Reshaping, Rotation, and Routing in Decoder LLM post-training

arXiv:2609.22146v1 Announce Type: cross Abstract: We study how post-training changes the weights of Large Language Models (LLMs) relative to their pretrained weights. Across 12 post-training chains wi

researcharxiv-cs-cl

22 Sept 2026

MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training

model-releasesarxiv-cs-cl

arXiv:2609.15037v1 Announce Type: cross Abstract: Low-rank gradient projection reduces the optimizer-state memory cost of large language model (LLM) pretraining, but the steps and wall-clock time need

15 Sept 2026

Refusal Reads Only a Slice of What the Model Knows: Harm-Keyed Routing and Its Exceptions Across Model Families

arXiv:2609.14759v1 Announce Type: cross Abstract: Alignment applied after pretraining is shallow in a measurable way: a single direction in a model's residual stream can be edited out, and the model s

15 Sept 2026

DATAFARM: Distribution-Aligned Task and Motion Planning for Fine-Tuning Vision-Language-Action Models

arXiv:2609.12316v1 Announce Type: new Abstract: Collecting high-quality robot data remains a fundamental challenge for training robot foundation models. Task and motion planning (TAMP) offers a scalab

researcharxiv-cs-ro

14 Sept 2026

DR-LabStack: Design and Implementation of a Clinician-Facing Web System for Diabetic Retinopathy Prediction

arXiv:2609.10796v1 Announce Type: new Abstract: Pretrained diabetic retinopathy (DR) prediction models differ in their input fields, serialization formats, preprocessing requirements, and output seman

researcharxiv-cs-lg

11 Sept 2026

NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction

model-releasesarxiv-cs-cl

arXiv:2609.10715v1 Announce Type: new Abstract: We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP). Alongsid

11 Sept 2026

Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

arXiv:2609.06807v1 Announce Type: new Abstract: In this work, we comprehensively evaluate three popular feature-extraction paradigms in AI-based neuroimaging modeling: (1) computation of anatomical su

tutorialsarxiv-cs-cv

9 Sept 2026

From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models

arXiv:2609.06912v1 Announce Type: new Abstract: Tabular foundation models (TFMs) are commonly pretrained on large collections of procedurally generated synthetic tasks, yet it remains unclear how well

9 Sept 2026

Hardware

Text Capability Loss in Vision-Language Adaptation: An Attention-Sink Diagnosis

arXiv:2609.00746v1 Announce Type: new Abstract: Fine-tuning a pretrained LLM into a vision-language model (VLM) can erode the backbone's text capability, with the damage concentrated on tasks that req

hardwarearxiv-cs-lg

2 Sept 2026

RIBOSPAN: A Long-Context RNA Foundation Model for Versatile RNA Modeling

arXiv:2608.22849v1 Announce Type: new Abstract: Full-length RNAs, particularly messenger RNAs, often exceed the context lengths used to pretrain existing RNA foundation models, limiting complete-trans

25 Aug 2026

Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts

arXiv:2608.16614v1 Announce Type: new Abstract: Geospatial Foundation Models (GeoFMs) are most commonly ranked and selected by accuracy on standard benchmark conditions via averaged ranks. We show tha

18 Aug 2026

G0.5: One Autoregressive Stream for Robot Reasoning and Action

arXiv:2608.11739v1 Announce Type: cross Abstract: The prevailing recipe for Vision-Language-Action (VLA) models couples a pretrained VLM with a separately trained flow-matching action expert. This mak