Search: “pretraining”

arXiv:2508.15706v3 Announce Type: replace Abstract: Communication-efficient distributed training algorithms (e.g., DiLoCo) have received considerable interest due to their benefits for training large

local-aiarxiv-cs-lg

24 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Modular Pretraining Enables Access Control

DGX agent

arXiv:2607.08077v1 Announce Type: new Abstract: AI developers face a dual-use dilemma. An AI capability that helps one user cure a disease can help another synthesize one. This dilemma could be resolv

researcharxiv-cs-lg

10 Jul 2026

GaussFusion: Towards Multimodal 3D Gaussian Pretraining

arXiv:2607.05906v1 Announce Type: new Abstract: 3D Gaussian Splatting provides an explicit representation that jointly models geometry and appearance, serving as a scalable foundation for 3D represent

safetyarxiv-cs-cv

8 Jul 2026

Predicting the Emergence of Induction Heads in Language Model Pretraining

arXiv:2511.16893v3 Announce Type: replace Abstract: Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern lang

7 Jul 2026

CLIMP: Contrastive Language-Image Mamba Pretraining

arXiv:2601.06891v2 Announce Type: replace Abstract: Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and

30 Jun 2026

Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

arXiv:2606.25546v1 Announce Type: new Abstract: Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet e

25 Jun 2026

Lightweight 3D Feature Pretraining by Bayesian Inversion of 2D Foundation Models

arXiv:2606.21292v1 Announce Type: new Abstract: We present Casper3D, a lightweight probabilistic framework for converting noisy multi-view 2D foundation-model embeddings into a latent 3D semantic repr

23 Jun 2026

Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization

arXiv:2606.21271v1 Announce Type: new Abstract: Sparse rewards pose a central challenge in reinforcement learning, since agents receive no informative signal until they reach their goal. Intrinsic-rew

23 Jun 2026

Local Ai

Unifying Local Communications and Local Updates for LLM Pretraining

arXiv:2606.11081v1 Announce Type: cross Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and low

local-aiarxiv-cs-ai

10 Jun 2026

CDPM-Align: Multi-Scale Guidance-Aligned Diffusion Pretraining for Robust Few-Shot Anatomical Landmark Detection

arXiv:2606.04898v1 Announce Type: new Abstract: Anatomical landmark detection is a fundamental task in medical image analysis supporting a wide range of diagnostic and interventional workflows. Althou

4 Jun 2026

Pretraining Language Models on Historical Text

arXiv:2606.02991v1 Announce Type: cross Abstract: We introduce TypewriterLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913. Developing History LMs requires add

3 Jun 2026

T-CLIP: Enabling Thermal Perception for Contrastive Language-Image Pretraining

arXiv:2606.00673v1 Announce Type: new Abstract: Thermal imaging offers a powerful alternative to visible-spectrum vision under challenging conditions such as low illumination and adverse weather, yet

2 Jun 2026

Tutorials

Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models

arXiv:2605.28132v1 Announce Type: new Abstract: Spatial intelligence requires visual representations that capture both semantic objects and geometric structure in the physical world. To support this,

tutorialsarxiv-cs-cv

28 May 2026

A comprehensive evaluation of pretraining strategies for channel-agnostic contrastive self-supervision of biosignals

arXiv:2410.19842v2 Announce Type: replace-cross Abstract: Contrastive learning yields impressive results for self-supervision in computer vision. The approach relies on the creation of positive pairs,

25 May 2026

MuellerPT: Decomposition Driven Pretraining for Dense Learning in Mueller Polarimetry

arXiv:2605.23840v1 Announce Type: new Abstract: Mueller matrix imaging provides rich, physically meaningful contrast for biomedical tissue analysis, but supervised learning is hindered by scarce dense

25 May 2026

MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency

arXiv:2510.25897v2 Announce Type: replace Abstract: The default paradigm of post-training text-to-image generators includes post-hoc selection of generated images, and subsequent training with one rew

20 May 2026

Protein Fold Classification at Scale: Benchmarking and Pretraining

model-releasesarxiv-cs-lg

arXiv:2605.18552v1 Announce Type: new Abstract: Classifying protein topology is essential for deciphering biological function, but progress is held back by the lack of large-scale benchmarks that avoi

Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?

arXiv:2511.08704v2 Announce Type: replace Abstract: This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unifie

Universal Pose Pretraining for Generalizable Vision-Language-Action Policies

arXiv:2602.19710v2 Announce Type: replace Abstract: Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency because they entangle high-level percept

safetyarxiv-cs-cv

Agents

Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining

arXiv:2605.14747v1 Announce Type: cross Abstract: Recent advances in multimodal large language models have driven growing interest in graphical user interface (GUI) agents, yet their generalization re

agentsarxiv-cs-ai

15 May 2026

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

arXiv:2605.12438v1 Announce Type: new Abstract: When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily sw

MambaNetBurst: Direct Byte-level Network Traffic Classification without Tokenization or Pretraining

arXiv:2605.11034v1 Announce Type: cross Abstract: We present MambaNetBurst, a compact tokenizer-free byte-level sequence classifier for network burst classification based on a Mamba-2 backbone. In con

MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification

arXiv:2605.11408v1 Announce Type: cross Abstract: Tabular data forms the backbone of high-stakes decision systems in finance, healthcare, and beyond. Yet industrial tabular datasets are inherently dif

applicationsarxiv-cs-cl

Prompting from the bench: Large-scale pretraining is not sufficient to prepare LLMs for ordinary meaning analysis

arXiv:2510.25356v2 Announce Type: replace Abstract: In the U.S. judicial system, a widespread approach to legal interpretation entails assessing how a legal text would be understood by an `ordinary' s

applicationsarxiv-cs-cl

TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuning

arXiv:2605.12236v1 Announce Type: cross Abstract: Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral clon

safetyarxiv-cs-lg

Finite-Size Gradient Transport in Large Language Model Pretraining: From Cascade Size to Intensive Transport Efficiency

arXiv:2605.02968v1 Announce Type: new Abstract: We introduce a finite-size gradient-transport framework for real language-model training, based on five observables (D,z,eta,elta,v_{rel}) that separate

6 May 2026

Label-Efficient School Detection from Aerial Imagery via Weakly Supervised Pretraining and Fine-Tuning

arXiv:2605.03968v1 Announce Type: new Abstract: Accurate school detection is essential for supporting education initiatives, including infrastructure planning and expanding internet connectivity to un

6 May 2026

Length Value Model: Scalable Value Pretraining for Token-Level Length Modeling

arXiv:2604.27039v1 Announce Type: new Abstract: Token serves as the fundamental unit of computation in modern autoregressive models, and generation length directly influences both inference cost and r

1 May 2026

UniDomain: Pretraining a Unified PDDL Domain from Real-World Demonstrations for Generalizable Robot Task Planning

arXiv:2507.21545v3 Announce Type: replace Abstract: Robotic task planning in real-world environments requires reasoning over implicit constraints from language and vision. While LLMs and VLMs offer st

applicationsarxiv-cs-ro

21 Apr 2026

End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering

arXiv:2511.09282v3 Announce Type: replace-cross Abstract: Significant progress has been made in spoken question answering (SQA) in recent years. However, many existing methods, including large audio l

safetyarxiv-cs-cl

14 Apr 2026

One Leak Away: How Pretrained Model Exposure Amplifies Jailbreak Risks in Finetuned LLMs

arXiv:2512.14751v3 Announce Type: replace-cross Abstract: Finetuning pretrained large language models (LLMs) has become the standard paradigm for developing downstream applications. However, its secur

7 Aug 2026

Statistical Benefits of Fine-Tuning from Pretrained Initialization in Diagonal Linear Networks

arXiv:2609.34756v1 Announce Type: cross Abstract: Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread pr

safetyarxiv-cs-lg

29 Sept 2026

Exploiting Stability-Plasticity Asymmetry in Pretrained Detectors for Incremental Object Detection

arXiv:2504.10214v2 Announce Type: replace Abstract: Pretrained model-based incremental object detection (PTMIOD) leverages the rich detection priors of pretrained detectors to learn new categories inc

2 Sept 2026

Hardware

STAPLE: Static Sparse Upcycling of Pretrained Checkpoints via Embedding Modules

arXiv:2610.12599v1 Announce Type: new Abstract: Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures

hardwarearxiv-cs-lg

12 Oct 2026

NeuroPB: Scaling Neural Decoding with Pretrained Behavioral Representations

arXiv:2608.04389v1 Announce Type: new Abstract: Decoding continuous motor trajectories from neural activity is essential for developing practical brain-computer interfaces (BCIs). However, current neu

6 Aug 2026

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?

arXiv:2607.27203v1 Announce Type: new Abstract: Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) thi

safetyarxiv-cs-lg

30 Jul 2026

Tutorials

Learning Decomposed Contextual Token Representations from Pretrained and Collaborative Signals for Generative Recommendation

arXiv:2509.10468v2 Announce Type: replace-cross Abstract: Recent advances in generative recommenders adopt a two-stage paradigm: items are first tokenized into semantic IDs using a pretrained tokenize

tutorialsarxiv-cs-cl

5 May 2026

Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent

arXiv:2609.16262v1 Announce Type: cross Abstract: Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objec

16 Sept 2026

Parameter-Efficient Adaptation of Pretrained Language Models for Time-Series Forecasting

arXiv:2609.15344v1 Announce Type: new Abstract: We study the adaptation of pretrained language models to univariate time-series forecasting through a parameter-efficient transfer learning framework, w

15 Sept 2026

Tutorials

BrainTaskonomy: Learning How to Pretrain and What to Transfer in fMRI Foundation Models

arXiv:2609.10518v1 Announce Type: new Abstract: fMRI foundation models increasingly aggregate heterogeneous data across brain states, cohorts, and acquisition settings, yet pretraining domains are com

tutorialsarxiv-cs-cv

10 Sept 2026

When Less Compute Is More: Adaptive Early Exit Improves Pretrained Outlier Detection

arXiv:2609.32898v1 Announce Type: new Abstract: Pretrained tabular foundation models process every dataset at a fixed depth, with inference costs growing with dataset size. To address this, we present

applicationsarxiv-cs-lg

29 Sept 2026

PORL: Pretrained Offline Reinforcement Learning for the Job Shop Scheduling Problem

arXiv:2609.30948v1 Announce Type: cross Abstract: The Job Shop Scheduling Problem (JSSP) is a fundamental combinatorial optimization problem in industrial optimization. This work introduces Pretrained

safetyarxiv-cs-ai

28 Sept 2026

PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention

arXiv:2607.25157v1 Announce Type: new Abstract: Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requi

researcharxiv-cs-ai

29 Jul 2026

Multisensory Continual Learning: Adapting Pretrained Visuomotor Policies to Force

arXiv:2606.30988v1 Announce Type: new Abstract: Robot manipulation often relies on sensory feedback beyond vision, particularly in contact-rich settings where force, tactile, or audio signals reveal i

safetyarxiv-cs-ro

1 Jul 2026

Clustering Guided Domain-Specific Pretrained Foundation Model Very High-Resolution Arctic Remote Sensing

arXiv:2605.30467v1 Announce Type: new Abstract: This study introduces a novel Arctic-focused remote sensing foundation model (RSFM) by combining diversity-aware regional-scale image curation with mask

1 Jun 2026

Extending Pretrained 10-Second ECG Foundation Models to Longer Horizons

arXiv:2605.16975v1 Announce Type: cross Abstract: Electrocardiogram (ECG) foundation models pretrained on typical diagnostic 10-second ECG segments, have demonstrated strong transferability across a r

Teacher-Feature Drifting: One-Step Diffusion Distillation with Pretrained Diffusion Representations

arXiv:2605.07327v1 Announce Type: new Abstract: Sampling from pretrained diffusion and flow-matching models typically requires many forward passes to generate diverse and high-fidelity images. Existin

11 May 2026

Should We Still Pretrain Encoders with Masked Language Modeling?

arXiv:2507.00994v4 Announce Type: replace Abstract: Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked