Search: “pretraining”

arXiv:2609.37255v1 Announce Type: cross Abstract: Time series foundation models (TSFMs) are pretrained on heterogeneous collections containing billions of observations, yet their training windows are

local-aiarxiv-cs-ai

30 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Tutorials

Pretraining Latent Information Feedback Transformers with Teacher Supervision

DGX agent

arXiv:2609.38149v1 Announce Type: new Abstract: Transformer language models (LMs) are feed-forward: deep-layer representations are never fed back to shallower layers, and the only pathway for informat

tutorialsarxiv-cs-cl

30 Sept 2026

Agents

ReScraper: Unified Scraping and Cleaning of Web Data for Effective LLM Pretraining

arXiv:2609.34287v1 Announce Type: cross Abstract: LLM pretraining corpora are normally cleaned by a stack of hand-written heuristics. A heuristic scraper extracts the main content from HTML, and dozen

agentsarxiv-cs-lg

29 Sept 2026

SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

arXiv:2609.34681v1 Announce Type: new Abstract: Learning-rate (LR) scheduling plays a central role in large language model (LLM) pretraining, yet current practice still relies heavily on hand-crafted

29 Sept 2026

Tutorials

Improving Molecular-Morphology Contrastive Pretraining using Deep-Learning-based Morphology Profiles

arXiv:2609.30433v1 Announce Type: new Abstract: Recent advancements in image-based profiling techniques have enabled the collection of high-volume cell morphology data, allowing new molecular embeddin

tutorialsarxiv-cs-lg

28 Sept 2026

Tutorials

Self-Play Pretraining with Zero Data

arXiv:2609.30063v1 Announce Type: new Abstract: Advances in language modeling have been driven by scaling pretraining on ever more data. Yet, the training data is still largely curated on the model's

tutorialsarxiv-cs-ai

25 Sept 2026

PermuFormer: Multi-Task Pretraining for Permutation Representation in Algebraic Combinatorics

arXiv:2609.25438v1 Announce Type: new Abstract: Diverse pretraining has been shown to be an effective method for learning reusable, domain-aware representations that provide a starting point for fine-

23 Sept 2026

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining

arXiv:2505.19893v2 Announce Type: replace-cross Abstract: Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introd

22 Sept 2026

Multi-Hop Knowledge Composition is Bound by Pretraining Exposure

arXiv:2606.09338v2 Announce Type: replace Abstract: Large Language Models fail at implicit multi-hop reasoning: a model answers 'When was X born?' and 'Who is Y's closest friend?' correctly but fails

17 Sept 2026

A Later Test Set Is Not a New Domain: Pretraining Familiarity Survives a Contamination-Free Hold-Out

arXiv:2609.10357v1 Announce Type: new Abstract: Time-series foundation models are evaluated almost exclusively on public archives that predate them, so a strong score cannot be separated from having s

10 Sept 2026

Good Pretraining, Bad SFT: Checkpoint Quality Across the Training Stack

arXiv:2609.08966v1 Announce Type: new Abstract: Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the b

10 Sept 2026

Large-Scale Pretraining for Improving Deep Learning-Based Geometric Distortion Correction of Diffusion-Weighted Imaging

arXiv:2609.06437v1 Announce Type: new Abstract: Diffusion-weighted imaging (DWI) is widely used in clinical settings but remains vulnerable to geometric distortion. Conventional correction methods oft

researcharxiv-cs-cv

9 Sept 2026

The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists

arXiv:2609.06934v1 Announce Type: new Abstract: Post-hoc safety training (RLHF, DPO) is the dominant way to align large language models, yet jailbreaks (Zou et al., 2023b), fine-tuning attacks (Qi et

9 Sept 2026

VidaForge: Open Research Infrastructure for Video Pretraining Data Recipes

model-releasesarxiv-cs-cv

arXiv:2609.06652v1 Announce Type: new Abstract: Video foundation models increasingly rely on large-scale pretraining data, yet the end-to-end data pipelines behind them remain largely closed and diffi

9 Sept 2026

UE5M3 FP4 Block Scaling for Stable Language Model Pretraining

arXiv:2609.02846v1 Announce Type: new Abstract: Stable 4-bit floating-point (FP4) pretraining is difficult because the E2M1 payload represents only a narrow range of magnitudes. NVIDIA's Transformer E

3 Sept 2026

Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

arXiv:2608.28442v1 Announce Type: new Abstract: Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditio

31 Aug 2026

One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

arXiv:2608.25904v1 Announce Type: new Abstract: Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages

27 Aug 2026

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

arXiv:2608.24814v1 Announce Type: new Abstract: We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the eff

26 Aug 2026

Measuring Task-Agnostic Training Data Influence Across Language Model Pretraining

arXiv:2608.13515v1 Announce Type: new Abstract: Measuring training data influence consistently across language model pretraining is challenging. It is difficult to select downstream tasks or validatio

14 Aug 2026

AgriField-40K: Adapting Vision Models to Agriculture With Efficient Continual Pretraining

model-releasesarxiv-cs-cv

arXiv:2608.07984v1 Announce Type: new Abstract: Field-based agricultural computer vision is important for precision agriculture, yet it largely depends on expensive annotations and costly adaptation o

11 Aug 2026

To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval

arXiv:2604.00715v2 Announce Type: replace-cross Abstract: Retrieval-augmented generation (RAG) improves language model (LM) performance by providing relevant context at test time for knowledge-intensi

11 Aug 2026

Robust-WAM: Bridging Generative Pretraining and Semantic Foresight in World-Action Models

arXiv:2608.05903v1 Announce Type: new Abstract: Mainstream World-Action Models (WAMs) adapt pretrained video generation models (VGMs) for robot control, transferring their learned dynamics prior for a

7 Aug 2026

Curriculum Matters: Data-Efficient Relational PFN Pretraining with Synthetic Data

arXiv:2607.29120v1 Announce Type: new Abstract: Relational Prior-Data Fitted Networks (PFNs) such as RDB-PFN approximate Bayesian inference over multi-table relational databases by pretraining on mill

3 Aug 2026

PluRel-to-RDB-PFN: Schema-Guided Synthetic Relational Pretraining

arXiv:2607.29129v1 Announce Type: new Abstract: Relational Foundation Models (RFMs) require large-scale synthetic relational databases for pretraining, but existing approaches tightly couple data gene

3 Aug 2026

Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT

arXiv:2607.25063v1 Announce Type: new Abstract: Developers judge a model checkpoint by how it behaves. After supervised fine-tuning (SFT), two checkpoints that perform about the same across relevant b

29 Jul 2026

Phylogenetic signal in marine mammal and bird vocalizations captured by audio foundation models: the limited benefit of domain-specific pretraining

arXiv:2607.22458v1 Announce Type: new Abstract: Do learned audio embeddings encode structure that nobody told them to encode? We probe four large pretrained audio models (AST, CLAP, BEATs-bio and Bird

27 Jul 2026

Fine-Grained Vision-Language Pretraining with Organ-Conditioned Pattern Tokens for CT Understanding

arXiv:2607.13892v1 Announce Type: new Abstract: Computed tomography (CT) vision-language pretraining from paired volumes and radiology reports is a scalable yet challenging task. Existing methods comm

16 Jul 2026

Pretraining in Actor-Critic Reinforcement Learning for Locomotion

arXiv:2510.12363v4 Announce Type: replace-cross Abstract: The pretraining-finetuning paradigm has facilitated numerous transformative advancements in artificial intelligence research in recent years.

safetyarxiv-cs-lg

16 Jul 2026

Self-Supervised Pretraining Improves Cross-Site and Cross-Scale Robustness of Point Cloud Leaf-Wood Segmentation

arXiv:2607.06948v1 Announce Type: cross Abstract: The accuracy of existing leaf-wood segmentation methods for tree point clouds varies across forest types and sites. Self-supervised learning (SSL) on

9 Jul 2026

Curriculum-Guided Layer Scaling for Language Model Pretraining

model-releasesarxiv-cs-cl

arXiv:2506.11389v4 Announce Type: replace Abstract: As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core tr

7 Jul 2026

LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives

arXiv:2607.00784v1 Announce Type: cross Abstract: Vision-language pretraining remains dominated by contrastive objectives, whereas vision-only self-supervised learning has largely adopted non-contrast

2 Jul 2026

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Off

arXiv:2510.01163v2 Announce Type: replace Abstract: The factors driving the performance of in-context learning (ICL) in large language models (LLMs) remain poorly understood despite ICL's surprising e

25 Jun 2026

Removing Noise, not Finding Gold: Quality Filtering for Large-Scale Pretraining

arXiv:2510.00866v3 Announce Type: replace-cross Abstract: Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A po

24 Jun 2026

Universal priors: solving empirical Bayes via Bayesian inference and pretraining

arXiv:2602.15136v2 Announce Type: replace-cross Abstract: We theoretically justify the recent empirical finding of [Teh et al., 2025] that a transformer pretrained on synthetically generated data achi

23 Jun 2026

Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

arXiv:2606.22079v1 Announce Type: cross Abstract: Web data curation has been widely studied for decoder Large Language Model (LLM) pretraining. Encoders for dense-terminology domains such as medicine,

23 Jun 2026

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

arXiv:2606.12366v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulatio

safetyarxiv-cs-ro

11 Jun 2026

Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

arXiv:2606.11499v1 Announce Type: cross Abstract: The performance of modern language models depends critically on pretraining data composition. Yet existing data selection methods rely on auxiliary cl

11 Jun 2026

Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining

arXiv:2606.09912v1 Announce Type: cross Abstract: Choosing the wrong synthetic generator for time-series foundation model pretraining is costly: under identical training budgets, the best and worst ge

10 Jun 2026

Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails

arXiv:2606.05936v1 Announce Type: new Abstract: Modern language models rely on pretraining filters to remove undesirable content from training corpora and inference-time guardrails to suppress undesir

safetyarxiv-cs-cl

5 Jun 2026

Hardware

Speedrunning Tabular Foundation Model Pretraining

arXiv:2606.03681v1 Announce Type: new Abstract: Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimiz

hardwarearxiv-cs-lg

3 Jun 2026

Local Ai

Latent Diffusion Pretraining for Crystal Property Prediction

arXiv:2606.00776v1 Announce Type: new Abstract: Fast and accurate prediction of crystal properties is a central challenge in new materials design. Graph neural networks and Transformer-based models ha

local-aiarxiv-cs-lg

2 Jun 2026

Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications

arXiv:2605.26133v1 Announce Type: cross Abstract: Large Language Models (LLMs) have become the predominant paradigm in NLP, advancing both research and industry. As model sizes and pretraining data gr

27 May 2026

TypedCSIP: Typed Counterfactual Pretraining for Chinese Legislative Conflict Classification

model-releasesarxiv-cs-cl

arXiv:2605.25474v1 Announce Type: new Abstract: TypedCSIP is a typed counterfactual pretraining method for the conflict-classification task of the LCR-CN benchmark (Zhao et al., 2026): given a (superi

26 May 2026

Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks

arXiv:2605.18491v1 Announce Type: new Abstract: Methods: Nine SSL methods spanning four pretext-task families were pretrained from scratch using the same 10{,}412 3D CT scans (1.89~M 2D axial slices)

19 May 2026

GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretraining

model-releasesarxiv-cs-cv

arXiv:2605.16769v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as a promising paradigm for adapting pretrained models under limited data conditions. However, most e

19 May 2026

Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction

arXiv:2605.16088v1 Announce Type: cross Abstract: Self-supervised pretraining on molecular graphs has emerged as a promising approach for molecular property prediction, yet most existing methods opera

18 May 2026

SpectraFlow: Unifying Structural Pretraining and Frequency Adaptation for Medical Image Segmentation

arXiv:2605.14566v1 Announce Type: new Abstract: Medical image segmentation remains challenging in low-data regimes, where scarce annotations often yield poor generalization and ambiguous boundaries wi

15 May 2026

A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup

arXiv:2605.11554v1 Announce Type: new Abstract: Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another, but such explanations require the