Search: “pretraining”

Universal priors: solving empirical Bayes via Bayesian inference and pretraining

arXiv:2602.15136v2 Announce Type: replace-cross Abstract: We theoretically justify the recent empirical finding of [Teh et al., 2025] that a transformer pretrained on synthetically generated data achi

23 Jun 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Where Does the Signal Live? A Web Data Recipe for Medical Encoder Pretraining

DGX agent

arXiv:2606.22079v1 Announce Type: cross Abstract: Web data curation has been widely studied for decoder Large Language Model (LLM) pretraining. Encoders for dense-terminology domains such as medicine,

model-releasesarxiv-cs-lg

23 Jun 2026

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

arXiv:2606.12366v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulatio

safetyarxiv-cs-ro

11 Jun 2026

Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality

model-releasesarxiv-cs-ai

arXiv:2606.11499v1 Announce Type: cross Abstract: The performance of modern language models depends critically on pretraining data composition. Yet existing data selection methods rely on auxiliary cl

11 Jun 2026

Mix, Don't Pick: Why Synthetic Corpus Composition Matters for Time Series Foundation Model Pretraining

arXiv:2606.09912v1 Announce Type: cross Abstract: Choosing the wrong synthetic generator for time-series foundation model pretraining is costly: under identical training budgets, the best and worst ge

10 Jun 2026

Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails

arXiv:2606.05936v1 Announce Type: new Abstract: Modern language models rely on pretraining filters to remove undesirable content from training corpora and inference-time guardrails to suppress undesir

safetyarxiv-cs-cl

5 Jun 2026

Hardware

Speedrunning Tabular Foundation Model Pretraining

arXiv:2606.03681v1 Announce Type: new Abstract: Pretraining cost is a major bottleneck for research on tabular foundation models, slowing the iteration cycle for new architectures, priors, and optimiz

hardwarearxiv-cs-lg

3 Jun 2026

Local Ai

Latent Diffusion Pretraining for Crystal Property Prediction

arXiv:2606.00776v1 Announce Type: new Abstract: Fast and accurate prediction of crystal properties is a central challenge in new materials design. Graph neural networks and Transformer-based models ha

local-aiarxiv-cs-lg

2 Jun 2026

Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications

arXiv:2605.26133v1 Announce Type: cross Abstract: Large Language Models (LLMs) have become the predominant paradigm in NLP, advancing both research and industry. As model sizes and pretraining data gr

27 May 2026

TypedCSIP: Typed Counterfactual Pretraining for Chinese Legislative Conflict Classification

arXiv:2605.25474v1 Announce Type: new Abstract: TypedCSIP is a typed counterfactual pretraining method for the conflict-classification task of the LCR-CN benchmark (Zhao et al., 2026): given a (superi

26 May 2026

Benchmarking transferability of SSL pretraining to same and different modality segmentation tasks

arXiv:2605.18491v1 Announce Type: new Abstract: Methods: Nine SSL methods spanning four pretext-task families were pretrained from scratch using the same 10{,}412 3D CT scans (1.89~M 2D axial slices)

safetyarxiv-cs-cv

19 May 2026

GLT-PEFT: Gated Lie-Tucker Parameter-Efficient Fine-Tuning for Alzheimer's Disease Diagnosis with Hippocampal Segmentation Pretraining

model-releasesarxiv-cs-cv

arXiv:2605.16769v1 Announce Type: new Abstract: Parameter-efficient fine-tuning (PEFT) has emerged as a promising paradigm for adapting pretrained models under limited data conditions. However, most e

19 May 2026

Multi-level Self-supervised Pretraining on Compositional Hierarchical Graph for Molecular Property Prediction

arXiv:2605.16088v1 Announce Type: cross Abstract: Self-supervised pretraining on molecular graphs has emerged as a promising approach for molecular property prediction, yet most existing methods opera

18 May 2026

SpectraFlow: Unifying Structural Pretraining and Frequency Adaptation for Medical Image Segmentation

arXiv:2605.14566v1 Announce Type: new Abstract: Medical image segmentation remains challenging in low-data regimes, where scarce annotations often yield poor generalization and ambiguous boundaries wi

safetyarxiv-cs-cv

15 May 2026

A Controlled Counterexample to Strong Proxy-Based Explanations of OOD Performance: in a Fixed Pretraining-and-Probing Setup

arXiv:2605.11554v1 Announce Type: new Abstract: Task-agnostic structure proxies are often used to interpret why one pretraining corpus transfers better than another, but such explanations require the

13 May 2026

When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining

arXiv:2605.07756v1 Announce Type: cross Abstract: Modern deep models are often pretrained on large-scale data with missing labels using composite objectives, where the relative weights of multiple los

11 May 2026

Cross-Scale Pretraining: Enhancing Self-Supervised Learning for Low-Resolution Satellite Imagery for Semantic Segmentation

arXiv:2601.12964v2 Announce Type: replace Abstract: Self-supervised pretraining in remote sensing is mostly done using mid-spatial resolution (MR) image datasets due to their high availability. Given

tutorialsarxiv-cs-cv

5 May 2026

PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training

arXiv:2604.22117v1 Announce Type: cross Abstract: Aligned large language models(LLMs) remain vulnerable to adversarial manipulation, and their dependence on web-scale pretraining creates a subtle but

27 Apr 2026

In Search of Lost DNA Sequence Pretraining

arXiv:2604.16570v1 Announce Type: new Abstract: DNA sequence encoding is fundamental to gene function prediction, protein synthesis, and diverse downstream biological tasks. Despite the substantial pr

21 Apr 2026

AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretraining

arXiv:2610.20086v1 Announce Type: new Abstract: Synthetic data is increasingly used in language-model pretraining. Existing knowledge-corpus synthesis pipelines, including WRAP and Nemotron-CC, usuall

15 Oct 2026

Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models

model-releasesarxiv-cs-lg

arXiv:2610.17590v1 Announce Type: new Abstract: Diffusion language models (DLMs) have attracted wide attention for their fast, parallel decoding, but they are believed to be far less efficient to trai

14 Oct 2026

Host-Aware Web Crawling for LLM Pretraining Data

arXiv:2610.13321v1 Announce Type: new Abstract: Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We i

tutorialsarxiv-cs-cl

12 Oct 2026

Stochastic Siamese MAE Pretraining for Longitudinal Medical Images

arXiv:2512.23441v2 Announce Type: replace-cross Abstract: Temporally aware image representations are crucial for capturing disease progression in 3D volumes of longitudinal medical datasets. However,

tutorialsarxiv-cs-cv

7 Oct 2026

ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining

arXiv:2603.02767v4 Announce Type: replace-cross Abstract: Image--text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield repres

1 Oct 2026

Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior

model-releasesarxiv-cs-ai

arXiv:2609.39827v1 Announce Type: cross Abstract: Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes

1 Oct 2026

Does Joint-Embedding Predictive Architecture Pretraining Help Time Series Forecasting?

arXiv:2609.31680v1 Announce Type: new Abstract: Joint-embedding predictive architectures (JEPA) have emerged as a promising self-supervised pretraining paradigm for time series, learning representatio

Relative Generalization Invariance of LLM Pretraining

arXiv:2609.33016v1 Announce Type: new Abstract: Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and

SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis

arXiv:2609.34479v1 Announce Type: new Abstract: Vision-language (VL) pretraining using paired chest X-ray (CXR) images and radiology reports has shown strong potential for medical image understanding.

researcharxiv-cs-cv

SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

model-releasesarxiv-cs-lg

arXiv:2609.35440v1 Announce Type: new Abstract: Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait fo

Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

arXiv:2609.35219v1 Announce Type: new Abstract: Relational deep learning models database rows and foreign-key links as a heterogeneous graph for prediction from record attributes and relational contex

Applications

A generalizable structural brain MRI foundation model built through dual-priority federated pretraining

arXiv:2609.27611v1 Announce Type: new Abstract: Foundation models hold promise for generalizable analysis of structural brain magnetic resonance imaging (MRI) across development, aging and disease. Ho

applicationsarxiv-cs-cv

24 Sept 2026

GDLAM: Group-Disentangled Latent Action Model for Highly Disentangled Embodied Pretraining

arXiv:2609.26118v1 Announce Type: new Abstract: Latent action models (LAMs) learn action-related representations from action-free videos via self-supervised future prediction, offering a scalable para

safetyarxiv-cs-ro

23 Sept 2026

Detecting Pretraining Data in Large Language Models from a Free-Energy Perspective

arXiv:2609.21888v1 Announce Type: cross Abstract: Detecting pretraining data in large language models is challenging because high likelihood can reflect either training exposure or strong generalizati

21 Sept 2026

From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

arXiv:2609.21788v1 Announce Type: cross Abstract: A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional ful

safetyarxiv-cs-lg

21 Sept 2026

Applications

LMEnt: A Suite for Analyzing Knowledge in Language Models from Pretraining Data to Representations

arXiv:2509.03405v2 Announce Type: replace Abstract: Language models (LMs) increasingly drive real-world applications that require world knowledge. However, the internal processes through which models

applicationsarxiv-cs-cl

18 Sept 2026

Measuring Annotation Efficiency for Handwritten Devanagari Recognition: Sample-Complexity Curves for Four Pretraining Regimes

arXiv:2609.16859v1 Announce Type: new Abstract: To train handwritten text recognition systems we need word images and their corresponding transcriptions, and these transcriptions are produced manually

researcharxiv-cs-cv

16 Sept 2026

Tabby: An Open Pretraining Recipe for Time Series Foundation Models

model-releasesarxiv-cs-lg

arXiv:2609.13956v1 Announce Type: new Abstract: In this report, we release Tabby, a long context probabilistic time series foundation model, together with a complete and open recipe of how it was buil

15 Sept 2026

Hardware

BRIDGE-EEG: Bridging Self-Supervised Pretraining and Efficient Deployment for Cross-Dataset EEG Classification

arXiv:2609.12218v1 Announce Type: cross Abstract: The growing use of electroencephalography (EEG) motivates automated analysis that is accurate, transferable, and deployable on constrained hardware. R

hardwarearxiv-cs-lg

14 Sept 2026

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation

arXiv:2609.05588v1 Announce Type: cross Abstract: World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction.

tutorialsarxiv-cs-cv

9 Sept 2026

Applications

OpenWAM: An Open, Modular Exploration Towards Systematic World-Action Model Pretraining

arXiv:2609.07398v1 Announce Type: new Abstract: World-Action Models inherit world knowledge from video-generative priors, and channel it into executable control signals through embodied experience. Ex

applicationsarxiv-cs-ro

9 Sept 2026

DISTAL: Distillation and Self-Supervised Pretraining for Structure-Agnostic Materials Property Prediction

model-releasesarxiv-cs-ai

arXiv:2609.00059v1 Announce Type: cross Abstract: Materials property prediction remains difficult in low-data settings, where many target properties are supported by only a limited number of labeled s

2 Sept 2026

EEG-VID: Task-Guided Latent Predictive Pretraining for EEG Decoding and Assistive Target Selection

arXiv:2609.00566v1 Announce Type: cross Abstract: We propose EEG-VID, a task-guided latent predictive pretraining framework for EEG decoding under session and subject shifts. EEG-VID predicts future l

2 Sept 2026

Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

arXiv:2608.26576v1 Announce Type: new Abstract: When researchers compare multilingual models for probing, interpretability, or cross-lingual transfer, they often align embedding spaces and assume that

28 Aug 2026

Robust Neural Stimulation Response Modeling Through Meta-Learning and Pretraining

arXiv:2608.26649v1 Announce Type: new Abstract: Objective: Model-based closed-loop neural stimulation holds promise for therapeutic applications ranging from Parkinson's disease to sensory restoration

28 Aug 2026

PaSTel: Anchoring Histology in Spatial Transcriptomics via Multi-Scale Hierarchical Bio-Prior Contrastive Pretraining

arXiv:2608.14924v1 Announce Type: cross Abstract: Spatial transcriptomics (ST) links tissue morphology with molecular programs, motivating multimodal pretraining methods that align histology images wi

18 Aug 2026

Fine-Tuning Qwen3-27B for C-to-Rust Code Translation: A Three-Stage Curriculum of Pretraining, Debugging-Aware SFT, and Task-Specific SFT

arXiv:2608.13681v1 Announce Type: cross Abstract: Translating C code into safe, idiomatic Rust is a longstanding software-engineering goal because it can eliminate entire classes of memory-safety vuln

17 Aug 2026

A Controlled Study of Self-Supervised Image and Video Pretraining under Limited Resources

arXiv:2608.13183v1 Announce Type: new Abstract: Visual foundation models are a cornerstone of image and video understanding but typically require large amounts of data and computation. The current sca

researcharxiv-cs-cv

14 Aug 2026

Reducing Pretraining-Generation Mismatch in Diffusion Language Models

arXiv:2608.09424v1 Announce Type: new Abstract: Autoregressive language models align training and use: generation conditions on a clean prompt, and training predicts future tokens from clean left cont