Search: “pretraining”

arXiv:2606.16246v2 Announce Type: replace Abstract: As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shift

arXiv:2603.13432v4 Announce Type: replace-cross Abstract: Spatial Transcriptomics (ST) profiles thousands of gene expression values at discrete spots with precise coordinates on tissue sections, prese

arXiv:2605.20105v1 Announce Type: new Abstract: Learning to generalise from limited data is a fundamental challenge for both artificial and biological systems. A common strategy is to extract reusable

arXiv:2604.12683v1 Announce Type: new Abstract: Current fMRI foundation models primarily rely on a limited range of brain states and mismatched pretraining tasks, restricting their ability to learn ge

arXiv:2604.09849v1 Announce Type: cross Abstract: We study whether in-domain pretraining of Bidirectional Encoder Representations from Transformer (BERT) model improves subdomain-level detection of ex

arXiv:2604.09258v1 Announce Type: new Abstract: Pretraining is the cornerstone of Large Language Models (LLMs), dominating the vast majority of computational budget and data to serve as the primary en

arXiv:2609.38764v1 Announce Type: new Abstract: Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract, a

arXiv:2609.38446v1 Announce Type: cross Abstract: A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how pretraining and midtraining supp

arXiv:2609.37642v1 Announce Type: new Abstract: Self-supervised pretraining reshaped prediction in language and vision, and brain foundation models (BFMs) inherited its promise. Representations learne

arXiv:2609.35090v1 Announce Type: new Abstract: Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains unde

arXiv:2609.31985v1 Announce Type: new Abstract: Vision-language pretraining objectives differ in the spatial granularity of their supervision, yet the implications of this distribution for chest X-ray

arXiv:2609.35457v1 Announce Type: new Abstract: Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instea

arXiv:2609.22868v1 Announce Type: new Abstract: End-to-end driving requires planning-relevant bird's-eye-view (BEV) representations, but existing pretraining approaches often rely on task annotations

arXiv:2609.23916v1 Announce Type: new Abstract: Large language models (LLMs) drift out of date the moment their pretraining ends, yet retraining from scratch is prohibitively expensive. Continued pret

arXiv:2609.20814v1 Announce Type: cross Abstract: Pretraining a neural PDE surrogate can reduce the amount of new CFD data needed when geometry or modeled physics changes. However, it remains unclear

arXiv:2609.14571v1 Announce Type: new Abstract: This study highlights the role of domain-specific pretraining profile (DSPP) in Transformer performance for modeling digital pragmatics in Arabic-Englis

arXiv:2609.07798v1 Announce Type: cross Abstract: Natural Language Processing (NLP) in the climate domain requires models to process heterogeneous text sources, including scientific literature, policy

arXiv:2609.06460v1 Announce Type: new Abstract: Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various d

arXiv:2510.10681v2 Announce Type: replace-cross Abstract: High-quality data is a cornerstone of large language model (LLM) pretraining, yet its growth has not kept pace with the needs of frontier mode

Dwarkesh Podcast: Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements — Breaking down 6 years of pretraining progress

arXiv:2601.14603v2 Announce Type: replace Abstract: Optimizer design plays a central role in efficient language model pretraining, directly affecting optimization dynamics, convergence speed, and comp

arXiv:2608.27395v1 Announce Type: cross Abstract: Video carries the temporal structure of the physical world, yet learning representations from it has remained computationally expensive: prevailing se

arXiv:2608.25701v1 Announce Type: new Abstract: We propose a novel pretraining strategy for skeleton-based zero-shot spatio-temporal action localization to estimate unseen actions for person instances

Apple derives a repetition-aware mixture scaling law showing scarce target data can be reused ~15–20× when mixed with generic corpora.

arXiv:2606.12499v2 Announce Type: replace Abstract: We present AEM, an Action-Effect Memory pretraining framework for robot manipulation that learns compact temporal representations from vision-action

arXiv:2608.15195v1 Announce Type: new Abstract: Vision Foundation Models (VFMs) have emerged as a promising approach in medical imaging, producing broadly applicable systems that can be efficiently ad

arXiv:2106.16038v3 Announce Type: replace Abstract: Recent pretraining models in Chinese neglect two important aspects specific to the Chinese language: glyph and pinyin, which carry significant synta

arXiv:2607.25271v1 Announce Type: cross Abstract: Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in whic

arXiv:2607.23343v1 Announce Type: cross Abstract: Intraoperative 2D/3D registration aligns preoperative CT volumes with intraoperative X-ray or fluoroscopic images and is essential for image-guided in

arXiv:2607.04846v1 Announce Type: cross Abstract: Transformers follow implicit curricula whereby some tasks are learned before others. However, how explicit pretraining curricula influence learning, g

arXiv:2602.20062v2 Announce Type: replace Abstract: Pretraining and fine-tuning are central stages in modern machine learning systems. In practice, feature learning plays an important role across both

arXiv:2509.10406v4 Announce Type: replace Abstract: Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs

arXiv:2604.02546v2 Announce Type: replace Abstract: Pretraining 3D encoders by aligning with Contrastive Language Image Pretraining (CLIP) has emerged as a promising direction to learn generalizable r

arXiv:2606.06888v1 Announce Type: new Abstract: Classical scaling laws for language model pretraining balance model size against training dataset size under a fixed compute budget, assuming abundant d

arXiv:2601.19936v2 Announce Type: replace-cross Abstract: The opacity of massive pretraining corpora in Large Language Models (LLMs) raises significant privacy and copyright concerns, making pretraini

arXiv:2605.23472v1 Announce Type: new Abstract: Vision foundation models pretrained on web-scale data have recently shown strong transfer capabilities on many downstream tasks, but their effectiveness

arXiv:2605.20613v1 Announce Type: new Abstract: The current pretraining paradigm for large language models relies on massive compute and internet-scale raw text, creating a significant barrier to foun

arXiv:2605.20449v1 Announce Type: new Abstract: Can language-pretrained transformers become effective time-series forecasters, and why? In this paper, we show that cross-modal transfer arises because

arXiv:2605.17849v1 Announce Type: cross Abstract: LLM pretraining is shifting from a compute-bound to a data-bound regime, where available human (organic) text falls far short of scaling demands. Howe

arXiv:2605.12382v1 Announce Type: new Abstract: Large language models (LLMs) exhibit systematic preferences for well-known entities, a phenomenon often attributed to popularity bias. However, the exte

arXiv:2605.10840v1 Announce Type: cross Abstract: We present Clin-JEPA, a multi-phase co-training framework for joint-embedding predictive (JEPA) pretraining on EHR patient trajectories. JEPA architec

arXiv:2601.21698v2 Announce Type: replace-cross Abstract: Curriculum learning changes the order of pretraining data, but it remains unclear how ordering changes the learning dynamics. We pretrain mode

ERNIE 5.1 is here 🚀 ERNIE 5.1 significantly reduces pretraining cost while compressing total parameters to ~1/3 and activated parameters to ~1/2 — using only ~6% of the pretraining cost compared to mo

arXiv:2509.05291v2 Announce Type: replace-cross Abstract: Large language models (LLMs) learn non-trivial abstractions during pretraining, such as detecting irregular plural noun subjects. However, bec

arXiv:2604.27448v1 Announce Type: new Abstract: This paper revisits camera pose estimation through the lens of self-supervised pretraining, focusing on inverse-dynamics pretraining as a scalable alter

arXiv:2604.12012v1 Announce Type: new Abstract: Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classificat