Search: “pretraining”

arXiv:2511.16893v3 Announce Type: replace Abstract: Specialized attention heads dubbed induction heads (IHs) have been argued to underlie the remarkable in-context learning capabilities of modern lang

arXiv:2601.06891v2 Announce Type: replace Abstract: Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and

arXiv:2606.25546v1 Announce Type: new Abstract: Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet e

arXiv:2606.21292v1 Announce Type: new Abstract: We present Casper3D, a lightweight probabilistic framework for converting noisy multi-view 2D foundation-model embeddings into a latent 3D semantic repr

arXiv:2606.21271v1 Announce Type: new Abstract: Sparse rewards pose a central challenge in reinforcement learning, since agents receive no informative signal until they reach their goal. Intrinsic-rew

arXiv:2606.11081v1 Announce Type: cross Abstract: Communication-efficient pre-training of LLMs is increasingly important as training draws on compute distributed across clusters, data centers, and low

arXiv:2606.04898v1 Announce Type: new Abstract: Anatomical landmark detection is a fundamental task in medical image analysis supporting a wide range of diagnostic and interventional workflows. Althou

arXiv:2606.02991v1 Announce Type: cross Abstract: We introduce TypewriterLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913. Developing History LMs requires add

arXiv:2606.00673v1 Announce Type: new Abstract: Thermal imaging offers a powerful alternative to visible-spectrum vision under challenging conditions such as low illumination and adverse weather, yet

arXiv:2605.28132v1 Announce Type: new Abstract: Spatial intelligence requires visual representations that capture both semantic objects and geometric structure in the physical world. To support this,

arXiv:2410.19842v2 Announce Type: replace-cross Abstract: Contrastive learning yields impressive results for self-supervision in computer vision. The approach relies on the creation of positive pairs,

arXiv:2605.23840v1 Announce Type: new Abstract: Mueller matrix imaging provides rich, physically meaningful contrast for biomedical tissue analysis, but supervised learning is hindered by scarce dense

arXiv:2510.25897v2 Announce Type: replace Abstract: The default paradigm of post-training text-to-image generators includes post-hoc selection of generated images, and subsequent training with one rew

arXiv:2605.18552v1 Announce Type: new Abstract: Classifying protein topology is essential for deciphering biological function, but progress is held back by the lack of large-scale benchmarks that avoi

arXiv:2511.08704v2 Announce Type: replace Abstract: This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unifie

arXiv:2602.19710v2 Announce Type: replace Abstract: Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency because they entangle high-level percept

arXiv:2605.14747v1 Announce Type: cross Abstract: Recent advances in multimodal large language models have driven growing interest in graphical user interface (GUI) agents, yet their generalization re

arXiv:2605.12438v1 Announce Type: new Abstract: When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily sw

arXiv:2605.11034v1 Announce Type: cross Abstract: We present MambaNetBurst, a compact tokenizer-free byte-level sequence classifier for network burst classification based on a Mamba-2 backbone. In con

arXiv:2605.11408v1 Announce Type: cross Abstract: Tabular data forms the backbone of high-stakes decision systems in finance, healthcare, and beyond. Yet industrial tabular datasets are inherently dif

arXiv:2510.25356v2 Announce Type: replace Abstract: In the U.S. judicial system, a widespread approach to legal interpretation entails assessing how a legal text would be understood by an `ordinary' s

arXiv:2605.12236v1 Announce Type: cross Abstract: Fine-tuning pre-trained robot policies with reinforcement learning (RL) often inherits the bottlenecks introduced by pre-training with behavioral clon

arXiv:2605.02968v1 Announce Type: new Abstract: We introduce a finite-size gradient-transport framework for real language-model training, based on five observables (D,z,eta,elta,v_{rel}) that separate

arXiv:2605.03968v1 Announce Type: new Abstract: Accurate school detection is essential for supporting education initiatives, including infrastructure planning and expanding internet connectivity to un

arXiv:2604.27039v1 Announce Type: new Abstract: Token serves as the fundamental unit of computation in modern autoregressive models, and generation length directly influences both inference cost and r

arXiv:2507.21545v3 Announce Type: replace Abstract: Robotic task planning in real-world environments requires reasoning over implicit constraints from language and vision. While LLMs and VLMs offer st

arXiv:2511.09282v3 Announce Type: replace-cross Abstract: Significant progress has been made in spoken question answering (SQA) in recent years. However, many existing methods, including large audio l

We've gone even farther: Nemotron 3 Super is 120B and pretrained on 25T tokens in NVFP4. Nemotron 3 Ultra is ~500B and also pretrained in NVFP4. Accelerated computing means we rethink every aspect of

arXiv:2512.14751v3 Announce Type: replace-cross Abstract: Finetuning pretrained large language models (LLMs) has become the standard paradigm for developing downstream applications. However, its secur

arXiv:2609.34756v1 Announce Type: cross Abstract: Adapting pretrained models to downstream tasks with limited data has become a central paradigm in modern deep learning. Yet, despite its widespread pr

arXiv:2504.10214v2 Announce Type: replace Abstract: Pretrained model-based incremental object detection (PTMIOD) leverages the rich detection priors of pretrained detectors to learn new categories inc

arXiv:2610.12599v1 Announce Type: new Abstract: Sparse computation can scale the parametric capacity of LLMs without proportionally increasing active computation, but pretraining sparse architectures

arXiv:2608.04389v1 Announce Type: new Abstract: Decoding continuous motor trajectories from neural activity is essential for developing practical brain-computer interfaces (BCIs). However, current neu

arXiv:2607.27203v1 Announce Type: new Abstract: Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) thi

arXiv:2509.10468v2 Announce Type: replace-cross Abstract: Recent advances in generative recommenders adopt a two-stage paradigm: items are first tokenized into semantic IDs using a pretrained tokenize

arXiv:2610.14558v1 Announce Type: new Abstract: Large pretrained models have achieved remarkable success across diverse domains, yet adapting them to multiple downstream tasks remains challenging: ful

arXiv:2609.16262v1 Announce Type: cross Abstract: Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objec

arXiv:2609.15344v1 Announce Type: new Abstract: We study the adaptation of pretrained language models to univariate time-series forecasting through a parameter-efficient transfer learning framework, w

arXiv:2609.10518v1 Announce Type: new Abstract: fMRI foundation models increasingly aggregate heterogeneous data across brain states, cohorts, and acquisition settings, yet pretraining domains are com

arXiv:2610.22075v1 Announce Type: new Abstract: Recent work shows that depth recurrence can be retrofitted into pretrained language models. In this work, we adapt this paradigm for visual understandin

arXiv:2610.14011v1 Announce Type: new Abstract: Modern experiments increasingly produce rich post-treatment objects---such as text, images, and video---that may provide early surrogate signals for out

arXiv:2609.32898v1 Announce Type: new Abstract: Pretrained tabular foundation models process every dataset at a fixed depth, with inference costs growing with dataset size. To address this, we present

arXiv:2609.30948v1 Announce Type: cross Abstract: The Job Shop Scheduling Problem (JSSP) is a fundamental combinatorial optimization problem in industrial optimization. This work introduces Pretrained

arXiv:2607.25157v1 Announce Type: new Abstract: Discrete masked diffusion language models support bidirectional generation and infilling, but adapting pretrained autoregressive (AR) transformers requi

arXiv:2606.30988v1 Announce Type: new Abstract: Robot manipulation often relies on sensory feedback beyond vision, particularly in contact-rich settings where force, tactile, or audio signals reveal i

arXiv:2605.30467v1 Announce Type: new Abstract: This study introduces a novel Arctic-focused remote sensing foundation model (RSFM) by combining diversity-aware regional-scale image curation with mask