Search: “pretraining”

arXiv:2507.00994v4 Announce Type: replace Abstract: Learning high-quality text representations is fundamental to a wide range of NLP tasks. While encoder pretraining has traditionally relied on Masked

arXiv:2603.00110v2 Announce Type: replace Abstract: The scarcity of large-scale robotic data has motivated the repurposing of foundation models from other modalities for policy learning. In this work,

arXiv:2604.10970v1 Announce Type: new Abstract: Background: Task-specific microscopy datasets are often small, making it difficult to train deep learning models that learn robust features. While self-

arXiv:2604.08627v1 Announce Type: cross Abstract: Pretrained models have become standard in both vision and language, yet they typically do not provide reliable measures of confidence. Existing uncert

arXiv:2609.35470v1 Announce Type: cross Abstract: Applied researchers increasingly convert images into features with pretrained encoders, then use those features in a downstream prediction model. The

arXiv:2609.29592v1 Announce Type: cross Abstract: Adapting large pretrained vision models under limited data and frozen-backbone constraints remains a central challenge in transfer learning. While lig

arXiv:2603.02293v3 Announce Type: replace Abstract: Frozen pretrained features are often treated as a safe interface for downstream learning: only a small linear readout is trained, while the backbone

arXiv:2609.17868v1 Announce Type: new Abstract: Pathology vision-language models are commonly built by pretraining or fine-tuning large encoders on paired image-caption data. We asked whether a pathol

arXiv:2604.05809v2 Announce Type: replace-cross Abstract: This paper presents Text-Guided Backdoor (TGB), an adjustable backdoor attack against multimodal pretrained models that uses natural-word trig

arXiv:2607.20265v1 Announce Type: cross Abstract: Large-scale pretrained language models such as T5 and BERT have demonstrated strong capabilities for generating structured knowledge. However, their p

arXiv:2606.28998v1 Announce Type: cross Abstract: Large Language Model (LLM) alignment trains an LLM using preference data to produce outputs that better meet established quality standards. While LLM

arXiv:2605.24059v1 Announce Type: cross Abstract: We present a three-step recipe for identifying attention-head circuits in pretrained transformers. A per-head spectral signal -- the time-integrated p

arXiv:2605.10468v1 Announce Type: new Abstract: Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are

arXiv:2605.00333v1 Announce Type: cross Abstract: Frozen Gemma 4 31B weights pretrained exclusively on text tokens, unmodified, transfer across modality boundaries through a thin trainable interface.

https://preview.redd.it/m5e9anluo8vh1.png?width=1080&format=png&auto=webp&s=4c80687e6fe40036843a97ed8975ce46eaab0662 I am excited to share a project I have been working on during the past few years du

arXiv:2609.36413v1 Announce Type: new Abstract: A pretrained video world model admits many plausible futures for a scene, but a robot must realize the exact task-conditioned one. To turn world models

arXiv:2609.24377v1 Announce Type: cross Abstract: We study latent-space watermarking through a pretrained generator using a prescribed latent-to-output stochastic mapping, called the renderer. A water

arXiv:2609.08250v1 Announce Type: cross Abstract: How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as th

arXiv:2511.23120v2 Announce Type: replace Abstract: Pretrained transformers provide rich, general-purpose embeddings, which are transferred to downstream tasks. However, current transfer strategies: f

arXiv:2608.27178v1 Announce Type: new Abstract: Accurate rectal cancer segmentation from magnetic resonance imaging (MRI) is essential for adaptive radiotherapy and tumor response assessment, but depl

arXiv:2608.19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural languag

verbalizing one of those aha moments i had that seems retroactively pretty obvious: if you prioritize pretrain data quality enough that commoncrawl isn't good enough for you, you have to build a Whole

arXiv:2606.21100v1 Announce Type: new Abstract: The integration of pretrained encoders with diffusion policies has become a dominant paradigm for visual robotic manipulation. However, it still struggl

arXiv:2603.05963v2 Announce Type: replace Abstract: Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including

arXiv:2606.22874v1 Announce Type: new Abstract: Long contexts have become standard in pretrained LLMs, yet they remain expensive to run: prefill compute grows quadratically with sequence length, and e

arXiv:2606.11542v1 Announce Type: cross Abstract: Modern pretrained self-supervised automatic speech recognition models are trained on large-scale audio data to encode speech into contextualized repre

arXiv:2606.06738v1 Announce Type: new Abstract: Building monolingual language models (LMs) for low-resource languages typically relies on adapting pretrained language models (PLMs) by finetuning the w

arXiv:2509.10452v2 Announce Type: replace Abstract: Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance. In man

arXiv:2605.03964v1 Announce Type: new Abstract: Training machine learning interatomic potentials (MLIPs) for reactive chemistry is often bottlenecked by the high cost of quantum chemical labels and th

Introducing nanowhale 🐳! A tiny DeepSeek model fully pretrained by an agent. Inspired by @karpathy's nanochat, we gave ml-intern the task of training a tiny MoE with all the architectural advancements

arXiv:2610.14863v1 Announce Type: new Abstract: Data-driven weather forecasting models are commonly trained on ERA5, yet operational deployment must use timely initial states from physics-based numeri

arXiv:2609.37362v1 Announce Type: new Abstract: Large language model (LLM) routing aims to assign each query to the most suitable model from a heterogeneous candidate pool, improving the quality--effi

arXiv:2609.30566v1 Announce Type: new Abstract: We present a new inference-time sampler for diffusion models that gives a pretrained model a capability it was never trained for: constructing the atlas

arXiv:2609.30469v1 Announce Type: new Abstract: Pretrained ASR systems perform poorly on noisy Broadcast Police Communication (BPC), hindering efforts to understand police decision-making. Pseudo-labe

arXiv:2601.08358v2 Announce Type: replace Abstract: Even though the ocean covers the majority of the planet's surface, it remains the least explored ecosystem. As light and radio waves do not propagat

arXiv:2609.21139v1 Announce Type: new Abstract: Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers

arXiv:2609.07440v1 Announce Type: new Abstract: This paper proposes an open-set ego-noise separation framework for legged-robot audition via annotation-free adaptation and pretrained-model transfer. T

arXiv:2609.03655v1 Announce Type: new Abstract: Vision Foundation Models (VFMs) provide transferable patch representations for few-shot industrial anomaly detection, but their attention computation is

arXiv:2608.11233v1 Announce Type: cross Abstract: A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only a

Pretrained ViTs see the world in rich, dense detail. Most policies pool it to a single vector before acting, discarding most of it. We introduce Patch Policy: a minimal architectural extension that en

arXiv:2505.03303v3 Announce Type: replace-cross Abstract: Lightweight convolutional neural networks are often compared using results obtained with different training recipes, input settings, and pretr

arXiv:2606.30104v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models aim to learn generalizable representations from large-scale brain recordings. However, the role of tempor

arXiv:2606.23607v1 Announce Type: new Abstract: Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods

arXiv:2506.10858v2 Announce Type: replace-cross Abstract: Medical image segmentation is a fundamental task in computer-aided diagnosis and treatment. Existing approaches based on CNNs, ViTs, Mamba, an

arXiv:2606.01947v1 Announce Type: cross Abstract: Research and applications in artificial intelligence have recently shifted with the rise of large pretrained models, which deliver state-of-the-art re

This was right five years ago, and still is: “Large scale pretrained models are certainly likely to figure prominently in artificial intelligence for the near future, and play an important role in com