Search: “pretraining”

arXiv:2605.24059v1 Announce Type: cross Abstract: We present a three-step recipe for identifying attention-head circuits in pretrained transformers. A per-head spectral signal -- the time-integrated p

arXiv:2605.10468v1 Announce Type: new Abstract: Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are

arXiv:2605.00333v1 Announce Type: cross Abstract: Frozen Gemma 4 31B weights pretrained exclusively on text tokens, unmodified, transfer across modality boundaries through a thin trainable interface.

arXiv:2609.36413v1 Announce Type: new Abstract: A pretrained video world model admits many plausible futures for a scene, but a robot must realize the exact task-conditioned one. To turn world models

arXiv:2609.24377v1 Announce Type: cross Abstract: We study latent-space watermarking through a pretrained generator using a prescribed latent-to-output stochastic mapping, called the renderer. A water

arXiv:2609.08250v1 Announce Type: cross Abstract: How far a pushed object slides depends on its mass and friction, which no single image reveals. Pretrained visual encoders are increasingly used as th

arXiv:2511.23120v2 Announce Type: replace Abstract: Pretrained transformers provide rich, general-purpose embeddings, which are transferred to downstream tasks. However, current transfer strategies: f

arXiv:2608.27178v1 Announce Type: new Abstract: Accurate rectal cancer segmentation from magnetic resonance imaging (MRI) is essential for adaptive radiotherapy and tumor response assessment, but depl

arXiv:2608.19529v1 Announce Type: cross Abstract: Many real-world AI systems represent entities, behaviors, and structured information using discrete machine-native symbols rather than natural languag

verbalizing one of those aha moments i had that seems retroactively pretty obvious: if you prioritize pretrain data quality enough that commoncrawl isn't good enough for you, you have to build a Whole

arXiv:2606.21100v1 Announce Type: new Abstract: The integration of pretrained encoders with diffusion policies has become a dominant paradigm for visual robotic manipulation. However, it still struggl

arXiv:2603.05963v2 Announce Type: replace Abstract: Recent advances in large-scale pretrained vision models have demonstrated impressive capabilities across a wide range of downstream tasks, including

arXiv:2606.22874v1 Announce Type: new Abstract: Long contexts have become standard in pretrained LLMs, yet they remain expensive to run: prefill compute grows quadratically with sequence length, and e

arXiv:2606.11542v1 Announce Type: cross Abstract: Modern pretrained self-supervised automatic speech recognition models are trained on large-scale audio data to encode speech into contextualized repre

arXiv:2606.06738v1 Announce Type: new Abstract: Building monolingual language models (LMs) for low-resource languages typically relies on adapting pretrained language models (PLMs) by finetuning the w

arXiv:2509.10452v2 Announce Type: replace Abstract: Pretrained automatic speech recognition (ASR) models such as Whisper perform well but still need domain adaptation to handle unseen parlance. In man

arXiv:2605.03964v1 Announce Type: new Abstract: Training machine learning interatomic potentials (MLIPs) for reactive chemistry is often bottlenecked by the high cost of quantum chemical labels and th

Introducing nanowhale 🐳! A tiny DeepSeek model fully pretrained by an agent. Inspired by @karpathy's nanochat, we gave ml-intern the task of training a tiny MoE with all the architectural advancements

arXiv:2609.37362v1 Announce Type: new Abstract: Large language model (LLM) routing aims to assign each query to the most suitable model from a heterogeneous candidate pool, improving the quality--effi

arXiv:2609.30566v1 Announce Type: new Abstract: We present a new inference-time sampler for diffusion models that gives a pretrained model a capability it was never trained for: constructing the atlas

arXiv:2609.30469v1 Announce Type: new Abstract: Pretrained ASR systems perform poorly on noisy Broadcast Police Communication (BPC), hindering efforts to understand police decision-making. Pseudo-labe

arXiv:2601.08358v2 Announce Type: replace Abstract: Even though the ocean covers the majority of the planet's surface, it remains the least explored ecosystem. As light and radio waves do not propagat

arXiv:2609.21139v1 Announce Type: new Abstract: Replacing attention in a pretrained language model is a compatibility problem: a plausible substitute may alter representations expected by later layers

arXiv:2609.07440v1 Announce Type: new Abstract: This paper proposes an open-set ego-noise separation framework for legged-robot audition via annotation-free adaptation and pretrained-model transfer. T

arXiv:2609.03655v1 Announce Type: new Abstract: Vision Foundation Models (VFMs) provide transferable patch representations for few-shot industrial anomaly detection, but their attention computation is

arXiv:2608.11233v1 Announce Type: cross Abstract: A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only a

Pretrained ViTs see the world in rich, dense detail. Most policies pool it to a single vector before acting, discarding most of it. We introduce Patch Policy: a minimal architectural extension that en

arXiv:2505.03303v3 Announce Type: replace-cross Abstract: Lightweight convolutional neural networks are often compared using results obtained with different training recipes, input settings, and pretr

arXiv:2606.30104v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models aim to learn generalizable representations from large-scale brain recordings. However, the role of tempor

arXiv:2606.23607v1 Announce Type: new Abstract: Linear mode connectivity (LMC) provides a promising foundation for understanding and merging independently trained neural networks, but existing methods

arXiv:2506.10858v2 Announce Type: replace-cross Abstract: Medical image segmentation is a fundamental task in computer-aided diagnosis and treatment. Existing approaches based on CNNs, ViTs, Mamba, an

arXiv:2606.01947v1 Announce Type: cross Abstract: Research and applications in artificial intelligence have recently shifted with the rise of large pretrained models, which deliver state-of-the-art re

This was right five years ago, and still is: “Large scale pretrained models are certainly likely to figure prominently in artificial intelligence for the near future, and play an important role in com

arXiv:2605.21653v1 Announce Type: cross Abstract: AI text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary. On raw encoders before any task supervision, pr

arXiv:2605.11011v1 Announce Type: new Abstract: Looped computation shows promise in improving the reasoning-oriented performance of LLMs by scaling test-time compute. However, existing approaches typi

To train better open models, we need predictable scaling. Delphi is Marin’s first step: we pretrained many small models with one recipe, then extrapolated 300× to predict a 25B-param / 600B-token run

arXiv:2601.10931v2 Announce Type: replace Abstract: Tree canopy detection from aerial imagery is an important task for environmental monitoring, urban planning, and ecosystem analysis. Simulating real

arXiv:2604.24474v1 Announce Type: new Abstract: Molecular similarity plays a central role in ligand-based drug discovery, such as virtual screening, analog searching, and goal-directed molecular gener

arXiv:2604.22334v1 Announce Type: new Abstract: Recent advances in pretraining 3D point cloud encoders (e.g., Point-BERT, Point-MAE) have produced powerful models, whose abilities are typically evalua

arXiv:2510.05482v2 Announce Type: replace Abstract: Molecular dynamics (MD) simulations underpin modern computational drug discovery, materials science, and biochemistry. Recent machine learning model

Sapiens2 is the highest quality ViT backbone that now exists in the public domain. It was pretrained on the equivalent of 1/2 of all human images on Flickr. First public release by a large lab that is

🦤 LeWorldModel: Learning Physics from Pixels — Stable World Models with Just Two Losses World models: 1️⃣ DINO-WM: pretrained ViT encoder (from ImageNet) → features → predictor. But encoder is frozen,

arXiv:2602.12370v2 Announce Type: replace Abstract: Recent progress in large models has led to significant advances in unified multimodal generation and understanding. However, the development of mode

arXiv:2604.10268v1 Announce Type: new Abstract: We propose EditCrafter, a high-resolution image editing method that operates without tuning, leveraging pretrained text-to-image (T2I) diffusion models

arXiv:2604.08598v1 Announce Type: cross Abstract: Text-based person search faces inherent limitations due to data scarcity, driven by stringent privacy constraints and the high cost of manual annotati

arXiv:2506.17585v4 Announce Type: replace-cross Abstract: Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving d