Search: “pretraining”

Accelerating Vision Foundation Models with Drop-in Depthwise Convolution

arXiv:2605.22132v1 Announce Type: new Abstract: Pretrained vision foundation models deliver strong performance across tasks with limited fine-tuning. However, their Vision Transformer (ViT) backbones

22 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Value-Gradient Hypothesis of RL for LLMs

DGX agent

arXiv:2605.21654v1 Announce Type: cross Abstract: Reinforcement learning substantially improves pretrained language models, but it remains understudied why critic-free methods such as PPO and GRPO wor

safetyarxiv-cs-cl

22 May 2026

An exponential mechanism based on quadratic approximations for fine-tuning machine learning models with privacy guarantees

arXiv:2605.20521v1 Announce Type: new Abstract: Fine-tuning adapts a pretrained machine learning model to a small, sensitive dataset, but this process risks memorizing individual new data points, maki

21 May 2026

FullFlow: Upgrading Text-to-Image Flow Matching Models for Bidirectional Vision--Language Generation

model-releasesarxiv-cs-cv

arXiv:2605.20316v1 Announce Type: new Abstract: Modern text-to-image diffusion models encode rich visual priors, but expose them only through one-way text-conditioned generation. Existing unified visi

21 May 2026

Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulation

arXiv:2605.21258v1 Announce Type: new Abstract: Current 3D-aware pretraining methods for embodied perception and manipulation are largely built on differentiable rendering frameworks, producing either

researcharxiv-cs-ro

21 May 2026

Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

arXiv:2605.18911v1 Announce Type: cross Abstract: Wildfire prediction is important for early warning and resource allocation, yet existing Earth foundation models (Earth FMs) are pretrained for genera

20 May 2026

Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

arXiv:2605.20005v1 Announce Type: new Abstract: Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining, a phenomenon known as cata

20 May 2026

When Tabular Foundation Models Meet Strategic Tabular Data: A Prior Alignment Approach

arXiv:2605.19662v1 Announce Type: new Abstract: Tabular foundation models based on pretrained prior-data fitted networks~(PFNs) have shown strong generalization on diverse tabular tasks, but they are

safetyarxiv-cs-ai

20 May 2026

Adaptive Control in Autonomous Driving via Real-Time Recurrent RL

arXiv:2602.02236v4 Announce Type: replace-cross Abstract: We study online fine-tuning of pretrained control policies for autonomous driving using Real-Time Recurrent Reinforcement Learning (RTRRL), a

safetyarxiv-cs-lg

Local Ai

CanViT: Toward Active-Vision Foundation Models

arXiv:2603.22570v2 Announce Type: replace Abstract: Active computer vision promises efficient, biologically plausible perception through sequential, localized glimpses, but lacks scalable general-purp

local-aiarxiv-cs-cv

Forecasting Downstream Performance of LLMs With Proxy Metrics

arXiv:2605.18607v1 Announce Type: new Abstract: Progress in language model development is often driven by comparative decisions: which architecture to adopt, which pretraining corpus to use, or which

Improved Baselines with Representation Autoencoders

arXiv:2605.18324v1 Announce Type: cross Abstract: Representation Autoencoders (RAE) replace traditional VAE with pretrained vision encoders. In this paper, we systematically investigate several design

safetyarxiv-cs-ai

Language Acquisition Device in Large Language Models

arXiv:2605.16758v1 Announce Type: new Abstract: Large Language Models (LLMs) remain substantially less data-efficient than humans. Pre-pretraining (PPT) on synthetic languages has been proposed to clo

Mitigating 3D Prostate Biparametric MRI Data Scarcity through Domain Adaptation using Locally-Trained Latent Diffusion Models for Prostate Cancer Detection

arXiv:2507.06384v2 Announce Type: replace-cross Abstract: Objective: Latent diffusion models (LDMs) could mitigate data scarcity challenges affecting machine learning development for medical image int

Olivia: Harmonizing Time Series Foundation Models with Power Spectral Density

arXiv:2605.17340v1 Announce Type: new Abstract: Time series foundation models rely on large-scale pretraining over diverse datasets across domains, yet their heterogeneity in temporal patterns could h

Ringmaster LMO: Asynchronous Linear Minimization Oracle Momentum Method

arXiv:2605.18174v1 Announce Type: new Abstract: Muon has recently emerged as a strong alternative to AdamW for training neural networks, with encouraging large-scale pretraining results and growing ev

SkyNative: A Native Multimodal Framework for Remote Sensing Visual Evidence Reasoning

model-releasesarxiv-cs-cv

arXiv:2605.17949v1 Announce Type: new Abstract: Remote sensing vision-language models commonly rely on pretrained visual encoders to convert images into semantic features before language-model reasoni

When Dynamics Shift, Robust Task Inference Wins: Offline Imitation Learning with Behavior Foundation Models Revisited

arXiv:2605.17017v1 Announce Type: cross Abstract: Behavior Foundation Models (BFMs) enable scalable imitation learning (IL) by pretraining task-agnostic representations that can be rapidly adapted to

safetyarxiv-cs-ai

Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

arXiv:2602.23410v3 Announce Type: replace-cross Abstract: Brain foundation models have achieved remarkable advances across a wide range of neuroscience tasks. However, most existing models are limited

18 May 2026

UAM: A Dual-Stream Perspective on Forgetting in VLA Training

model-releasesarxiv-cs-ai

arXiv:2605.15735v1 Announce Type: cross Abstract: Vision--language--action (VLA) models are typically built by fine-tuning a pretrained vision--language model (VLM) on action data. However, we show th

18 May 2026

Chem-GMNet: A Sphere-Native Geometric Transformer for Molecular Property Prediction

arXiv:2605.13262v1 Announce Type: new Abstract: Modern SMILES-based chemical language models obtain strong MoleculeNet performance by treating SMILES as generic text and compensating with multi-millio

14 May 2026

20/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone

arXiv:2605.11405v1 Announce Type: new Abstract: Data curation has shifted the quality-compute frontier for language-model and contrastive image-text pretraining, but its role for vision-language model

Efficient Adjoint Matching for Fine-tuning Diffusion Models

arXiv:2605.11480v1 Announce Type: new Abstract: Reward fine-tuning has become a common approach for aligning pretrained diffusion and flow models with human preferences in text-to-image generation. Am

h-control: Training-Free Camera Control via Block-Conditional Gibbs Refinement

arXiv:2605.11871v1 Announce Type: new Abstract: Training-free camera control for pretrained flow-matching video generators is a partial-observation inverse problem: a depth-warped guidance video suppl

Rotation-Preserving Supervised Fine-Tuning

arXiv:2605.10973v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that this degradatio

Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenizatio

arXiv:2605.10780v1 Announce Type: cross Abstract: Representation autoencoders that reuse frozen pretrained vision encoders as visual tokenizers have achieved strong reconstruction and generation quali

CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models

model-releasesarxiv-cs-cv

arXiv:2605.10903v1 Announce Type: new Abstract: This paper proposes a novel approach to address the challenge that pretrained VLA models often fail to effectively improve performance and reduce adapta

Composing diffusion priors with explicit physical context via generative Gibbs sampling

arXiv:2605.10642v1 Announce Type: new Abstract: Pretrained diffusion models provide powerful learned priors, but in scientific sampling the target distribution often depends on physical context that i

Learning Graph Foundation Models on Riemannian Graph-of-Graphs

arXiv:2605.09993v1 Announce Type: new Abstract: Graph foundation models (GFMs), pretrained on massive graph data, have transformed graph machine learning by supporting general-purpose reasoning across

MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image

model-releasesarxiv-cs-cl

arXiv:2605.10616v1 Announce Type: cross Abstract: Tabular Foundation Models have recently established the state of the art in supervised tabular learning, by leveraging pretraining to learn generaliza

SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations

arXiv:2605.09485v1 Announce Type: new Abstract: Latent representations learned by neural networks often exhibit semantic structure, where concept similarity is reflected by geometric proximity in embe

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

arXiv:2605.08738v1 Announce Type: cross Abstract: Structured pruning and knowledge distillation (KD) are typical techniques for compressing large language models, but it remains unclear how they shoul

Supervised Dimensionality Reduction Revisited: Why LDA on Frozen CNN Features Deserves a Second Look

arXiv:2604.03928v2 Announce Type: replace-cross Abstract: Frozen pretrained image representations are widely used for transfer learning: a backbone is kept fixed, feature vectors are extracted, and a

Tabular Foundation Model for Generative Modelling

arXiv:2605.09424v1 Announce Type: new Abstract: Generative modelling is a demanding test of foundation models, because it requires robust, holistic representation learning for a given data modality, r

Applications

AT-VLA: Adaptive Tactile Injection for Enhanced Feedback Reaction in Vision-Language-Action Models

arXiv:2605.07308v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have significantly advanced the capabilities of robotic agents in executing diverse tasks; however, they still face

applicationsarxiv-cs-ro

Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning

arXiv:2605.07333v1 Announce Type: new Abstract: In-context reinforcement learning (ICRL) studies agents that, after pretraining, adapt to new tasks by conditioning on additional context without parame

ProteinJEPA: Latent prediction complements protein language models

arXiv:2605.07554v1 Announce Type: cross Abstract: Protein language models are trained primarily with masked language modeling (MLM), which predicts amino-acid identities at masked positions. We ask wh

Toward Better Geometric Representations for Molecule Generative Models

arXiv:2605.07693v1 Announce Type: new Abstract: Geometric representation-conditioned molecule generation provides an effective paradigm that decouples molecule representation modeling from structure g

safetyarxiv-cs-lg

UFT: Unifying Fine-Tuning of SFT and RLHF/DPO/UNA through a Generalized Implicit Reward Function

arXiv:2410.21438v3 Announce Type: replace Abstract: By pretraining on trillions of tokens, an LLM gains the capability of text generation. However, to enhance its utility and reduce potential harm, SF

safetyarxiv-cs-cl

FAAST: Forward-Only Associative Learning via Closed-Form Fast Weights for Test-Time Supervised Adaptation

arXiv:2605.04651v1 Announce Type: cross Abstract: Adapting pretrained models typically involves a trade-off between the high training costs of backpropagation and the heavy inference overhead of memor

7 May 2026

Compress Then Adapt? No, Do It Together via Task-aware Union of Subspaces

model-releasesarxiv-cs-ai

arXiv:2605.02829v1 Announce Type: new Abstract: Adapting large pretrained models to diverse tasks is now routine, yet the two dominant strategies of parameter-efficient fine-tuning (PEFT) and low-rank

6 May 2026

Contrastive Regularization for Accent-Robust ASR

arXiv:2605.03297v1 Announce Type: cross Abstract: ASR systems based on self-supervised acoustic pretraining and CTC fine-tuning achieve strong performance on native speech but remain sensitive to acce

6 May 2026

Tutorials

GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models

arXiv:2605.01256v1 Announce Type: new Abstract: A promising paradigm for adapting instruction-tuned language models is to learn task-specific updates on a pretrained base model and subsequently merge

tutorialsarxiv-cs-cl

InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

arXiv:2605.02364v1 Announce Type: new Abstract: Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweigh

MVP-LAM: Learning Action-Centric Latent Action via Cross-Viewpoint Reconstruction

arXiv:2602.03668v2 Announce Type: replace-cross Abstract: Latent actions learned from diverse human videos serve as pseudo-labels for vision-language-action (VLA) pretraining, but provide effective su

SignMAE: Segmentation-Driven Self-Supervised Learning for Sign Language Recognition

arXiv:2605.02094v1 Announce Type: new Abstract: Subtle hand differences make sign language recognition challenging, yet many existing methods rely on encoders pretrained on generic action datasets tha

SpectraDINO: Bridging the Spectral Gap in Vision Foundation Models via Lightweight Adapters

arXiv:2605.02258v1 Announce Type: new Abstract: Vision Foundation Models (VFMs) pretrained on large-scale RGB data have demonstrated remarkable representation quality, yet their applicability to multi

safetyarxiv-cs-cv

Local Ai

From Birdsong to Rumbles: Classifying Elephant Calls with Out-of-Species Embeddings