Search: “pretraining”

CellWorld: From Gene-Level Reconstruction to Latent Cell Prediction in Spatial Transcriptomics Foundation Models

arXiv:2608.06659v1 Announce Type: new Abstract: This paper shows that latent-space predictive pretraining can provide a scalable route to foundation models for spatial transcriptomics. Existing spatia

10 Aug 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Tutorials

Beyond Gene Reconstruction: Learning Cell Representations through Complementary Transcriptomic Views

DGX agent

arXiv:2608.00985v1 Announce Type: new Abstract: The rapid growth of single-cell transcriptomic data has enabled the development of foundation models pretrained primarily by reconstructing masked expre

tutorialsarxiv-cs-lg

4 Aug 2026

Learning the Pareto Frontier of Predictive Models under Distribution Shift

arXiv:2608.00632v1 Announce Type: new Abstract: Modern machine learning pipelines increasingly rely on reusing pretrained and foundation models across downstream tasks. These pretrained models can dif

applicationsarxiv-cs-lg

4 Aug 2026

Rad-JEPA 3D: Radiology Joint-Embedding Predictive Model for 3D Computed Tomography

model-releasesarxiv-cs-cv

arXiv:2607.26196v1 Announce Type: new Abstract: Self-supervised pretraining is central to 3D medical image analysis, where unlabeled CT volumes are abundant but expert annotations are scarce. Yet exis

30 Jul 2026

FlowWAM: Optical Flow as a Unified Action Representation for World Action Models

arXiv:2607.13017v1 Announce Type: cross Abstract: World Action Models (WAMs) are able to leverage pretrained video generators for both world modeling and action prediction. However, directly leveragin

15 Jul 2026

Dive Into the Implicit Biases of Low-rank Vision-language Alignment

model-releasesarxiv-cs-cv

arXiv:2607.08194v1 Announce Type: new Abstract: Vision-language alignment, the stage that bridges pretrained vision encoders and large language models, is widely treated as a form of pretraining requi

10 Jul 2026

NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation

arXiv:2607.06678v1 Announce Type: new Abstract: How can pretrained Vision-Language-Action (VLA) models retain long-horizon visual histories with high-frequency updates without sacrificing efficiency?

safetyarxiv-cs-ro

9 Jul 2026

Learning When to Attend: Conditional Memory Access for Long-Context LLMs

model-releasesarxiv-cs-cl

arXiv:2603.17484v2 Announce Type: replace Abstract: Language models struggle to generalize beyond pretraining context lengths, limiting long-horizon reasoning and retrieval. Continued pretraining on l

7 Jul 2026

Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving

arXiv:2607.01733v1 Announce Type: new Abstract: Speech-LLM integration has shown promising results by leveraging extensive textual pretraining, yet its specific benefits for automatic speech recogniti

researcharxiv-cs-cl

3 Jul 2026

From Phase to Phenomenon: Self-Supervised Learning of Subsurface Scattering with Minimal Phase-shift Inputs

arXiv:2606.29461v1 Announce Type: new Abstract: We propose a self-supervised pretraining framework for learning sub-surface scattering (SSS) light transport representations from minimal input. Our met

30 Jun 2026

Learning Transferable Dynamics Priors from Action to World Modeling

arXiv:2606.29501v1 Announce Type: new Abstract: We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict

safetyarxiv-cs-ro

30 Jun 2026

PoLAR: Factorizing Extent and Mode in Latent Actions for Robot Policy Learning

arXiv:2606.21139v1 Announce Type: cross Abstract: Latent action pretraining learns representations of visual change from pairs of observations, but existing methods typically encode each transition as

safetyarxiv-cs-lg

23 Jun 2026

QuoVLA: Quotient Space for Vision-Language-Action Models

arXiv:2605.24890v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models commonly adapt pretrained Vision-Language Models (VLMs) to robot control by mapping visual observations and lang

9 Jun 2026

What Makes Video World Model Latents Action-Relevant: Prediction over Reconstruction

arXiv:2606.07687v1 Announce Type: cross Abstract: Video world models are increasingly used to provide predictive visual representations, yet it remains unclear which pretraining signals induce action-

9 Jun 2026

WALL-WM: Carving World Action Modeling at the Event Joints

arXiv:2606.01955v1 Announce Type: cross Abstract: WALL-WM is a World Action Model that shifts video-action learning from chunk-centric optimization to event-grounded Vision-Language-Action pretraining

applicationsarxiv-cs-cv

2 Jun 2026

WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Models

model-releasesarxiv-cs-cl

arXiv:2510.22276v3 Announce Type: replace-cross Abstract: Contrastive vision-language models have achieved remarkable progress through large-scale pretraining. Recent work has shown that removing Engl

2 Jun 2026

Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructions

arXiv:2501.04661v3 Announce Type: replace-cross Abstract: The web-scale of pretraining data has created an important evaluation challenge: to disentangle linguistic competence on cases well-represente

applicationsarxiv-cs-ai

1 Jun 2026

Adapting Automotive Aerodynamics Surrogates to New Vehicle Families via Transfer Learning

model-releasesarxiv-cs-lg

arXiv:2605.27968v1 Announce Type: cross Abstract: Deploying Scientific Machine Learning surrogates in industrial CFD workflows requires adapting pretrained models to new vehicle families without large

29 May 2026

FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales

model-releasesarxiv-cs-ai

arXiv:2605.28174v1 Announce Type: cross Abstract: Foundation models offer a promising route to transferable remote sensing representations, but many current approaches depend on very large pretraining

28 May 2026

TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Models

arXiv:2605.26161v1 Announce Type: cross Abstract: Time series foundation models (TSFMs) are increasingly pretrained on large corpora, raising concerns that evaluation datasets may have been exposed du

27 May 2026

PilotWiMAE: Pilot-Native Representation Learning for Wireless Channels

arXiv:2605.22856v1 Announce Type: cross Abstract: Channel foundation models assume access to fully observed channels, an assumption that fails in deployment. We introduce PilotWiMAE, a self-supervised

safetyarxiv-cs-ai

25 May 2026

PointACT: Vision-Language-Action Models with Multi-Scale Point-Action Interaction

arXiv:2605.21414v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have shown strong potential for general-purpose robotic manipulation by leveraging large pretrained vision-languag

21 May 2026

From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation

model-releasesarxiv-cs-ai

arXiv:2605.18865v1 Announce Type: cross Abstract: Self-attention serves as the core foundation of large-scale transformer pretraining, but its quadratic token interaction cost makes inference expensiv

20 May 2026

Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Study

arXiv:2605.14031v1 Announce Type: cross Abstract: Bioacoustic recognition requires fine-grained acoustic understanding to distinguish similar-sounding species. However, many large-scale data repositor

applicationsarxiv-cs-cv

15 May 2026

FLAME: Adaptive Mixture-of-Experts for Continual Multimodal Multi-Task Learning

model-releasesarxiv-cs-lg

arXiv:2605.09355v1 Announce Type: new Abstract: Real-world model deployment across multiple domains requires multimodal models to operate under two complementary regimes: (1) multi-task pretraining, t

MePo: Meta Post-Refinement for Rehearsal-Free General Continual Learning

arXiv:2602.07940v3 Announce Type: replace Abstract: To cope with uncertain changes of the external world, intelligent systems must continually learn from complex, evolving environments and respond in

safetyarxiv-cs-ai

Optimal Attention Temperature Improves the Robustness of In-Context Learning under Distribution Shift in High Dimensions

arXiv:2511.01292v2 Announce Type: replace-cross Abstract: Pretrained Transformers can perform in-context learning (ICL) from a few demonstrations, but this ability can fail sharply when the test distr

Tutorials

PriorVLA: Prior-Preserving Adaptation for Vision-Language-Action Models

arXiv:2605.10925v1 Announce Type: new Abstract: Large-scale pretraining has made Vision-Language-Action (VLA) models promising foundations for generalist robot manipulation, yet adapting them to downs

tutorialsarxiv-cs-ro

Rhamba: Region-Aware Hybrid Attention-Mamba Framework for Self-Supervised Learning in Resting-State fMRI

arXiv:2605.01240v1 Announce Type: new Abstract: Self-supervised pretraining is promising for large-scale neuroimaging, yet the impact of region-aware masking and hybrid sequence modeling remains under

5 May 2026

You've Got a Golden Ticket: Improving Generative Robot Policies With A Single Noise Vector

arXiv:2603.15757v2 Announce Type: replace-cross Abstract: What happens when a pretrained generative robot policy is provided a constant initial noise as input, rather than repeatedly sampling it from

safetyarxiv-cs-ai

13 Apr 2026

UniLACT: Depth-Aware RGB Latent Action Learning for Vision-Language-Action Models

arXiv:2602.20231v2 Announce Type: replace-cross Abstract: Latent action representations learned from unlabeled videos have recently emerged as a promising paradigm for pretraining vision-language-acti

applicationsarxiv-cs-cv

10 Apr 2026

Tutorials

What do Language Models Learn and When? The Implicit Curriculum Hypothesis

arXiv:2604.08510v1 Announce Type: new Abstract: Large language models (LLMs) can perform remarkably complex tasks, yet the fine-grained details of how these capabilities emerge during pretraining rema

tutorialsarxiv-cs-cl

10 Apr 2026

Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

arXiv:2610.17233v1 Announce Type: new Abstract: Contrastive vision-language (VL) pretraining in computational pathology has generally relied on curated image-caption datasets sourced from social media

BridgeTwist: Twisting Schrodinger Bridges for Training-Free Conditional Sampling

arXiv:2610.18658v1 Announce Type: new Abstract: Generative models are often required to serve many downstream conditional tasks. In the training-free setting, a pretrained unconditional model is reuse

Tutorials

Characterizing the Separability Gap in Prior-Data Fitted Networks

arXiv:2610.14989v1 Announce Type: new Abstract: Prior-data fitted networks (PFNs) are transformers pretrained on simulated datasets to approximate Bayesian posterior predictive distributions. Standard

tutorialsarxiv-cs-lg

Data-Free Continual Learning for World Action Model with Rectified Generative Replay

arXiv:2610.18038v1 Announce Type: cross Abstract: World action models (WAMs) built on pretrained video generation models adapt quickly to new robotic tasks, but they forget previously learned tasks un

DEFT: Disentanglement-Enhanced Fine-Tuning for EEG Foundation Models

arXiv:2610.14497v1 Announce Type: new Abstract: EEG foundation models (EFMs) have improved EEG representation learning by transferring knowledge from large-scale multi-subject pretraining to downstrea

Layer-wise Input Recoverability in Deep Neural Networks

arXiv:2610.14290v1 Announce Type: new Abstract: The inner workings of deep neural networks are still not well understood. The information bottleneck framework, which characterizes deep networks by the

TRACE: Efficient Transfer Learning for Neural Whole-Body Control Across Humanoid Embodiments

arXiv:2610.17999v1 Announce Type: new Abstract: Humanoid behavior foundation models (BFMs) enable diverse whole-body control, but adapting a pretrained controller to a new embodiment remains challengi

safetyarxiv-cs-ro

Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels

arXiv:2610.13674v1 Announce Type: cross Abstract: Linearizing softmax attention offers a route to efficient Transformer inference, but preserving pretrained attention remains challenging. We identify

12 Oct 2026

Overcoming Prior Barriers: Supervised Fine-Tuning under Long-Tail Distribution

arXiv:2610.12345v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) adapts pretrained large language models (LLMs) to downstream tasks, but the required concepts can receive substantially dif

9 Oct 2026

Rare Gate Disagreements Can Limit Plasticity: When Gradient Flow Mispredicts Finite-Batch SGD

arXiv:2610.11475v1 Announce Type: new Abstract: Population gradient flow is a common tool for reasoning about how neural networks adapt, including after pretraining. We show that it can mispredict fin

9 Oct 2026

Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives

arXiv:2610.11734v1 Announce Type: cross Abstract: We introduce Timer-M1, a pretrained multivariate time series foundation model that learns with primitives for zero-shot forecasting. Across domains, t

applicationsarxiv-cs-ai

9 Oct 2026

LiDAR Resolution Recovery via Foundation-Model-Guided Diffusion

arXiv:2610.08620v1 Announce Type: new Abstract: High-beam-count LiDAR sensors are costly, yet many perception pipelines require dense angular sampling. Using a pretrained Stable Diffusion model as the

7 Oct 2026

Erased, Rerouted, or Rescaled? Post-Training and the Causal Quotient of a Language Model's Belief State

arXiv:2610.05292v1 Announce Type: new Abstract: What happens to information a pretrained model already encodes when post-training no longer rewards using it? The common language of representation comp

safetyarxiv-cs-lg

FASTER: Fast Adjoint Stochastic Transport for Endpoint Refinement in Reward-Guided Image Editing

arXiv:2610.04538v1 Announce Type: new Abstract: Reward-guided image editing at test time seeks to improve a specified reward while preserving source content and visual plausibility. Many existing appr

GRAM: Correcting Frozen Time-Series Foundation Models via Graph-Retrieved Amplitude Memory

arXiv:2610.04827v1 Announce Type: new Abstract: Time-series foundation models (TSFMs) enable zero-shot forecasting through large-scale cross-domain pretraining, while retrieval augmentation further im

safetyarxiv-cs-lg

PWM: Personalized World Models with Online Reinforcement Learning

arXiv:2610.04920v1 Announce Type: new Abstract: Pretrained world models can generate diverse environments, yet users often want to explore a particular scene specified by their own video. This require