Search: “pretraining”

Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts

arXiv:2605.29283v1 Announce Type: cross Abstract: Recent physics foundation models claim general spatiotemporal forecasting ability, yet their evaluations often collapse performance into a single aver

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

DGX agent

arXiv:2601.14758v4 Announce Type: replace-cross Abstract: Post-training pretrained autoregressive models (ARMs) into masked diffusion models (MDMs) has emerged as a cost-effective way to overcome the

researcharxiv-cs-ai

29 May 2026

Stable-Layers: Fine-Tuning Image Layer Decomposition Models with VLM-Scored Reinforcement Learning

arXiv:2605.30257v1 Announce Type: new Abstract: We present Stable-Layers, a reinforcement learning framework that eliminates the need for paired supervision by fine-tuning a pretrained layer decomposi

The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling

arXiv:2605.28864v1 Announce Type: new Abstract: The Cognitive Categorical Transformer (CCT) is a 306M-parameter architecture that augments a pretrained GPT-2 Small backbone with cognitively grounded c

Local Ai

Unsupervised Semantic Segmentation Facilitates Model Understanding

arXiv:2605.29691v1 Announce Type: new Abstract: Self-supervised learning (SSL) has produced a diverse landscape of vision transformers (ViTs) whose pretrained representations support a wide range of d

local-aiarxiv-cs-cv

Bio-Inspired Self-Supervised Learning for Wrist-worn Accelerometer Data

arXiv:2603.10961v2 Announce Type: replace Abstract: Wearable accelerometers enable large-scale health monitoring, yet learning robust human-activity representations has been constrained by scarce labe

Entropy-aware Masking for Masked Language Modeling

arXiv:2605.28526v1 Announce Type: new Abstract: Masked language modeling has become a standard pretraining objective for training encoder-based language models. In this approach, certain tokens in the

Formula-One Prompting: A Composable Equation-First Prefix for Applied Mathematics

arXiv:2601.19302v3 Announce Type: replace Abstract: This paper introduces Formula Prompting (FP) and Formula-One Prompting (F-1), two single-call methods that elicit governing equations before solving

tutorialsarxiv-cs-cl

Hardware

Heterogeneous Parallelism for Multimodal Large Language Model Training

arXiv:2605.27678v1 Announce Type: new Abstract: Foundation model training is becoming multimodal, from post-training pipelines to large-scale pretraining. As modality coverage broadens, context window

hardwarearxiv-cs-lg

On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation

arXiv:2601.06329v2 Announce Type: replace-cross Abstract: Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content while preserving a

Safe In-Context Reinforcement Learning

model-releasesarxiv-cs-lg

arXiv:2509.25582v3 Announce Type: replace Abstract: In-context reinforcement learning (ICRL) is an emerging RL paradigm where an agent, after pretraining, can adapt to out-of-distribution test tasks w

Sense Representations Are Inducible Interfaces

arXiv:2605.28669v1 Announce Type: cross Abstract: Sense representations (explicit, per-token meaning decompositions) are useful for disambiguation, steering, and cross-lingual alignment, but existing

safetyarxiv-cs-ai

Sparse Scheduled Diffusion Guidance for Inverse Problems

arXiv:2603.07860v2 Announce Type: replace Abstract: Pretrained diffusion models are effective priors for Bayesian inverse problems, but posterior sampling with these priors is often costly because dat

Erased but Exploitable: Black-box Embedding-Aware Prompting Against Unlearned Text-to-Image Diffusion Models

arXiv:2605.26332v1 Announce Type: cross Abstract: Machine unlearning aims to remove specific concepts from pretrained text-to-image diffusion models, yet several white- and black-box attacks have been

safetyarxiv-cs-ai

27 May 2026

Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling

arXiv:2605.27286v1 Announce Type: cross Abstract: Time series foundation models (TSFMs) are transforming the forecasting paradigm through large-scale cross-domain pretraining. However, most existing T

27 May 2026

Model Unlearning Objectives Vary for Distinct Language Functions

arXiv:2605.26454v1 Announce Type: new Abstract: Large language models (LLMs) learn undesirable properties during pretraining, including dangerous knowledge and toxic text generation. Just as post-trai

tutorialsarxiv-cs-cl

27 May 2026

All Leaks Count, Some Count More: Interpretable Temporal Contamination Detection and Mitigation in LLM Backtesting

arXiv:2602.17234v2 Announce Type: replace Abstract: Backtesting LLMs on resolved events assumes models reason only from pre-cutoff knowledge, yet pretrained models inevitably leak post-cutoff knowledg

Hardware

Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs

arXiv:2605.24684v1 Announce Type: cross Abstract: Multimodal Attributed Graph Learning (MAGL) integrates intrinsic node attributes with structural topology via graph aggregation. However, as pretraine

hardwarearxiv-cs-ai

Concept Unlearning via Cross-Attention Activation Projection for Diffusion Models

arXiv:2605.25765v1 Announce Type: cross Abstract: Concept unlearning aims to erase a target concept from a pretrained text-to-image diffusion model without retraining. Closed-form methods are attracti

Context-CoT: Enhancing Context Learning via High-Quality Reasoning Synthesis

arXiv:2605.25354v1 Announce Type: new Abstract: While LLMs excel at reasoning over prompts using static pretrained knowledge, they struggle significantly with context learning-the ability to dynamical

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

arXiv:2605.25477v1 Announce Type: cross Abstract: The ability to efficiently and reliably learn new tasks has been a foundational challenge in robotics. Vision-Language-Action (VLA) models have demons

tutorialsarxiv-cs-ai

Factorize to Generalize: Retrieval-Guided Invariant-Dynamic Decomposition for Time Series Forecasting

arXiv:2605.24911v1 Announce Type: cross Abstract: Time series foundation models (TSFMs) have recently achieved strong zero-shot forecasting performance through large-scale pretraining and retrieval-au

tutorialsarxiv-cs-ai

Uncovering the Latent Potential of Deep Intermediate Representations

arXiv:2605.23033v1 Announce Type: cross Abstract: Foundational Models pretrained on huge amount of data learn representations that evolve across depth, forming a hierarchy of embeddings with distinct

tutorialsarxiv-cs-ai

25 May 2026

Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation

arXiv:2602.02214v3 Announce Type: replace Abstract: To achieve real-time interactive video generation, current methods distill pretrained bidirectional video diffusion models into few-step autoregress

researcharxiv-cs-cv

Hierarchical Variational Policies for Reward-Guided Diffusion

arXiv:2605.21661v1 Announce Type: cross Abstract: Adapting pretrained diffusion models to downstream objectives such as inverse problems often requires expensive test-time guidance or optimization. We

MAP4TS: A Multi-Aspect Prompting Framework for Time-Series Forecasting with Large Language Models

arXiv:2510.23090v2 Announce Type: replace Abstract: Recent advances have investigated the use of pretrained large language models (LLMs) for time-series forecasting by aligning numerical inputs with L

Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling

arXiv:2605.22484v1 Announce Type: new Abstract: Vision-Language Models (VLMs) excel at tasks like zero-shot classification and cross-modal retrieval by mapping images and text to a shared space, but t

Chronicle: A Multimodal Foundation Model for Joint Language and Time Series Understanding

arXiv:2605.20268v1 Announce Type: cross Abstract: Real-world time series come with text: metadata, descriptions, news, reports. Yet time series foundation models process numerical sequences in isolati

Applications

GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation

arXiv:2605.20752v1 Announce Type: new Abstract: Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors from pretrained vision-lan

applicationsarxiv-cs-ro

Applications

Latent Space Guided Scenario Sampling for Multimodal Segmentation Under Missing Modalities

arXiv:2605.20372v1 Announce Type: new Abstract: Multimodal semantic segmentation benefits remote sensing analysis by combining complementary information from different sensor modalities. In real-world

applicationsarxiv-cs-cv

LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws

arXiv:2502.12120v3 Announce Type: replace-cross Abstract: Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and co

Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics

arXiv:2602.24138v2 Announce Type: replace Abstract: Automated recognition of surgical phases and steps is a fundamental capability for intraoperative decision support, workflow automation, and skill a

ROAR-3D: Routing Arbitrary Views for High-Fidelity 3D Generation

arXiv:2605.21121v1 Announce Type: new Abstract: Single-image-to-3D generative models can now produce high-quality geometry, yet conditioning on a single view inevitably introduces ambiguity about unse

researcharxiv-cs-cv

Today we release a study on decoupling the benefits of subword tokenization for language model training, by simulating each suspected benefi…

model-releasesnous-research--x

Today we release a study on decoupling the benefits of subword tokenization for language model training, by simulating each suspected benefit one at a time inside a 1.7B byte-level pretraining pipelin

Variance Reduction for Expectations with Diffusion Teachers

arXiv:2605.21489v1 Announce Type: cross Abstract: Pretrained diffusion models serve as frozen teachers feeding downstream pipelines such as text-to-3D, single-step distillation, and data attribution.

researcharxiv-cs-cv

A Bitter Lesson for Data Filtering

arXiv:2605.19407v1 Announce Type: cross Abstract: We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an app

Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls

arXiv:2605.19728v1 Announce Type: new Abstract: Foundation video models produce visually impressive results, but their use in embodied AI remains limited because they are primarily trained on natural

Block-Based Double Decoders

arXiv:2605.18807v1 Announce Type: cross Abstract: Encoder-decoder models offer substantial inference-time savings over decoder-only models, but their pretraining objectives suffer from sparse supervis

Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers

arXiv:2601.15014v2 Announce Type: replace-cross Abstract: We study in-context learning for nonparametric regression with alpha-Holder smooth regression functions, for some alpha>0. We prove that, with

Nonlinearity as Rank: Generative Low-Rank Adapter with Radial Basis Functions

arXiv:2602.05709v2 Announce Type: replace Abstract: Low-rank adaptation (LoRA) approximates the update of a pretrained weight matrix using the product of two low-rank matrices. However, standard LoRA

TrajTok: Adaptive Spatial Tokenization for Trajectory Representation Learning

arXiv:2605.20134v1 Announce Type: new Abstract: Learning generalizable trajectory representations from raw GPS traces remains difficult because the data is continuous, noisy, and irregularly sampled.

Where Does Authorship Signal Emerge in Encoder-Based Language Models?

arXiv:2605.19908v1 Announce Type: new Abstract: Authorship attribution models fine-tuned with the same pretrained encoder, data, and loss can differ four-fold in performance depending only on their sc

researcharxiv-cs-cl

Local Ai

ADMEDTAGGER: an annotation framework for distillation of expert knowledge for the Polish medical language

arXiv:2601.09722v2 Announce Type: replace-cross Abstract: In this work, we present an annotation framework that demonstrates how a multilingual LLM pretrained on a large corpus can be used as a teache

local-aiarxiv-cs-ai

Benchmarking Recurrent Event-Based Object Detection for Industrial Multi-Class Recognition on MTevent

arXiv:2603.21787v2 Announce Type: replace Abstract: Event cameras are attractive for industrial robotics because they provide high temporal resolution, high dynamic range, and reduced motion blur. How

MixSD: Mixed Contextual Self-Distillation for Knowledge Injection

arXiv:2605.16865v1 Announce Type: new Abstract: Supervised fine-tuning (SFT) is widely used to inject new knowledge into language models, but it often degrades pretrained capabilities such as reasonin

MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation

arXiv:2605.17743v1 Announce Type: new Abstract: Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams while retaining past competence, yet texture

RAVE: Re-Allocating Visual Attention in Large Multimodal Models

arXiv:2605.18359v1 Announce Type: new Abstract: Large multimodal models (LMMs) inherit the self-attention mechanism of pretrained language backbones, yet standard attention can exhibit suboptimal allo

Reverse-Engineering Model Editing on Language Models

arXiv:2602.10134v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are pretrained on corpora containing trillions of tokens and, therefore, inevitably memorize sensitive informatio