Search: “pretraining”

Training Transformers for KV Cache Compressibility

arXiv:2605.05971v2 Announce Type: replace Abstract: Long-context language modeling is increasingly constrained by the Key-Value (KV) cache, whose memory and decode-time access costs scale linearly wit

13 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Bangla-WhisperDiar: Fine-Tuning Whisper and PyAnnote for Bangla Long-Form Speech Recognition and Speaker Diarization

DGX agent

arXiv:2605.08214v1 Announce Type: cross Abstract: Automatic Speech Recognition (ASR) and speaker diarization in Bangla remain challenging due to long form recordings, diverse acoustic conditions, and

researcharxiv-cs-ai

12 May 2026

BenchHAR: Benchmarking Self-Supervised Learning for Generalizable Sensor-based Activity Recognition

model-releasesarxiv-cs-cv

arXiv:2605.08296v1 Announce Type: new Abstract: Human Activity Recognition (HAR) from wearable sensors supports broad healthcare and behavior science applications. However, data heterogeneity and the

Control-Augmented Autoregressive Diffusion for Data Assimilation

arXiv:2510.06637v3 Announce Type: replace-cross Abstract: Despite advances in test-time scaling and diffusion finetuning, guidance for Auto-Regressive Diffusion Models (ARDMs) remains underexplored. W

Cross-Modal Semantic-Enhanced Diffusion Framework for Diabetic Retinopathy Grading

arXiv:2605.09242v1 Announce Type: cross Abstract: Automated grading of diabetic retinopathy (DR) faces several critical challenges: subtle inter-grade visual distinctions in fine-grained lesion patter

DiffATS: Diffusion in Aligned Tensor Space

model-releasesarxiv-cs-lg

arXiv:2605.09275v1 Announce Type: new Abstract: Direct diffusion modeling of high-resolution spatiotemporal fields is computationally challenging. Parameter-efficient primitives address this by repres

EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting

arXiv:2602.01194v2 Announce Type: replace Abstract: Long-term weather forecasting is critical for socioeconomic planning and disaster preparedness. While recent approaches employ finetuning to extend

Inverse Design for Conditional Distribution Matching

arXiv:2605.09439v1 Announce Type: new Abstract: Generative models are powerful tools for sampling from a learned distribution P(Y mid X), and inverse-design methods invert this map to find an input x

Large Language Models for Sequential Decision-Making: Improving In-Context Learning via Supervised Fine-Tuning

arXiv:2605.09009v1 Announce Type: cross Abstract: Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities, yet their potential for sequential decision-making remains

LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language

model-releasesarxiv-cs-cl

arXiv:2605.09015v1 Announce Type: new Abstract: Sardinian, a Romance language with roughly one million speakers, has minimal presence in modern NLP. Commercial services do not support it, and current

MicroDiffuse3D: A Foundation Model for 3D Microscopy Imaging Restoration

arXiv:2605.08566v1 Announce Type: new Abstract: Chemical imaging enables label-free visualization of cells, tissues and living systems while providing direct biochemical information that is difficult

On Distinguishing Capability Elicitation from Capability Creation in Post-Training: A Free-Energy Perspective

arXiv:2605.08368v1 Announce Type: new Abstract: Debates about large language model post-training often treat supervised fine-tuning (SFT) as imitation and reinforcement learning (RL) as discovery. But

local-aiarxiv-cs-ai

Positional Encoding via Token-Aware Phase Attention

arXiv:2509.12635v3 Announce Type: replace-cross Abstract: We prove under practical assumptions that Rotary Positional Embedding (RoPE) introduces an intrinsic distance-dependent bias in attention scor

Preserve and Personalize: Personalized Text-to-Image Diffusion Models without Distributional Drift

model-releasesarxiv-cs-cv

arXiv:2505.19519v3 Announce Type: replace Abstract: Personalizing text-to-image diffusion models involves integrating novel visual concepts from a small set of reference images while retaining the mod

ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design

arXiv:2605.10189v1 Announce Type: cross Abstract: Designing proteins with desired functions or properties represents a core goal in synthetic biology and drug discovery. Recent advances in protein lan

Scaling Vision Models Does Not Consistently Improve Localisation-Based Explanation Quality

arXiv:2605.10142v1 Announce Type: cross Abstract: Artificial intelligence models are increasingly scaled to improve predictive accuracy, yet it remains unclear whether scale improves the quality of po

Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning

model-releasesarxiv-cs-lg

arXiv:2605.10247v1 Announce Type: new Abstract: Using Large Language Models (LLMs) to process graph-structured data is an active research area, yet current state-of-the-art approaches typically rely o

TFM-Retouche: A Lightweight Input-Space Adapter for Tabular Foundation Models

arXiv:2605.06047v2 Announce Type: replace-cross Abstract: Tabular foundation models (TFMs), such as TabPFN-2.6, TabICLv2, ConTextTab, Mitra, LimiX, and TabDPT, achieve strong zero-shot performance thr

Tracing Moral Foundations in Large Language Models

arXiv:2601.05437v2 Announce Type: replace-cross Abstract: Large language models often produce human-like moral judgments, but it is unclear whether this reflects an internal conceptual structure or su

WATCH: Wide-Area Archaeological Site Tracking for Change Detection

arXiv:2605.08160v1 Announce Type: cross Abstract: Monitoring archaeological sites at scale is vital for protecting cultural heritage, yet pinpointing when disturbances occur remains difficult because

WavesFM: Hierarchical Representation Learning for Longitudinal Wearable Sensor Waveforms

arXiv:2605.09173v1 Announce Type: cross Abstract: Wearable sensors enable the continuous acquisition of high-resolution physiological waveforms, such as photoplethysmography and accelerometry, under f

local-aiarxiv-cs-ai

When More Parameters Hurt: Foundation Model Priors Amplify Worst-Client Disparity Under Extreme Federated Heterogeneity

arXiv:2605.08992v1 Announce Type: new Abstract: Federated learning (FL) is increasingly used to fine-tune foundation models (FMs) on distributed private data. The community largely assumes that large-

Cloud Storage Rapid: Turbocharged object storage for AI and analytics

model-releasesgoogle-cloud-ai

At Google Cloud Next ’26 we announced Cloud Storage Rapid, a family of object storage capabilities for data-intensive workloads like AI and analytics. Out of the gate, Cloud Storage Rapid consists of

Excluding the Target Domain Improves Extrapolation: Deconfounded Hierarchical Physics Constraints

arXiv:2605.07485v1 Announce Type: cross Abstract: Extrapolation to out-of-distribution conditions is a fundamental challenge for physics-constrained deep generative models. Existing methods apply phys

Inference-Time Attribute Distribution Alignment for Unconditional Diffusion

arXiv:2605.07456v1 Announce Type: new Abstract: Inference-time controllable generation is essential for real-world applications of unconditional diffusion models. However, most existing techniques foc

Applications

Large Video Planner Enables Generalizable Robot Control

arXiv:2512.15840v2 Announce Type: replace-cross Abstract: General-purpose robots require decision-making models that generalize across diverse tasks and environments. Recent works build robot foundati

applicationsarxiv-cs-cv

Pre-trained Tabular Foundation Models as Versatile Summary Networks for Neural Posterior Estimation

arXiv:2605.07765v1 Announce Type: new Abstract: In this work, we study TabPFN as a training-free, modular summary network for simulation-based Bayesian inference (SBI). Tabular foundation models such

Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts

arXiv:2605.07307v1 Announce Type: new Abstract: Modern reasoning language models generate dense, sequential chain-of-thought traces implicitly assuming that every token contributes and that steps must

researcharxiv-cs-cl

Retrieve, Integrate, and Synthesize: Spatial-Semantic Grounded Latent Visual Reasoning

arXiv:2605.07106v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have made remarkable progress on vision-language reasoning, yet most methods still compress visual evidence int

researcharxiv-cs-cl

Slowly Annealed Langevin Dynamics: Theory and Applications to Training-Free Guided Generation

arXiv:2605.07950v1 Announce Type: new Abstract: We study Slowly Annealed Langevin Dynamics (SALD), a sampler for tracking a path of moving target distributions and approximating the terminal target th

TextLDM: Language Modeling with Continuous Latent Diffusion

arXiv:2605.07748v1 Announce Type: new Abstract: Diffusion Transformers (DiT) trained with flow matching in a VAE latent space have unified visual generation across images and videos. A natural next st

safetyarxiv-cs-cl

Tutorials

Transfer Learning Across Fast- and Full-Simulation Domains in High-Energy Physics

arXiv:2605.07471v1 Announce Type: new Abstract: Machine-learning models in high-energy physics are often trained on simulated data, where fully simulated samples are computationally expensive while fa

tutorialsarxiv-cs-lg

A large-scale heterogeneous 3D magnetic resonance brain imaging dataset for self-supervised learning

arXiv:2506.14432v3 Announce Type: replace-cross Abstract: We present FOMO260K, a large-scale, heterogeneous dataset of 260,927 brain Magnetic Resonance Imaging (MRI) scans from 77,589 MRI sessions and

FMI_SU_Yotkova_Kastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals

arXiv:2605.04157v1 Announce Type: new Abstract: SemEval-2026 Task 13 investigates machine-generated code detection across multiple programming languages and application scenarios, asking participating

researcharxiv-cs-cl

POMA-3D: The Point Map Way to 3D Scene Understanding

arXiv:2511.16567v3 Announce Type: replace Abstract: In this paper, we introduce POMA-3D, the first self-supervised 3D representation model learned from point maps. Point maps encode explicit 3D coordi

safetyarxiv-cs-cv

Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training

arXiv:2605.04913v1 Announce Type: new Abstract: LLM post-training typically propagates task gradients through the full depth of the model. Although this end-to-end structure is simple and general, it

local-aiarxiv-cs-cl

TumorXAI: Self-Supervised Deep Learning Framework for Explainable Brain MRI Tumor Classification

arXiv:2605.01999v1 Announce Type: new Abstract: Classifying brain tumors using magnetic resonance imaging (MRI) is crucial for early diagnosis and treatment; however, tumor heterogeneity and a dearth

researcharxiv-cs-ai

6 May 2026

High-Fidelity Mobile Avatars with Pruned Local Blendshapes

arXiv:2605.01854v1 Announce Type: new Abstract: We propose a method to reconstruct high-fidelity human avatars from multi-view video that can run on mobile devices. Many works can model high-quality G

local-aiarxiv-cs-cv

Linearizing Vision Transformer with Test-Time Training

arXiv:2605.02772v1 Announce Type: new Abstract: While linear-complexity attention mechanisms offer a promising alternative to Softmax attention for overcoming the quadratic bottleneck, training such m

local-aiarxiv-cs-cv

Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models

model-releasesarxiv-cs-cl

arXiv:2602.04509v4 Announce Type: replace Abstract: Fine-tuning Multimodal Large Language Models (MLLMs) on task-specific data is an effective way to improve performance on downstream applications. Ho

Perturb and Correct: Post-Hoc Ensembles using Affine Redundancy

arXiv:2605.01632v1 Announce Type: new Abstract: Models that are indistinguishable on in-distribution data can behave very differently under distribution shift. We introduce Perturb-and-Correct (P&C),

PointCSP: Cross-Sample Semantic Propagation and Stability Preservation in Self-Supervised Point Cloud Learning

model-releasesarxiv-cs-cv

arXiv:2605.01759v1 Announce Type: new Abstract: Scene-level point cloud self-supervised learning (PC-SSL) has demonstrated potential in enhancing the generalization capability of 3D vision models. Des

Probe-Geometry Alignment: Erasing the Cross-Sequence Memorization Signature Below Chance

model-releasesarxiv-cs-lg

arXiv:2605.01699v1 Announce Type: new Abstract: Recent attacks show that behavioural unlearning of large language models leaves internal traces recoverable by adversarial probes. We characterise where

Representation learning from OCT images

arXiv:2605.02589v1 Announce Type: new Abstract: Optical Coherence Tomography (OCT) has become one of the most used imaging modality in ophthalmology. It provides high-resolution, non-invasive visualiz

safetyarxiv-cs-cv

Tutorials

Rethinking Electro-Optical Vision Foundation Models for Remote Sensing Retrieval: A Controlled Comparison with Generalist VFM

arXiv:2605.02283v1 Announce Type: new Abstract: Vision foundation models have attracted significant attention for their ability to leverage large-scale unlabeled visual data. This advantage is particu

tutorialsarxiv-cs-cv

Alethia: A Foundational Encoder for Voice Deepfakes

model-releasesarxiv-cs-cl

arXiv:2605.00251v1 Announce Type: cross Abstract: Existing voice deepfake detection and localization models rely heavily on representations extracted from speech foundation models (SFMs). However, dow

4 May 2026

Something from Nothing: Data Augmentation for Robust Severity Level Estimation of Dysarthric Speech

arXiv:2603.15988v2 Announce Type: replace-cross Abstract: Dysarthric speech quality assessment (DSQA) is critical for clinical diagnostics and inclusive speech technologies. However, subjective evalua

4 May 2026

CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Model

arXiv:2506.09110v3 Announce Type: replace Abstract: Electroencephalography (EEG) provides real-time insights into brain activity and supports diverse applications in neuroscience. While EEG foundation