Knowledge catalogue
arXiv:2604.21104v1 Announce Type: new Abstract: New geospatial foundation models introduce a new model architecture and pretraining dataset, often sampled using different notions of data diversity. Pe
arXiv:2410.21582v4 Announce Type: replace-cross Abstract: Large-scale pretrained models are widely leveraged as foundations for learning new specialized tasks via fine-tuning, with the goal of maintai
arXiv:2606.27295v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are commonly pretrained on robot demonstrations by jointly mapping visual observations and language instructions to
arXiv:2606.06194v1 Announce Type: cross Abstract: Egocentric human video offers a scalable alternative to robot data for pretraining, yet models pretrained on such video consistently underperform thos
arXiv:2607.16097v2 Announce Type: replace-cross Abstract: Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is l
arXiv:2608.08800v1 Announce Type: new Abstract: Pretraining LLMs on artificial languages ('pre-pretraining') is a technique that could reportedly increase token efficiency by 33%, i.e., save up to 33%
arXiv:2605.13436v1 Announce Type: cross Abstract: Subword regularization methods such as BPE dropout are typically applied only during fine-tuning, while pretraining is usually done with deterministic
arXiv:2609.17831v1 Announce Type: cross Abstract: Molecular property prediction is often limited by the small size of labeled downstream datasets, motivating pretraining on large corpora of unlabeled
arXiv:2608.13482v1 Announce Type: cross Abstract: As language-model-based AI is increasingly deployed in autonomous settings, aligning its goals and values with those of humans becomes critical. Today
arXiv:2606.07996v1 Announce Type: cross Abstract: Pretraining is fundamental to the development of Large Language Models (LLMs), yet the opacity of pretraining data complicates model analysis and rais
arXiv:2606.07590v1 Announce Type: cross Abstract: Pathology foundation models are pretrained on large streams of WSI-derived patches, while supervision during data construction is often slide-level, s
arXiv:2605.12241v1 Announce Type: cross Abstract: Specialized foundation models are beginning to emerge in various medical subdomains, but pretraining methodologies and parametric scaling with the siz
arXiv:2605.07302v1 Announce Type: new Abstract: Finetuning pretrained models occurs in a low-dimensional subspace of the full parameter space. Prior work has focused on characterizing this optimizatio
arXiv:2606.19468v2 Announce Type: replace Abstract: The narrative composition of web-scale LLM pretraining corpora remains largely unexplored, even though narrative is a fundamental mode of human comm
arXiv:2608.03930v1 Announce Type: cross Abstract: Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining task
arXiv:2608.03260v1 Announce Type: new Abstract: Pretraining has shown strong potential for learning transferable representations, yet it remains underexplored for electron-density-based molecular lear
arXiv:2510.06048v4 Announce Type: replace Abstract: Effective data selection is essential for pretraining large language models (LLMs), enhancing efficiency and improving generalization to downstream
arXiv:2601.21725v2 Announce Type: replace Abstract: Pretraining language models directly on web-scale corpora is the de facto paradigm. We study an alternative where the model is initially exposed to
arXiv:2509.18919v2 Announce Type: replace Abstract: The pretraining-finetuning paradigm is a crucial strategy in metallic surface defect detection for mitigating the challenges posed by data scarcity.
arXiv:2605.25924v1 Announce Type: new Abstract: Recent automated essay scoring (AES) studies increasingly use pretrained transformer models, but these models are usually pretrained on general-domain E
arXiv:2604.16380v1 Announce Type: new Abstract: Large language models (LLMs) rely on pretraining on massive and heterogeneous corpora, where training data composition has a decisive impact on training
arXiv:2511.21613v2 Announce Type: replace Abstract: Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prio
arXiv:2311.04799v2 Announce Type: replace Abstract: Pretraining language models is still a challenge for many researchers due to its substantial computational costs. As such, there is growing interest
arXiv:2610.02784v1 Announce Type: new Abstract: Tactile sensing provides essential contact information for robotic manipulation, yet incorporating it into pretrained vision-language-action (VLA) model
arXiv:2610.02866v1 Announce Type: new Abstract: Financial time-series pretraining typically learns from masked observations, contrastive relations, or future outcomes---yet existing objectives struggl
arXiv:2610.03290v1 Announce Type: cross Abstract: Lung ultrasound (LUS) is attractive for tuberculosis (TB) screening at primary-care level, but labelled cohorts are small. Echocardiography carries no
arXiv:2609.35877v1 Announce Type: cross Abstract: Many DNA foundation models are pretrained by masking parts of a sequence and asking the model to reconstruct them. Standard masked pretraining exposes
arXiv:2609.37642v1 Announce Type: new Abstract: Self-supervised pretraining reshaped prediction in language and vision, and brain foundation models (BFMs) inherited its promise. Representations learne
arXiv:2603.03806v2 Announce Type: replace Abstract: The state space model Mamba has recently emerged as a promising paradigm in computer vision, attracting considerable attention for its efficient han
arXiv:2609.23860v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) commonly reuse visual encoders pretrained with CLIP, although the features of these ViTs are ultimately consume
arXiv:2609.09863v1 Announce Type: new Abstract: Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflicti
arXiv:2609.04555v1 Announce Type: new Abstract: Vision foundation models (VFMs) are valuable in data-scarce domains such as surgery, where a single pretrained backbone can provide rich representations
arXiv:2609.04261v1 Announce Type: cross Abstract: Self-supervised pretraining has transformed language and vision, but its value for molecular graph neural networks remains contested. We ask whether p
arXiv:2608.20005v1 Announce Type: new Abstract: Pretraining time series foundation models across heterogeneous datasets necessitates effective handling of varying sampling frequencies. Current methods
arXiv:2608.19115v1 Announce Type: new Abstract: Modern pretrained encoders make representations from heterogeneous views increasingly reusable, but the procedure that determines view utility and combi
arXiv:2607.24717v1 Announce Type: cross Abstract: Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixe
arXiv:2601.19618v2 Announce Type: replace Abstract: Differential privacy protects the patients whose images train medical imaging models, but it lowers diagnostic accuracy, and the initialization is t
arXiv:2606.22056v1 Announce Type: new Abstract: Adversarial imitation learning (AIL) achieves high-quality imitation compared to behavioral cloning (BC), but demands substantial online environment int
arXiv:2606.16246v2 Announce Type: replace Abstract: As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shift
arXiv:2606.03773v1 Announce Type: new Abstract: High-quality pretraining data is a central ingredient in modern language models, but German-language resources remain far less developed than their Engl
arXiv:2604.15706v1 Announce Type: new Abstract: Everyday tasks come with a target, and pretraining models around this target is what turns them into experts. In this paper, we study target-oriented la
arXiv:2610.08879v1 Announce Type: new Abstract: Pretraining strategies significantly impact the quality of language models, yet existing comparisons of Masked Language Modeling (MLM), Whole Word Maski
arXiv:2609.39403v1 Announce Type: new Abstract: Egocentric human video offers a scalable data source for dexterous manipulation, yet using it to train humanoid robots presents two challenges: (1) an e
arXiv:2609.38764v1 Announce Type: new Abstract: Language models are typically pretrained from random initialization. Recent work challenges this convention, showing that a brief warm-up on abstract, a
arXiv:2609.38446v1 Announce Type: cross Abstract: A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how pretraining and midtraining supp
arXiv:2609.35701v1 Announce Type: new Abstract: The success of large language models (LLMs) has been accompanied by continued growth in model size and pretraining costs. Muon offers high accuracy and
arXiv:2609.35090v1 Announce Type: new Abstract: Video-text pretraining has achieved remarkable progress through the scaling of models and datasets, yet the quality of language supervision remains unde
arXiv:2609.31985v1 Announce Type: new Abstract: Vision-language pretraining objectives differ in the spatial granularity of their supervision, yet the implications of this distribution for chest X-ray
arXiv:2609.35457v1 Announce Type: new Abstract: Most modern multimodal large language models (MLLMs) build on a pretrained visual encoder that provides a strong visual prior. Encoder-free MLLMs instea
arXiv:2609.05766v1 Announce Type: new Abstract: The dominant approach to building domain-specific pretraining corpora is to filter large web archives such as CommonCrawl. This works well for popular d
arXiv:2608.30817v1 Announce Type: cross Abstract: Geospatial foundation models have emerged as state-of-the-art methods for downstream Earth observation tasks. However, existing pretraining methodolog
arXiv:2608.05000v1 Announce Type: new Abstract: Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, t
arXiv:2607.04422v1 Announce Type: cross Abstract: Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving optimizer states, optimizer arithmetic and attention underexplored i
arXiv:2606.14752v2 Announce Type: replace-cross Abstract: Modern Vision-Language-Action (VLA) models must bridge pretrained vision-language reasoning and precise continuous robot control. Existing act
arXiv:2603.13432v4 Announce Type: replace-cross Abstract: Spatial Transcriptomics (ST) profiles thousands of gene expression values at discrete spots with precise coordinates on tissue sections, prese
arXiv:2605.20105v1 Announce Type: new Abstract: Learning to generalise from limited data is a fundamental challenge for both artificial and biological systems. A common strategy is to extract reusable
arXiv:2604.12683v1 Announce Type: new Abstract: Current fMRI foundation models primarily rely on a limited range of brain states and mismatched pretraining tasks, restricting their ability to learn ge
arXiv:2604.09849v1 Announce Type: cross Abstract: We study whether in-domain pretraining of Bidirectional Encoder Representations from Transformer (BERT) model improves subdomain-level detection of ex
arXiv:2604.09258v1 Announce Type: new Abstract: Pretraining is the cornerstone of Large Language Models (LLMs), dominating the vast majority of computational budget and data to serve as the primary en
arXiv:2609.22868v1 Announce Type: new Abstract: End-to-end driving requires planning-relevant bird's-eye-view (BEV) representations, but existing pretraining approaches often rely on task annotations