Search: “pretraining”

arXiv:2605.21075v1 Announce Type: new Abstract: Earth observation (EO) foundation models (FMs) are increasingly trained on multisensor data, spanning multispectral imagery (MSI), synthetic aperture ra

arXiv:2605.19282v1 Announce Type: new Abstract: Muon is a matrix-aware optimizer that leverages Newton-Schulz (NS) iterations to enforce spectral gradient orthogonalization by driving all singular val

This article from Latent Space provides guidance on securing employment at cutting-edge AI research laboratories, with a focus on roles related to pretraining large language models. It likely covers p

arXiv:2603.07900v2 Announce Type: replace Abstract: Foundation models pretrained on electronic health records (EHR) have demonstrated zero-shot clinical prediction capabilities by generating synthetic

arXiv:2605.15599v1 Announce Type: cross Abstract: Extreme low-data fine-grained classification is common in expert domains where labeling is expensive, yet practitioners still need principled guidance

arXiv:2605.13769v1 Announce Type: cross Abstract: We study dense and mixture-of-experts (MoE) transformers in a tiny-scale pretraining regime under a shared LLaMA-style decoder training recipe. The sp

Anthropic: Anthropic researchers detail “model spec midtraining”, which adds a stage between pretraining and fine-tuning to improve generalization from alignment training — Sara Price2, Samuel Marks2,

arXiv:2605.02500v1 Announce Type: new Abstract: Sleep foundation models have recently demonstrated strong performance on in-domain polysomnography tasks, including sleep staging, apnea detection, and

arXiv:2312.12339v3 Announce Type: replace Abstract: Understanding visual inputs for a given task amidst varied changes is a key challenge posed by visual reinforcement learning agents. We propose exti

arXiv:2511.18903v2 Announce Type: replace-cross Abstract: Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even

arXiv:2604.16952v1 Announce Type: new Abstract: Learning robust representations across extremely heterogeneous modalities remains a fundamental challenge in multi-modal vision. As a critical and profo

arXiv:2604.13977v1 Announce Type: new Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing stra

arXiv:2610.18870v1 Announce Type: new Abstract: We generally perceive that training text-to-image diffusion models requires heavy compute: it is true if we aim at a comprehensive frontier model. This

arXiv:2610.17637v1 Announce Type: new Abstract: Since scaling laws were introduced for modern LLMs (Kaplan et al., 2020), several works have extended them to account for quantization, tracking the eme

arXiv:2610.17867v1 Announce Type: new Abstract: Foundation models for tables and relational databases are increasingly used for predictive tasks across various scientific and industrial domains. Howev

arXiv:2610.12528v1 Announce Type: new Abstract: Continuous glucose monitoring (CGM) foundation models encode each day as one vector that mixes who the person is, what happened that day and how the sen

arXiv:2610.11159v1 Announce Type: new Abstract: Speech-to-LLM systems often connect a frozen speech encoder to a frozen large language model (LLM) through a small trainable bridge. The bridge is usual

arXiv:2610.09611v1 Announce Type: new Abstract: Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features

arXiv:2610.09700v1 Announce Type: new Abstract: Synthetic hard negatives generated in the representation space have proven effective for unimodal self-supervised learning, but transferring this idea t

arXiv:2610.02347v1 Announce Type: new Abstract: Training-data attribution aims to identify which training examples shape model behavior, yet validating such claims is difficult because causal training

arXiv:2610.00053v1 Announce Type: new Abstract: Four-bit floating-point (FP4) Tensor Cores accelerate matrix multiplication, but scale computation, operand packing, layout construction, and saved back

arXiv:2603.11950v2 Announce Type: replace-cross Abstract: Multimodal language models have demonstrated strong semantic understanding and reasoning over physiological and behavioral signals captured fr

arXiv:2609.33591v1 Announce Type: new Abstract: Low-precision Transformer systems increasingly quantize attention matrix multiplications, while softmax often remains at higher precision. During pretra

arXiv:2609.30043v1 Announce Type: new Abstract: Dense vessel annotation in digital subtraction angiography (DSA) is labor-intensive, yet every unlabeled sequence records how contrast passes through th

arXiv:2609.26290v1 Announce Type: cross Abstract: Causal tabular foundation models amortize effect estimation across synthetic mechanisms, but latent-effect supervision rewards posterior shrinkage ins

arXiv:2609.26617v1 Announce Type: cross Abstract: Clinical decision making heavily relies on predicting the disease progression trajectory by seeking to understand patient's health status which is cha

arXiv:2609.21948v1 Announce Type: cross Abstract: Learning large-scale vision-language-action (VLA) models from multi-embodiment datasets remains challenging due to heterogeneous action spaces across

arXiv:2509.25050v2 Announce Type: replace Abstract: Reinforcement Learning (RL) has emerged as a central paradigm for advancing Large Language Models (LLMs), where both pre-training and RL post-traini

arXiv:2609.01151v1 Announce Type: new Abstract: In the standard LM training pipeline, subword tokenisation is applied as a preprocessing step. Subword segmental language modelling is an alternative pa

Sori-1B is a 1B-parameter audio-language model built by a single SNU researcher whose core claim to fame is that its decoder is trained entirely from scratch on audio-paired text — no text-only pretra

arXiv:2602.03702v2 Announce Type: replace-cross Abstract: Large language models are increasingly trained in continual or open-ended settings, where the total training horizon is not known in advance.

Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown promise in token ef

arXiv:2608.24381v1 Announce Type: new Abstract: Self-supervised learning (SSL) has emerged as a promising approach for tabular data, yet its efficacy under extreme label scarcity and test-time missing

arXiv:2608.15412v1 Announce Type: cross Abstract: Encoder-based code representation models remain widely deployed for discriminative tasks such as clone detection and code classification, where their

arXiv:2608.09264v1 Announce Type: new Abstract: Magnetic field strength is a major source of domain shift in magnetic resonance imaging (MRI), affecting signal-to-noise ratio, tissue contrast, spatial

arXiv:2608.00432v1 Announce Type: new Abstract: Deep research agents are often trained on expensive, environment-grounded tool-use trajectories that require repeated retrieval, document inspection, an

arXiv:2608.00135v1 Announce Type: cross Abstract: Design and architectural archives encode expert human knowledge in graphical formats, providing a critical testbed for design-inspired Machine Learnin

arXiv:2607.27372v1 Announce Type: cross Abstract: The deep learning revolution, kicked off by AlexNet, taught us that end-to-end training beats decomposing a problem into hand-designed stages. Generat

arXiv:2607.24769v1 Announce Type: new Abstract: With the growing capabilities of frontier models, AI alignment becomes increasingly critical in high-risk deployment settings. While recent work has emp

arXiv:2607.21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks. This limited approach can make it challenging to

arXiv:2607.19524v1 Announce Type: cross Abstract: Federated learning (FL) offers a promising approach to privacy-preserving clinical risk prediction, but its deployment remains limited by restricted d

arXiv:2607.03925v1 Announce Type: new Abstract: EEG foundation models have shown strong potential in learning generalized representations across subjects and tasks. However, most existing approaches f

arXiv:2607.05247v1 Announce Type: new Abstract: Dense spatial perception is essential for physical intelligence, where visual systems are expected to recover structured, metric, and actionable represe

arXiv:2607.02466v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models are fundamentally bottlenecked by the scarcity of expert demonstrations -- triplets of observations, instructions,

arXiv:2602.23135v2 Announce Type: replace-cross Abstract: Edge classification on directed dynamic graphs requires modeling interactions between source and destination nodes exhibiting asymmetrical beh

arXiv:2606.15129v2 Announce Type: replace-cross Abstract: Color fundus photography (CFP) is the mainstay of large-scale retinal screening, but its diagnostic capacity is limited by the lack of depth-r