Search: “pretraining”

arXiv:2608.10473v1 Announce Type: cross Abstract: Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction

arXiv:2608.10595v1 Announce Type: cross Abstract: Proteolysis-targeting chimeras (PROTACs) induce protein degradation by recruiting a target protein to an E3 ubiquitin ligase, making degradation a joi

arXiv:2608.10590v1 Announce Type: new Abstract: Vision Transformers (ViTs) are widely believed to require more labeled data than CNNs for industrial dense prediction. Through controlled experiments on

arXiv:2608.06122v1 Announce Type: cross Abstract: Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether simi

Financial Times: Sources: ByteDance is pretraining an AI model with up to 10T parameters, roughly 3x larger than Kimi K3 and larger than the 8T estimate for Anthropic's Mythos 5 — TikTok owner trainin

arXiv:2608.03089v1 Announce Type: new Abstract: Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level

arXiv:2608.00231v1 Announce Type: new Abstract: Volumetric CT vision-language pretraining learns 3D representations from scan-report pairs, but global and anatomy-aware objectives supervise only corre

arXiv:2603.11838v2 Announce Type: replace Abstract: Large language models pretrained on internet-scale data risk lookahead bias in forecasting tasks, as they may have already seen the true outcome dur

This is for pretraining based on nemotron nvp4 figures, mfu etc Large context, multimodal, long RL we are seeing now could make it a multiple of this Worth noting how good the lite model is, similar t

arXiv:2607.08765v1 Announce Type: new Abstract: In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream t

arXiv:2606.16730v2 Announce Type: replace-cross Abstract: We re-interpret Transformer pretraining as a fast-slow, singularly perturbed flow along depth, with untied weights as its non-autonomous featu

arXiv:2601.22054v2 Announce Type: replace-cross Abstract: Scaling has powered recent advances in vision foundation models, yet extending this paradigm to metric depth estimation remains challenging du

🔥 We introduce LeVLJEPA: the first fully non-contrastive end-to-end vision-language pretraining method competitive with CLIP & SigLIP 💪🏼 👀 No negatives. No temperature. No momentum encoder. No teacher

arXiv:2606.30749v1 Announce Type: new Abstract: Large-scale dexterous grasp datasets encode rich priors over hand-object interaction, but their use has largely been confined to grasp generation and pi

arXiv:2606.29773v1 Announce Type: new Abstract: Graphs are widely used to model relational systems, with applications in domains such as social networks, finance, and biomedicine. Graph neural network

arXiv:2606.30634v1 Announce Type: new Abstract: Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bu

arXiv:2606.27711v1 Announce Type: cross Abstract: We introduce a neural network-based framework for learning time series estimators through a process we term decision-theoretic pretraining. Analysts s

arXiv:2606.26050v1 Announce Type: cross Abstract: Midway through an ordinary pretraining run, a small language model learns the pronoun-gender rule: cued with a girl's name ('Sue cried because'), it r

arXiv:2606.23113v1 Announce Type: new Abstract: This report presents our solution for the ICRA 2026 GOOSE 2D Fine-Grained Semantic Segmentation Challenge, which requires parsing unstructured outdoor s

arXiv:2606.11508v1 Announce Type: new Abstract: Accurate prediction of absorption, distribution, metabolism, and excretion (ADME) properties is critical to drug discovery, but remains challenging beca

arXiv:2502.11034v3 Announce Type: replace Abstract: Loss spikes remain a persistent obstacle in large-scale language model pretraining. While previous research has attempted to identify the root cause

arXiv:2601.21218v2 Announce Type: replace Abstract: Large language models (LLMs) are highly capable of answering questions, but they are often unaware of their own knowledge boundary, i.e., knowing wh

arXiv:2606.06373v1 Announce Type: cross Abstract: Wireless foundation models have emerged as a promising alternative to building separate models for each wireless task. However, existing approaches re

arXiv:2606.05186v1 Announce Type: cross Abstract: Budget-constrained micro-pretraining often requires triaging many candidate recipes on a shared accelerator before larger search budgets are spent. We

arXiv:2509.22468v2 Announce Type: replace-cross Abstract: High-quality molecular representations are essential for property prediction and molecular design, yet large labeled datasets remain scarce. W

arXiv:2605.31156v1 Announce Type: new Abstract: Causal discovery aims to recover directed causal relations from observational and interventional data, providing a basis for mechanistic understanding a

arXiv:2605.26194v1 Announce Type: new Abstract: Clinical time-series learning is routinely constrained by small, heterogeneous cohorts and protocol drift, while its downstream use spans both classific

It’ll be interesting to see if the post training for this uses a multiple of the compute of pretraining as cursor did when they tuned Kimi as the base model Grok foundation model V9-Medium (1.5T) has

arXiv:2506.16659v3 Announce Type: replace Abstract: Training large language models (LLMs) relies on adaptive optimizers such as Adam, which introduce extra operations and require significantly more me

arXiv:2605.22651v1 Announce Type: new Abstract: CLIP-style contrastive pretraining typically curates web-scale image-text pairs using sample-level filtering signals, often based on pair-level alignmen

arXiv:2605.21075v1 Announce Type: new Abstract: Earth observation (EO) foundation models (FMs) are increasingly trained on multisensor data, spanning multispectral imagery (MSI), synthetic aperture ra

arXiv:2605.19282v1 Announce Type: new Abstract: Muon is a matrix-aware optimizer that leverages Newton-Schulz (NS) iterations to enforce spectral gradient orthogonalization by driving all singular val

This article from Latent Space provides guidance on securing employment at cutting-edge AI research laboratories, with a focus on roles related to pretraining large language models. It likely covers p

arXiv:2603.07900v2 Announce Type: replace Abstract: Foundation models pretrained on electronic health records (EHR) have demonstrated zero-shot clinical prediction capabilities by generating synthetic

arXiv:2605.15599v1 Announce Type: cross Abstract: Extreme low-data fine-grained classification is common in expert domains where labeling is expensive, yet practitioners still need principled guidance

arXiv:2605.13769v1 Announce Type: cross Abstract: We study dense and mixture-of-experts (MoE) transformers in a tiny-scale pretraining regime under a shared LLaMA-style decoder training recipe. The sp

Anthropic: Anthropic researchers detail “model spec midtraining”, which adds a stage between pretraining and fine-tuning to improve generalization from alignment training — Sara Price2, Samuel Marks2,

arXiv:2605.02500v1 Announce Type: new Abstract: Sleep foundation models have recently demonstrated strong performance on in-domain polysomnography tasks, including sleep staging, apnea detection, and

arXiv:2312.12339v3 Announce Type: replace Abstract: Understanding visual inputs for a given task amidst varied changes is a key challenge posed by visual reinforcement learning agents. We propose exti

arXiv:2511.18903v2 Announce Type: replace-cross Abstract: Due to the scarcity of high-quality data, large language models (LLMs) are often trained on mixtures of data with varying quality levels, even

arXiv:2604.16952v1 Announce Type: new Abstract: Learning robust representations across extremely heterogeneous modalities remains a fundamental challenge in multi-modal vision. As a critical and profo

arXiv:2604.13977v1 Announce Type: new Abstract: Synthetic data is a standard component in training large language models, yet systematic comparisons across design dimensions, including rephrasing stra

arXiv:2610.12528v1 Announce Type: new Abstract: Continuous glucose monitoring (CGM) foundation models encode each day as one vector that mixes who the person is, what happened that day and how the sen

arXiv:2610.11159v1 Announce Type: new Abstract: Speech-to-LLM systems often connect a frozen speech encoder to a frozen large language model (LLM) through a small trainable bridge. The bridge is usual

arXiv:2610.09611v1 Announce Type: new Abstract: Large neural networks often acquire capabilities that small models fail to learn. Does this stem from large models learning more representative features

arXiv:2610.09700v1 Announce Type: new Abstract: Synthetic hard negatives generated in the representation space have proven effective for unimodal self-supervised learning, but transferring this idea t