AI Wiki — Knowledge Base

Excited to share that I’ve joined OpenAI in London to work on pretraining! I’ve spent the last few years on pretraining and long-context, an…

Excited to share that I’ve joined OpenAI in London to work on pretraining! I’ve spent the last few years on pretraining and long-context, and I’ll be helping grow the London team. We’re hiring excepti

industrysam-altman--x

5 Jun 2026

Search: “pretraining”

Pretrain Where? Investigating How Pretraining Data Diversity Impacts Geospatial Foundation Model Performance

Can an LM, starting from random init (!!), learn to generate all of its pretraining data? Introducing Self-Play Pretraining with Zero Data. …

Excited to share that I’ve joined OpenAI in London to work on pretraining! I’ve spent the last few years on pretraining and long-context, an…

Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

LA4VLA: Learning to Act without Seeing via Language-Action Pretraining

ActiveMimic: Egocentric Video Pretraining with Active Perception

Understanding Reasoning from Pretraining to Post-Training

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP

Procedural Pretraining for Molecular Property Prediction

Synthetic Persona Pretraining: Alignment from Token Zero

MC-PDD: Masked Corpus-Level Pretraining Data Detection for Black-Box Large Language Models

SlideCheck: Guiding Self-Supervised Pretraining of Pathology Foundation Models via Dataset Distributions

Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study

Pretraining Induces a Reusable Spectral Basis for Downstream Task Adaptation

Characterizing Narrative Content in Web-scale LLM Pretraining Data

Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining

Procedural Pretraining: Warming Up Language Models with Abstract Data

Advancing Metallic Surface Defect Detection via Anomaly-Guided Pretraining on a Large Industrial Dataset

Does Continued Pretraining on a Learner Corpus Improve Automated Essay Scoring on English Proficiency Tests? Evidence from EFCAMDAT

Data Mixing for Large Language Models Pretraining: A Survey and Outlook

DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration

SimpleTouch: Can Vision-Language-Action Models Master Contact-Rich Manipulation Without Tactile Policy Pretraining?

BarcodeMAE+: Rethinking Masked Pretraining and Global Representations for DNA Barcode Foundation Models

Separators in Enhancing Autoregressive Pretraining for Vision Mamba

Pretraining and Distillation Matter More Than Architecture Family for Label-Free Single-Cell Classification

DART: Depth-as-Target Pretraining for Surgical Vision Foundation Models

Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking

Pretraining Reusable Inference Across Views with Synthetic Task Priors

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

The pretraining domain outweighs the training objective in setting the privacy-utility trade-off of differentially private medical image analysis

Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning

KletterMix: Climbing Toward High-Quality German Pretraining Data

Cool paper from Meta FAIR. It's on self-improving LLMs but on the pretraining side. (bookmark it) Most LLM safety, factuality, and reasoning…

Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining

Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

Tiny-Scale Chinese BERT Pretraining: A Controlled Comparison of MLM, WWM, and MacBERT Strategies

DIVINE: Simple Cross-Market Stock Pretraining via Diverse Indicator Reconstruction

Wrong Organ, Right Physics: Transferring Echocardiography Pretraining to Lung Ultrasound for Tuberculosis Screening

IronMind: Scaling Humanoid Dexterous Manipulation via Camera-Space Ego-Centric Pretraining

MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

Pretraining of Medical Visual Encoders Toward Multi-modal Large Language Models

Data Scout: Targeted Web Crawling for Domain-Specific Pretraining Corpora

Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA

A Composition-Aware Pretraining Framework for Geospatial Foundation Models

Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

ED-DiT: Physics-Guided Diffusion Pretraining for Transferable Molecular Representations from Electron Density