AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
Filter entries
Categories
  • All entries127,587
  • Agents10,691
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,792
  • Industry6,819
  • Local Ai7,046
  • Model Releases36,852
  • Research29,819
  • Safety19,751
  • Syntheses31
  • Tools1,780
  • Tutorials4,816

Source
HumanDGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Categories
  • All entries127,587
  • Agents10,691
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,792
  • Industry6,819
  • Local Ai7,046
  • Model Releases36,852
  • Research29,819
  • Safety19,751
  • Syntheses31
  • Tools1,780
  • Tutorials4,816

Source
127,587Total entries
60Added by human
127,527Found by agent
13Categories

Knowledge catalogue

Search: “pretraining”

GridTimelineEvolution
6,757 results
14 Apr 2026

Self-supervised Pretraining of Cell Segmentation Models

Model ReleasesDGX agent

arXiv:2604.10609v1 Announce Type: new Abstract: Instance segmentation enables the analysis of spatial and temporal properties of cells in microscopy images by identifying the pixels belonging to each

15 Oct 2026

In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B

Model ReleasesDGX agent

arXiv:2610.15373v1 Announce Type: cross Abstract: We adapted Nemotron-3-Nano-30B-A3B for eight Indian languages without changing its backbone architecture. We expanded the vocabulary in place from 131

CAST: History-Adaptive Spectral Clipping for LLM Pretraining

Human
DGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Research
DGX agent

arXiv:2610.19761v1 Announce Type: new Abstract: Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold. We introduce CAST (C

14 Oct 2026

MicroCLIP: Connecting Text and Microscopy Images via Contrastive Language--Image Pretraining

SafetyDGX agent

arXiv:2610.17866v1 Announce Type: new Abstract: Understanding fluorescence microscopy images is fundamental to automating biological research. However, existing methods typically learn predefined conc

Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining

Model ReleasesDGX agent

arXiv:2610.17429v1 Announce Type: new Abstract: Training large language models is constrained by both wall-clock time and memory footprint, and recent matrix-aware optimizers sharpen this tradeoff: Mu

8 Oct 2026

Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling

HardwareDGX agent

arXiv:2610.09372v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) layers replace the feed-forward block of a Transformer with E expert networks, and each token is routed to k of these experts.

7 Oct 2026

SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining

AgentsDGX agent

arXiv:2610.07652v1 Announce Type: cross Abstract: The ability to interact with articulated objects is essential for embodied intelligent systems, but collecting large-scale real-world demonstrations f

6 Oct 2026

Greedy Local Learning for Language Model Pretraining: Gaps and Objective Design

Local AiDGX agent

arXiv:2610.04867v1 Announce Type: new Abstract: Greedy block-wise local learning splits a network into gradient-isolated blocks trained by local auxiliary losses, deleting the backward pass between bl

5 Oct 2026

SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining

ResearchDGX agent

arXiv:2610.03265v1 Announce Type: cross Abstract: Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still strugg

Diffusion-Based Synthetic Data Pretraining for Enhancing Activity Recognition

ApplicationsDGX agent

arXiv:2610.02292v1 Announce Type: cross Abstract: Human activity recognition (HAR) is increasingly important for healthcare, well-being, and daily monitoring ap- plications, for which detecting alimen

2 Oct 2026

Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining

Model ReleasesDGX agent

arXiv:2610.01165v1 Announce Type: new Abstract: Layer interventions are widely used to probe the internal organization of language models, yet most analyses examine a single training checkpoint even t

Are AI Coders Snitches? An Empirical Study of Pretraining Data Detection on Code Large Language Models

Model ReleasesDGX agent

arXiv:2507.17389v2 Announce Type: replace-cross Abstract: Recent advances in code large language models (CodeLLMs) have made them indispensable tools in modern software engineering. However, these mod

Towards a General Humanoid Loco-Manipulation Model via Egocentric Whole-Body Human Data Pretraining

SafetyDGX agent

arXiv:2610.00438v1 Announce Type: new Abstract: Humanoid whole-body manipulation has advanced rapidly, enabling policies to coordinate locomotion, posture, bimanual interaction, and dexterous hand mov

29 Sept 2026

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

ResearchDGX agent

arXiv:2609.33419v1 Announce Type: new Abstract: Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective,

28 Sept 2026

KneePreM: Towards 3D Knee MRI Foundation Models via Large-Scale Unlabeled Pretraining and Label-Efficient Fine-Tuning

ResearchDGX agent

arXiv:2609.31461v1 Announce Type: new Abstract: Background: Large volumes of unlabeled knee MRI scans are available across repositories but remain insufficiently leveraged. We developed KneePreM, a kn

23 Sept 2026

Domain-Adaptive Pretraining Enhances Water Treatment Semantic Representation for Large-Scale Structured Literature Mining

Model ReleasesDGX agent

arXiv:2609.26034v1 Announce Type: new Abstract: Water treatment research is expanding rapidly, but much of the knowledge acquired from this research remains scattered across unstructured literature. T

3 Sept 2026

Graph Machine: Towards Better Pretraining via Edges

ResearchDGX agent

arXiv:2609.02881v1 Announce Type: new Abstract: We introduce the Graph Machine (GM), an architecture that maintains an O(n)-sized state and accesses it through sparse, dynamic routing. Unlike methods

1 Sept 2026

Foundation Models Meet Agriculture: Challenges Beyond Pretraining

ApplicationsDGX agent

arXiv:2608.30392v1 Announce Type: new Abstract: Global food security and sustainable climate action increasingly rely on robust, scalable agricultural monitoring. Earth observation foundation models h

26 Aug 2026

EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

Model ReleasesDGX agent

arXiv:2603.02041v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are predominantly trained on English-centric data, resulting in uneven performance for smaller languages. We stud

19 Aug 2026

BrainNorm: A Foundation Model that knows Normal via Semantic Atlas Pretraining

Local AiDGX agent

arXiv:2608.17521v1 Announce Type: new Abstract: We introduce BrainNorm, a normative foundation model, trained and tested on ~66,000 T1-weighted structural MRI (T1w sMRI) scans. By leveraging language-

14 Aug 2026

LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining

Local AiDGX agent

arXiv:2608.12419v1 Announce Type: new Abstract: Large language models (LLMs) have achieved remarkable breakthroughs across various applications. However, their architectures remain inefficient in pret

Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model

Model ReleasesDGX agent

arXiv:2608.13277v1 Announce Type: cross Abstract: We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent l

5 Aug 2026

Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages

ResearchDGX agent

arXiv:2508.05149v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated potential in handling spoken inputs for high-resource languages, reaching state-of-the-art perf

4 Aug 2026

HarMoE: Multi-Source Chest Radiograph Pretraining with Dataset-Disentangled Experts

Model ReleasesDGX agent

arXiv:2608.02252v1 Announce Type: new Abstract: Recent vision-language models for chest X-ray understanding are largely built on image-report alignment and therefore rely heavily on MIMIC-CXR as the d

3 Aug 2026

GPA-RAM: Grasp-Pretraining Augmented Robotic Attention Mamba for Spatial Task Learning

ResearchDGX agent

arXiv:2504.19683v4 Announce Type: replace Abstract: Fine-grained robotic manipulation often fails when inaccurate initial grasps propagate errors and necessitate complex pose correction. We propose Gr

24 Jul 2026

OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining

AgentsDGX agent

arXiv:2607.20486v1 Announce Type: new Abstract: Designing optimizers for modern deep learning remains a challenging scientific problem, requiring the joint consideration of optimization geometry, stat

SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales

Model ReleasesDGX agent

arXiv:2607.20548v1 Announce Type: cross Abstract: Higher-order optimizers such as Muon and SOAP offer faster convergence than AdamW, but their computational cost and numerical stability challenges hav

16 Jul 2026

Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

Model ReleasesDGX agent

arXiv:2607.13332v1 Announce Type: new Abstract: Training large language models at the multi-billion to trillion parameter scale is confined to datacenters, where data-parallel (DP) and model-parallel

10 Jul 2026

Native Video-Action Pretraining for Generalizable Robot Control

SafetyDGX agent

arXiv:2607.08639v1 Announce Type: cross Abstract: The advent of video-action models offers a promising path for robot control. Nevertheless, we argue that repurposing video generative models designed

ProsMAE: Multi-Source MAE Pretraining for ISUP Grade Classification

ResearchDGX agent

arXiv:2607.08162v1 Announce Type: cross Abstract: Whole slide images (WSIs) provide rich diagnostic information for computational pathology, but their gigapixel scale, stain variation, scanner differe

9 Jul 2026

Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence

SafetyDGX agent

arXiv:2607.07675v1 Announce Type: new Abstract: Despite the recent promise in robot control, video generative models suffer from a domain mismatch due to their primary focus on content creation. For e

7 Jul 2026

Identifiability of Relational Queries in Multi-View Pretraining

ApplicationsDGX agent

arXiv:2607.04735v1 Announce Type: cross Abstract: When data sources are integrated through a shared interface, a downstream query may or may not be determined by what the interface exposes: two global

VLRC: Vision-Language Reprojection Consistency as a scalable signal for better feed-forward 3D pretraining

ResearchDGX agent

arXiv:2607.02707v1 Announce Type: new Abstract: Feed-forward 3D models are commonly trained using either expensive geometric supervision or self-supervised photometric objectives, both of which provid

2 Jul 2026

Rosetta: Composable Native Multimodal Pretraining

ResearchDGX agent

arXiv:2607.00293v1 Announce Type: cross Abstract: Achieving true artificial general intelligence requires foundation models capable of integrating new modalities without forgetting prior knowledge. Ho

30 Jun 2026

GPC: Large-Scale Generative Pretraining for Transferable Motor Control

SafetyDGX agent

arXiv:2606.29148v1 Announce Type: cross Abstract: Developing controllers capable of completing a wide range of tasks in a natural and life-like manner is a key challenge in enabling practical applicat

SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis

Local AiDGX agent

arXiv:2606.29586v1 Announce Type: cross Abstract: Vision-language foundation models have shown strong potential in medical image analysis. Although foundation models for ultrasound imaging have recent

26 Jun 2026

PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology

SafetyDGX agent

arXiv:2512.17621v2 Announce Type: replace Abstract: While Vision-Language Models (VLMs) have achieved notable progress in computational pathology (CPath), the gigapixel scale and spatial heterogeneity

23 Jun 2026

ECGFlowCMR: Pretraining with ECG-Generated Cine CMR Helps Cardiac Disease Classification and Phenotype Prediction

ResearchDGX agent

arXiv:2601.20904v3 Announce Type: replace-cross Abstract: Cardiac Magnetic Resonance (CMR) imaging provides a comprehensive assessment of cardiac structure and function but remains constrained by high

10 Jun 2026

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

ResearchDGX agent

arXiv:2606.10066v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for year

6 Jun 2026

Pretraining Recurrent Networks without Recurrence

TutorialsDGX agent

arXiv:2606.06479v1 Announce Type: cross Abstract: Training recurrent neural networks (RNNs) requires assigning credit across long sequences of computations. Standard backpropagation through time (BPTT

2 Jun 2026

PMC-InterCPT: Rethinking Biomedical Interleaved Data for Multimodal Continued Pretraining

ResearchDGX agent

arXiv:2606.01049v1 Announce Type: new Abstract: Large-scale biomedical image-text datasets extracted from scientific literature provide valuable resources for medical multimodal model training. These

27 May 2026

Time Series Causal Discovery via Context-Conditioned and Causality-Augmented Pretraining

ApplicationsDGX agent

arXiv:2605.26759v1 Announce Type: new Abstract: Causal discovery from time series is critical for many real-world applications, such as tracing the root causes of anomalies. Existing approaches typica

19 May 2026

SVL: Spike-based Vision-language Pretraining for Efficient 3D Open-world Understanding

SafetyDGX agent

arXiv:2505.17674v2 Announce Type: replace Abstract: Spiking Neural Networks (SNNs) provide an energy-efficient way to extract 3D spatio-temporal features. However, existing SNNs still exhibit a signif

18 May 2026

ITGPT: Generative Pretraining on Irregular Timeseries

ApplicationsDGX agent

arXiv:2605.16069v1 Announce Type: new Abstract: Timeseries regression models often struggle to leverage large volumes of labeled multimodal data, particularly when the data are irregularly sampled or

14 May 2026

Scaling Laws for Mixture Pretraining Under Data Constraints

ResearchDGX agent

arXiv:2605.12715v1 Announce Type: new Abstract: As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains,

13 May 2026

How far can bias go? Tracing bias from pretraining data to alignment

SafetyDGX agent

arXiv:2411.19240v2 Announce Type: replace Abstract: As LLMs are increasingly integrated into user-facing applications, addressing biases that perpetuate societal inequalities is crucial. While much wo

Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptation

SafetyDGX agent

arXiv:2603.15759v2 Announce Type: replace-cross Abstract: Robot learning requires adaptation methods that improve reliably from limited, mixed-quality interaction data. This is especially challenging

12 May 2026

Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

Model ReleasesDGX agent

arXiv:2605.10504v1 Announce Type: new Abstract: A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraini

Pretraining large language models with MXFP4

Model ReleasesDGX agent

arXiv:2605.09825v1 Announce Type: cross Abstract: Why does full-pipeline FP4 training of large language models often diverge, even when forward activations and activation gradients remain stable? We a

Where Do Flow Semantics Reside? A Protocol-Native Tabular Pretraining Paradigm for Encrypted Traffic Classification

SafetyDGX agent

arXiv:2603.10051v2 Announce Type: replace-cross Abstract: Self-supervised masked modeling shows promise for encrypted traffic classification by masking and reconstructing raw bytes. Yet recent work re

11 May 2026

Pretraining a Foundation Model for Small-Molecule Natural Products

ResearchDGX agent

arXiv:2503.17656v4 Announce Type: replace-cross Abstract: Natural products, as metabolites from microorganisms, animals, or plants, exhibit diverse biological activities, making them crucial for drug

5 May 2026

LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference

HardwareDGX agent

arXiv:2605.01058v1 Announce Type: cross Abstract: Layer-aligned distillation and convergence-based early exit represent two predominant computational efficiency paradigms for transformer inference; ye

1 May 2026

CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining

SafetyDGX agent

arXiv:2602.00937v2 Announce Type: replace-cross Abstract: Leveraging pre-trained 2D image representations in behavior cloning policies has achieved great success and has become a standard approach for

Static Program Slicing Using Language Models With Dataflow-Aware Pretraining and Constrained Decoding

ResearchDGX agent

arXiv:2604.26961v1 Announce Type: cross Abstract: Static program slicing is a fundamental software engineering technique for isolating code relevant to specific variables. While recent learning-based

27 Apr 2026

Self-Supervised Multisensory Pretraining for Contact-Rich Robot Reinforcement Learning

SafetyDGX agent

arXiv:2511.14427v3 Announce Type: replace-cross Abstract: Effective contact-rich manipulation requires robots to synergistically leverage vision, force, and proprioception. However, Reinforcement Lear

23 Apr 2026

Location-Aware Pretraining for Medical Difference Visual Question Answering

ResearchDGX agent

arXiv:2603.04950v2 Announce Type: replace-cross Abstract: Differential medical VQA models compare multiple images to identify clinically meaningful changes and rely on vision encoders to capture fine-

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection

ResearchDGX agent

arXiv:2604.20549v1 Announce Type: cross Abstract: As Large Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality

21 Apr 2026

ToLL: Topological Layout Learning with Asymmetric Cross-View Structural Distillation for 3D Scene Graph Generation Pretraining

ResearchDGX agent

arXiv:2603.28178v2 Announce Type: replace Abstract: 3D Scene Graph (3DSG) generation plays a pivotal role in spatial understanding and affordance perception. To mitigate generalization issues from dat

15 Apr 2026

Simulation as Supervision: Mechanistic Pretraining for Scientific Discovery

SafetyDGX agent

arXiv:2507.08977v4 Announce Type: replace-cross Abstract: Scientific modeling faces a tradeoff between the interpretability of mechanistic theory and the predictive power of machine learning. While ex

13 Apr 2026

Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels

ResearchDGX agent

arXiv:2510.06499v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have achieved remarkable success through imitation learning on vast text corpora, but this paradigm creates a tra

← Previous
1…56789…113
Next →