Search: “pretraining”

arXiv:2604.22117v1 Announce Type: cross Abstract: Aligned large language models(LLMs) remain vulnerable to adversarial manipulation, and their dependence on web-scale pretraining creates a subtle but

arXiv:2604.16570v1 Announce Type: new Abstract: DNA sequence encoding is fundamental to gene function prediction, protein synthesis, and diverse downstream biological tasks. Despite the substantial pr

arXiv:2610.13321v1 Announce Type: new Abstract: Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We i

arXiv:2512.23441v2 Announce Type: replace-cross Abstract: Temporally aware image representations are crucial for capturing disease progression in 3D volumes of longitudinal medical datasets. However,

arXiv:2603.02767v4 Announce Type: replace-cross Abstract: Image--text contrastive pretraining has become a dominant paradigm for visual representation learning, yet existing methods often yield repres

arXiv:2609.39827v1 Announce Type: cross Abstract: Pre-pretraining (PPT) on synthetic non-natural language data improves token efficiency during language model pre-training (PT). Prior work attributes

arXiv:2609.31680v1 Announce Type: new Abstract: Joint-embedding predictive architectures (JEPA) have emerged as a promising self-supervised pretraining paradigm for time series, learning representatio

arXiv:2609.33016v1 Announce Type: new Abstract: Large Language Model (LLM) pretraining performance is jointly shaped by three components of the training triplet: the optimizer, model architecture, and

arXiv:2609.34479v1 Announce Type: new Abstract: Vision-language (VL) pretraining using paired chest X-ray (CXR) images and radiology reports has shown strong potential for medical image understanding.

arXiv:2609.35440v1 Announce Type: new Abstract: Large language models are trained with backpropagation, whose global gradient coordinates all layers but forces each to hold its activations and wait fo

arXiv:2609.35219v1 Announce Type: new Abstract: Relational deep learning models database rows and foreign-key links as a heterogeneous graph for prediction from record attributes and relational contex

arXiv:2609.27611v1 Announce Type: new Abstract: Foundation models hold promise for generalizable analysis of structural brain magnetic resonance imaging (MRI) across development, aging and disease. Ho

arXiv:2609.26118v1 Announce Type: new Abstract: Latent action models (LAMs) learn action-related representations from action-free videos via self-supervised future prediction, offering a scalable para

arXiv:2609.21888v1 Announce Type: cross Abstract: Detecting pretraining data in large language models is challenging because high likelihood can reflect either training exposure or strong generalizati

arXiv:2609.21788v1 Announce Type: cross Abstract: A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional ful

arXiv:2509.03405v2 Announce Type: replace Abstract: Language models (LMs) increasingly drive real-world applications that require world knowledge. However, the internal processes through which models

arXiv:2609.16859v1 Announce Type: new Abstract: To train handwritten text recognition systems we need word images and their corresponding transcriptions, and these transcriptions are produced manually

arXiv:2609.13956v1 Announce Type: new Abstract: In this report, we release Tabby, a long context probabilistic time series foundation model, together with a complete and open recipe of how it was buil

arXiv:2609.12218v1 Announce Type: cross Abstract: The growing use of electroencephalography (EEG) motivates automated analysis that is accurate, transferable, and deployable on constrained hardware. R

arXiv:2609.05588v1 Announce Type: cross Abstract: World-action models (WAM) predict future states to guide robot actions, enabling learning from both action-free video and action-labeled interaction.

arXiv:2609.07398v1 Announce Type: new Abstract: World-Action Models inherit world knowledge from video-generative priors, and channel it into executable control signals through embodied experience. Ex

arXiv:2609.00059v1 Announce Type: cross Abstract: Materials property prediction remains difficult in low-data settings, where many target properties are supported by only a limited number of labeled s

arXiv:2609.00566v1 Announce Type: cross Abstract: We propose EEG-VID, a task-guided latent predictive pretraining framework for EEG decoding under session and subject shifts. EEG-VID predicts future l

arXiv:2608.26576v1 Announce Type: new Abstract: When researchers compare multilingual models for probing, interpretability, or cross-lingual transfer, they often align embedding spaces and assume that

arXiv:2608.26649v1 Announce Type: new Abstract: Objective: Model-based closed-loop neural stimulation holds promise for therapeutic applications ranging from Parkinson's disease to sensory restoration

arXiv:2608.14924v1 Announce Type: cross Abstract: Spatial transcriptomics (ST) links tissue morphology with molecular programs, motivating multimodal pretraining methods that align histology images wi

arXiv:2608.13681v1 Announce Type: cross Abstract: Translating C code into safe, idiomatic Rust is a longstanding software-engineering goal because it can eliminate entire classes of memory-safety vuln

arXiv:2608.13183v1 Announce Type: new Abstract: Visual foundation models are a cornerstone of image and video understanding but typically require large amounts of data and computation. The current sca

arXiv:2608.09424v1 Announce Type: new Abstract: Autoregressive language models align training and use: generation conditions on a clean prompt, and training predicts future tokens from clean left cont

arXiv:2607.22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly

arXiv:2607.22114v1 Announce Type: new Abstract: Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajec

arXiv:2607.07494v1 Announce Type: cross Abstract: Gradient communication is a primary scaling bottleneck in large language model (LLM) pretraining. Communicating gradients in low-precision formats, su

arXiv:2607.04541v1 Announce Type: cross Abstract: Camera-radar (CR) fusion is a practical sensing configuration for autonomous driving, but existing models are typically trained with task-specific sup

arXiv:2511.16757v2 Announce Type: replace-cross Abstract: Audio-language pretraining (ALP) holds promise for learning general-purpose audio representation, yet remains underexplored. Crucially, there

arXiv:2606.11387v1 Announce Type: cross Abstract: Short pretraining runs can reduce experimental cost, but they can also over-promote configurations that only look strong at tiny budgets. We study an

arXiv:2606.03938v1 Announce Type: cross Abstract: Multi-epoch training is becoming the standard now that compute is growing faster than the supply of high-quality text. But pretraining a single model

arXiv:2606.03508v1 Announce Type: new Abstract: Printed circuit board (PCB) defect detection is an essential part of automated optical inspection (AOI); yet it remains challenging in practice because

arXiv:2606.00955v1 Announce Type: new Abstract: Despite the growing availability of cryo-electron microscopy (cryo-EM) density maps, effectively leveraging them for protein representation remains chal

arXiv:2605.22981v1 Announce Type: cross Abstract: Fill-in-the-middle (FIM) is a pretraining objective widely used to equip causal language models with infilling ability, yet its effect on verbatim mem

arXiv:2605.21906v1 Announce Type: new Abstract: Computed tomography (CT) is a central to three-dimensional medical imaging, yet CT-based artificial intelligence remains fragmented across task-specific

arXiv:2605.21070v1 Announce Type: new Abstract: Amos et al. (2024) showed that the accuracy of Transformer models in sequence classification can be significantly improved by first pretraining with a m

arXiv:2605.16600v1 Announce Type: cross Abstract: Cross-entropy pretraining and preference alignment update the same transformer weights, but leave geometrically distinct traces. We characterise this

arXiv:2605.13059v1 Announce Type: new Abstract: Clinical diagnostic workups typically follow a modality escalation pathway: after initial clinical evaluation, clinicians begin with routine structural

arXiv:2605.13329v1 Announce Type: cross Abstract: How large language models internally represent high-level behaviors is a core interpretability question with direct relevance to AI safety: it determi

arXiv:2605.08809v1 Announce Type: cross Abstract: Pretraining large language models (LLMs) with next-token prediction has led to remarkable advances, yet the context-dependent nature of token embeddin

arXiv:2605.07938v1 Announce Type: new Abstract: Single-cell representation learning (SCRL) from gene expression data offers a way to uncover the complex regulatory logic underlying cellular function.