Search: “pretraining”

arXiv:2609.03177v1 Announce Type: new Abstract: Frontier large language models (LLMs) have become attractive priors for optimization due to their large-scale pretraining that enables them to navigate

arXiv:2609.03702v1 Announce Type: new Abstract: General-purpose code embeddings power tools for code search, classification, and retrieval. Compact transformer encoders for code typically rely on eith

arXiv:2609.02187v1 Announce Type: new Abstract: General-purpose vision-language models (VLMs) now support strong visual recognition, instruction following, and generation. However, most pretrained vis

arXiv:2609.00454v1 Announce Type: new Abstract: Pretrained transformer-based language models achieve strong performance across a wide range of NLP tasks but remain limited in encoding geo-locational s

arXiv:2603.16792v2 Announce Type: replace-cross Abstract: Pixel-space diffusion has recently re-emerged as a strong alternative to latent diffusion, enabling high-quality generation without pretrained

arXiv:2512.03400v2 Announce Type: replace-cross Abstract: We study how explicit world-modeling objectives affect the internal representations and downstream capability of Transformers, using Rubik's C

arXiv:2608.28878v1 Announce Type: cross Abstract: This paper develops a hybrid offline-online multi-agent reinforcement learning framework based on decision transformers. The policy is first pretraine

arXiv:2608.29426v1 Announce Type: cross Abstract: Reliable semantic representations derived from city-scale 3D models are increasingly important for urban analysis, infrastructure monitoring, autonomo

arXiv:2608.28308v1 Announce Type: new Abstract: We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling ex

arXiv:2608.28059v1 Announce Type: cross Abstract: In-context learning (ICL) allows a pretrained model to infer a new task from examples supplied in its prompt without updating its parameters. In linea

arXiv:2608.28549v1 Announce Type: new Abstract: Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveragi

arXiv:2608.26992v1 Announce Type: new Abstract: Representation learning has attracted great atten- tion and managed to reach good performances as a pretraining method for downstream tasks or as a firs

arXiv:2608.26354v1 Announce Type: cross Abstract: Simulation-based inference (SBI) for parameter estimation is vulnerable to model misspecification: neural summaries and density estimators trained on

arXiv:2608.26673v1 Announce Type: new Abstract: Large pretrained vision-language-action models dominate modern robot-manipulation benchmarks, but it remains unclear how much model scale is necessary f

arXiv:2608.25768v1 Announce Type: new Abstract: Turkish encoder models have adopted modern architectures while leaving the pretraining objective fixed at masked language modelling. This paper introduc

arXiv:2608.24042v1 Announce Type: cross Abstract: While Vision-Language-Action (VLA) models pretrained on large-scale robot datasets provide a strong foundation for robot manipulation, their performan

arXiv:2608.24727v1 Announce Type: cross Abstract: EEG foundation models pretrained via self-supervised learning promise transferable representations, but their generalization remains limited, especial

arXiv:2608.22059v1 Announce Type: cross Abstract: Pretrained image encoders are central to medical image classification, where expert annotation is costly and task-specific cohorts are often limited.

arXiv:2608.19766v1 Announce Type: new Abstract: Self-supervised pretraining on remote sensing imagery typically treats all samples as equally informative, despite large variability in geographic and v

arXiv:2608.20157v1 Announce Type: new Abstract: Egocentric action understanding is often addressed using large video models pretrained on extensive exocentric datasets. However, many first-person acti

arXiv:2608.19858v1 Announce Type: new Abstract: Generalizable dynamic graph anomaly detection (DGAD) enables pretrained detectors to identify anomalies in unseen target domains without costly retraini

arXiv:2608.19589v1 Announce Type: cross Abstract: Pretrained Vision-Language-Action models provide a strong foundation for robot learning, but sequentially adapting them to diverse skills can perturb

arXiv:2608.18094v1 Announce Type: cross Abstract: Large pretrained language models have demonstrated remarkable capabilities across diverse languages, yet critically underrepresented low-resource lang

arXiv:2608.17567v1 Announce Type: cross Abstract: Pretrained molecular language models are increasingly used as molecular encoders for learning structure-property relationships. However, their practic

arXiv:2603.06505v2 Announce Type: replace Abstract: Automatic speech recognition (ASR) has benefited from advances in pretrained speech and language models, yet most systems remain constrained to mono

DeepSeek released on a paper on pretrained engrams in January. I’m surprised ds didn’t release engrams with v4 pro. When will ds or another lab release a fixed engram model? Fixed engrams will be the

arXiv:2608.15972v1 Announce Type: cross Abstract: Synchronized camera and wireless measurements observe the same scene through different physical channels. The central difficulty is that a representat

arXiv:2601.21979v2 Announce Type: replace Abstract: Feature embeddings acquired from pretrained models are widely used in medical applications of deep learning to assess the characteristics of dataset

arXiv:2608.15647v1 Announce Type: cross Abstract: Semantic segmentation of very-high-resolution (VHR) remote sensing imagery increasingly benefits from strong pretrained hierarchical encoders, yet exp

arXiv:2608.15317v1 Announce Type: new Abstract: Scene coordinate regression (SCR) achieves strong performance in outdoor LiDAR localization, but it usually requires scene-specific training that can ta

arXiv:2608.14492v1 Announce Type: new Abstract: The Muon optimizer shows clear benefits versus alternatives when pretraining neural networks. However, it is used less frequently for parameter-efficien

arXiv:2608.11656v1 Announce Type: new Abstract: Recent advances in EEG foundation models have demonstrated the potential of large-scale pretraining to enable generalizable neural decoding across subje

arXiv:2608.11332v1 Announce Type: new Abstract: Sign-language research for resource-constrained languages is often limited by the cost of dense linguistic labels such as glosses, temporal boundaries,

arXiv:2608.08244v1 Announce Type: new Abstract: General wearable foundation models are pretrained across broad sensor streams and populations, but are not designed around women's-health tasks. We intr

arXiv:2608.09427v1 Announce Type: new Abstract: Pretrained self-supervised representations have emerged as a core component of current deepfake detection methods, yet it remains unclear which of their

arXiv:2608.09381v1 Announce Type: new Abstract: Robust robot control benefits from explicitly modeling state transitions, but video-generation world action models (WAMs) introduce substantial deployme

arXiv:2503.06211v3 Announce Type: replace-cross Abstract: Text-pretrained language models (LMs) encode rich world knowledge, but adapting them to process and generate perceptual modalities such as aud

arXiv:2608.09558v1 Announce Type: new Abstract: How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in

arXiv:2608.07340v1 Announce Type: cross Abstract: Registration-based Few-shot medical image segmentation (RFMIS) aims to generate pseudo-labels for unlabeled images by warping a labeled image through

arXiv:2608.06792v1 Announce Type: cross Abstract: Foundation model(FM) for recommendation has shown strong ability to model long-horizon sequential user behavior. In practice, a single pretrained foun

arXiv:2608.06612v1 Announce Type: cross Abstract: The plethora of readily available geospatial data offers exciting opportunities to learn high quality representations of the planet, but the sheer siz

arXiv:2608.07053v1 Announce Type: new Abstract: Pretrained partial differential equation (PDE) foundation models can generalize across different equations, but adapting them to unseen PDE systems typi

arXiv:2608.05691v1 Announce Type: new Abstract: Scientific images are essential for communicating experimental observations, quantitative evidence and conceptual knowledge. Unlike natural images, thei

arXiv:2606.14591v2 Announce Type: replace-cross Abstract: Recent advances in pretrained large audio-language models (LALMs) have demonstrated strong capabilities across speech, sound, and music. To ad

arXiv:2608.05088v1 Announce Type: new Abstract: Muon has recently emerged as a promising alternative to AdamW for language model pretraining by orthogonalizing momentum matrices using Newton-Schulz it

arXiv:2608.04234v1 Announce Type: cross Abstract: We study the problem of aligning data from multiple modalities into a shared representation space, focusing on settings where strong pretrained unimod