Search: “pretraining”

arXiv:2605.02105v1 Announce Type: cross Abstract: Pretraining optimizers are tuned to produce the strongest possible base model, on the assumption that a stronger starting point yields a stronger mode

arXiv:2511.22958v2 Announce Type: replace Abstract: Deep learning has revolutionized solar image analysis, yet most approaches train task-specific encoders from scratch or rely on natural-image pretra

arXiv:2604.22989v1 Announce Type: cross Abstract: Recent medical multimodal foundation models are built as multimodal LLMs (MLLMs) by connecting a CLIP-pretrained vision encoder to an LLM using LLaVA-

arXiv:2604.20392v1 Announce Type: new Abstract: Vision Transformers (ViTs) dominate self-supervised learning (SSL). While they have proven highly effective for large-scale pretraining, they are comput

arXiv:2604.09782v1 Announce Type: new Abstract: Chagas disease screening via ECGs is limited by scarce and noisy labels in existing datasets. We propose a biomarker-based pretraining approach, where a

arXiv:2604.11066v1 Announce Type: new Abstract: We present KS-PRET-5M, the largest publicly available pretraining dataset for the Kashmiri language, comprising 5,090,244 (5.09M) words, 27,692,959 (27.

arXiv:2604.06829v2 Announce Type: cross Abstract: Synthetic data rephrasing has emerged as a powerful technique for enhancing knowledge acquisition during large language model (LLM) pretraining. Howev

arXiv:2610.11585v1 Announce Type: cross Abstract: Recent advances in large language model (LLM) pretraining highlight the role of high-quality training data in improving performance. While model-based

arXiv:2610.09709v1 Announce Type: new Abstract: Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require

arXiv:2610.05235v1 Announce Type: new Abstract: Surface electromyography (sEMG) is a low-power, cost-effective biosignal for hand-pose estimation and gesture classification. In this work, we examine w

arXiv:2609.40089v1 Announce Type: new Abstract: Continued pretraining enables language models to adapt to new domains and knowledge, but often at the cost of forgetting previously acquired capabilitie

arXiv:2609.36794v1 Announce Type: new Abstract: Signal features derived from photoplethysmography (PPG) require different signal durations to characterize. Existing PPG foundation models treat duratio

arXiv:2609.33405v1 Announce Type: new Abstract: Autoregressive pretraining increasingly draws on heterogeneous data, making it important to understand how a model learns from an individual token. The

arXiv:2609.34915v1 Announce Type: new Abstract: Muon is increasingly used for language-model pretraining, yet its large-step dynamics are not captured by the classical edge-of-stability (EoS) picture

arXiv:2609.29376v1 Announce Type: new Abstract: Automatic 12-lead electrocardiogram (ECG) classification requires representations that jointly capture local waveform morphology, long-range temporal dy

arXiv:2609.29156v1 Announce Type: cross Abstract: Long-tailed chest X-ray classification requires visual representations that capture both common abnormalities and subtle, infrequent findings. We prop

arXiv:2609.29403v1 Announce Type: new Abstract: Learning surrogates for time-dependent partial differential equations often requires a new simulation corpus when the governing operator changes. We int

arXiv:2609.27925v1 Announce Type: new Abstract: Language models can over-condition on irrelevant preceding text: predictions already supported by local context may still change when distant, unrelated

arXiv:2609.25582v1 Announce Type: new Abstract: Public surface electromyography (EMG) datasets vary widely in electrode layout, channel count, frequency support, and size. Simply mixing them for pretr

arXiv:2609.25482v1 Announce Type: new Abstract: Large language model (LLM) pretraining conventionally returns the raw final iterate. This couples two design choices: the learning-rate schedule that ge

arXiv:2609.22789v1 Announce Type: new Abstract: Image-to-layer decomposition converts a flattened image into editable RGBA layers, enabling element-level editing in design workflows. Existing diffusio

arXiv:2609.21362v1 Announce Type: new Abstract: Conventional tokenizers represent text as characters or statistically derived subwords, overlooking the internal phonological structure of syllables and

arXiv:2609.22064v1 Announce Type: new Abstract: Advances in large-scale neural recording have made it possible to collect data across many animals and distributed brain regions, raising the question o

arXiv:2609.21288v1 Announce Type: new Abstract: Surface electromyography (sEMG)-based silent speech interfaces are limited by cross-user variability and calibration burden. We study a limited-data set

arXiv:2609.19291v1 Announce Type: cross Abstract: Large Language Models (LLMs) have made remarkable progress in the processing and modeling of many languages. Yet, unlike human multilinguals, they exh

arXiv:2609.16551v1 Announce Type: new Abstract: Self-supervised learning (SSL) can reduce the need for labelled medical images, but the choice of pretext objective remains unclear for lung ultrasound

arXiv:2609.14234v1 Announce Type: cross Abstract: Financial fraud in corporate transaction networks has grown more coordinated and harder to detect with rule-based engines and with classical learning

arXiv:2609.15221v1 Announce Type: cross Abstract: Passive acoustic monitoring can measure biodiversity at larger scales, but time--frequency annotation of animal vocalizations is expensive, site-speci

arXiv:2609.13507v1 Announce Type: new Abstract: Brain-computer interfaces (BCIs) decode neural activity to restore lost function. Typically, training a high-performance neural decoder requires a large

arXiv:2609.12721v1 Announce Type: new Abstract: Applying an imitation learning policy to a new manipulation task usually requires collecting new demonstrations and retraining the model, which makes sa

arXiv:2609.10706v1 Announce Type: cross Abstract: Human video datasets have emerged as a compelling alternative to expensive real-robot data, offering rich diversity at scale. To bridge the human-to-r

arXiv:2601.02031v3 Announce Type: replace-cross Abstract: Pretraining of large language models is not only expensive but also prone to certain training instabilities. A specific instability that often

arXiv:2601.15236v2 Announce Type: replace Abstract: Large language models can memorize competing locale-specific facts yet fail to select among them when the locale changes, defaulting instead to a si

arXiv:2608.23922v1 Announce Type: new Abstract: Data mixing is a central design problem in large language model pretraining: given a fixed token budget, practitioners must decide how much data to allo

arXiv:2608.10473v1 Announce Type: cross Abstract: Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction

arXiv:2608.10595v1 Announce Type: cross Abstract: Proteolysis-targeting chimeras (PROTACs) induce protein degradation by recruiting a target protein to an E3 ubiquitin ligase, making degradation a joi

arXiv:2608.10590v1 Announce Type: new Abstract: Vision Transformers (ViTs) are widely believed to require more labeled data than CNNs for industrial dense prediction. Through controlled experiments on

arXiv:2608.06122v1 Announce Type: cross Abstract: Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether simi

arXiv:2608.03089v1 Announce Type: new Abstract: Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level

arXiv:2608.00231v1 Announce Type: new Abstract: Volumetric CT vision-language pretraining learns 3D representations from scan-report pairs, but global and anatomy-aware objectives supervise only corre

arXiv:2603.11838v2 Announce Type: replace Abstract: Large language models pretrained on internet-scale data risk lookahead bias in forecasting tasks, as they may have already seen the true outcome dur

arXiv:2607.08765v1 Announce Type: new Abstract: In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream t

arXiv:2606.16730v2 Announce Type: replace-cross Abstract: We re-interpret Transformer pretraining as a fast-slow, singularly perturbed flow along depth, with untied weights as its non-autonomous featu

arXiv:2601.22054v2 Announce Type: replace-cross Abstract: Scaling has powered recent advances in vision foundation models, yet extending this paradigm to metric depth estimation remains challenging du

arXiv:2606.30749v1 Announce Type: new Abstract: Large-scale dexterous grasp datasets encode rich priors over hand-object interaction, but their use has largely been confined to grasp generation and pi

arXiv:2606.29773v1 Announce Type: new Abstract: Graphs are widely used to model relational systems, with applications in domains such as social networks, finance, and biomedicine. Graph neural network