**Demystifying Training-Time Augmentation for Data-Constrained Language Model Pr

**Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining**

> As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate...

**来源信息**
- **来源**：HuggingFace Daily Papers（社区热门论文）
- **分类**：ai-models
- **发布时间**：2026-06-23 08:00（北京时间）
- **原文**：[打开原文](https://huggingface.co/papers/2606.16246)