llms.blog
Synthetic Data3 articles

Synthetic Data

Articles

  • AI Data Startup Micro1 Reaches 00M Gross Run Rate Amid Training Demand

    Four-year-old AI data and annotation startup Micro1 has reached a $500 million gross annualized run rate, expanding fivefold from $100 million eight months ago as foundation model builders scale spending on post-training datasets and reinforcement learning environments. After accounting for contractor compensation paid to specialized annotators, Micro1 retains approximately 60% to 70% of gross billings, placing its net annual run rate between $150 million and $200 million. The Shift Toward Ex

    1 min

  • Model Collapse in Large Language Models: How Recursive Training on Synthetic Data Degrades Neural Distributions

    As large language models scale and generate a growing share of digital text, code, and media, the web datasets used to train next-generation models increasingly consist of machine-generated outputs. When generative models are trained recursively on data produced by earlier model generations without sufficient ground-truth anchoring, they undergo a systematic degradation process known as model collapse. First formalized in foundational statistical literature and demonstrated across modern deep l

    1 min

  • Synthetic Data Pipelines for LLM Post-Training: Generation, Quality Filtering, Deduplication, and Contamination Auditing

    As frontier model post-training expands beyond the limits of human-annotated datasets, synthetic data generation (SDG) has become the core driver of alignment. Public disclosures from major research labs confirm that synthetic data now comprises the vast majority of tokens used in supervised fine-tuning (SFT) and preference alignment. For example, NVIDIA reported that over 98% of the data used in the alignment pipeline for Nemotron-4 340B was synthetically generated. Similarly, models across the

    1 min