What the evidence and regulators are saying

By diving into a landmark machine learning whitepaper titled "Will we run out of data? Limits of LLM scaling based on human-generated data," authored by Pablo Villalobos and a team of researchers from Epoch AI. Originally released in late 2022 and substantially updated to its second version (v2) in June 2024, this paper tackles one of the most critical structural questions facing the artificial intelligence industry today: what happens when our language models become so massive that they literally run out of original human text to read?

To figure this out, the researchers built probabilistic models to forecast the future demand for training data and compared it against the total global stock of public human-generated text data. Under current AI development paradigms, model performance scales predictably according to neural scaling laws. These laws state that to efficiently improve an AI's capabilities, developers must scale model parameter sizes and the volume of training datasets proportionally. The authors meticulously calculated the total effective stock of text available across the indexed web, placing the figure at approximately $4 \times 10^{14}$ tokens (where one token corresponds to roughly 0.8 English words). This massive pool corresponds to an estimated training compute capacity of around $5 \times 10^{28}$ FLOP for non-overtrained models.

By extrapolating historical dataset growth rates alongside compute-optimal scaling projections, the paper delivers a striking timeline. If current trends continue, AI developers will completely exhaust the available stockpile of public human-generated text data somewhere between the years 2026 and 2032. This represents an 80% confidence interval, with the central trend line pointing heavily toward the late 2020s. This is a notable adjustment from the authors' original 2022 paper, which famously predicted high-quality text data would be fully depleted much earlier. The updated, slightly extended timeline in v2 is due to a more nuanced understanding of multi-epoch training (running the same data through a model for multiple passes, or overtraining) and a more detailed breakdown of data filtering techniques across the web.

Because this "data wall" represents a hard barrier to brute-force scaling, the whitepaper explores several alternative pathways for continued progress in language modeling:

Synthetic Data Generation: Having existing high-performing AI models manufacture entirely new text to train future models.

Transfer Learning from Data-Rich Domains: Tapping into non-textual, highly abundant modalities such as video, images, or audio.

Data Efficiency Techniques: Refining tokenization schemes, utilizing non-public data repositories, and optimizing how models handle repeated tokens over multiple training epochs.

The deeper, unstated takeaway from this research is that the initial, wild "land-grab" era of generative AI—where performance was boosted simply by throwing larger and larger chunks of the internet into a cluster of graphics cards—is coming to a hard physical end. This paper alerts readers that human-generated data is a finite, scarce, and highly precious planetary resource. As we operate within the predicted 2026–2032 bottleneck window, the focus of the tech world is fundamentally shifting away from raw size and moving toward extreme architectural efficiency. The next generation of breakthroughs will not come from finding a magic, unread corner of the internet, but from building algorithms that can learn deeply, accurately, and contextually from a limited footprint of authentic human expression.

- Citizen

The swift advancement of artificial intelligence has been fundamentally supported by an abundant supply of data created by humans. Nevertheless, tech magnate Elon Musk claims that this resource has reached its limit. In a thought-provoking discussion early last year, Musk remarked that the "total accumulation of human knowledge has been depleted for AI training. " To clarify, the foundational systems behind the world’s leading chatbots gained their impressive skills by collecting and analyzing nearly all publicly available online information. Every digital book, discussion on public forums, Wikipedia entry, news article, and social media update has already been absorbed by the neural networks. Since tech companies have essentially utilized all the available high-quality human text, they now encounter significant challenges in training the next generation of even more intelligent models.

To overcome this data barrier, Musk and other prominent AI experts propose that the field should shift significantly towards "synthetic data. " This approach involves leveraging current AI systems to produce new text, essays, and scenarios, which are then reintegrated into newer models to support their learning and improvement. Although companies such as Meta, Microsoft, and OpenAI are actively testing this method, specialists caution that it poses considerable risks. Musk highlighted that AI's propensity to "hallucinate"—asserting incorrect information confidently as truth—renders learning from synthetic data exceedingly difficult. Furthermore, professionals like Andrew Duncan from the Alan Turing Institute caution about a risk called "model collapse. " If an AI relies too much on outputs generated by other AIs, it runs the risk of becoming a mere imitation; quality declines, creativity diminishes, and biases become intensified in a repetitive cycle.

Takeaway - The depletion of human data carries significant implications. For the past ten years, the tech sector has operated under "scaling laws," assuming that simply increasing computing power and data would ensure AI would improve linearly. But what occurs when there is a complete exhaustion of human history to input into the machine? This points to the possibility that we may be nearing a natural limit in AI abilities unless there is a significant innovation in how machines think, rather than merely recognizing patterns within vast datasets. Additionally, it reveals a developing cultural and legal crisis: as the internet becomes inundated with AI-generated outputs, the remaining original human data becomes increasingly precious, which explains why publishers and creative sectors are currently engaged in intense legal struggles to safeguard and profit from their copyrighted materials.

- Citizen