Can AI Scaling Continue at 4x Per Year Through 2030?

In recent years, the rapid advancement of AI models has been a testament to the power of computational scaling. Our research indicates that this growth in computational resources is responsible for a significant portion of AI performance improvements (Epoch AI). The consistent and predictable gains from scaling have driven AI labs to aggressively expand training compute at an astounding rate of about 4x per year.

To put this into perspective, this 4x annual growth in AI training compute outpaces some of the fastest technological expansions in recent history. For instance, it surpasses the peak growth rates of mobile phone adoption (2x/year from 1980-1987), solar energy capacity installation (1.5x/year from 2001-2010), and human genome sequencing (3.3x/year from 2008-2015).

Key Factors Constraining AI Scaling

Given this rapid pace, a critical question arises: Is it technically feasible for the current rate of AI training scaling-approximately 4x per year-to continue through 2030? We examine four key factors that could constrain this scaling:

• Power Availability: The energy required to power these massive computational tasks is substantial. However, projections from electricity providers suggest significant capacity growth in the coming years.
• Chip Manufacturing Capacity: Advanced chip packaging facilities are being planned and constructed, which will increase the supply of high-performance chips needed for AI training.
• Data Scarcity: While data is a critical input for training models, the vast amount of digital content available mitigates this constraint to some extent.
• Latency Wall: This refers to the fundamental speed limit imposed by unavoidable delays in AI training computations. Despite this, ongoing research aims to optimize and reduce these delays.

Analysis and Projections

Our analysis incorporates various public sources, including semiconductor foundries' planned expansions, electricity providers' capacity growth forecasts, and other relevant industry data. Here are some key findings:

• Power Availability: The construction of additional power plants and the geographic spread of data centers to leverage multiple power networks will help meet the growing demand for energy.
• Chip Manufacturing Capacity: Planned growth in advanced chip packaging facilities will increase the supply of high-performance chips, crucial for handling the computational demands of AI training.
• Data Scarcity: The exponential growth of digital content continues to provide ample data for training large models.
• Latency Wall: Research into optimizing algorithms and hardware is making strides in reducing unavoidable delays.

Feasibility by 2030

Based on our analysis, it is likely that training runs of 2e29 FLOP will be feasible by the end of this decade. To put this in context, if pursued, we might see by the end of the decade advances in AI as drastic as the difference between the rudimentary text generation of GPT-2 in 2019 and the sophisticated problem-solving abilities of GPT-4 in 2023.

Economic Considerations

While the technical feasibility is promising, the economics of such a massive investment are another story. Training models at this scale will require hundreds of billions of dollars over the coming years. Whether AI developers will be willing to make these investments depends on their long-term strategic goals and financial capabilities.

Conclusion

The rapid scaling of AI training compute has been a driving force behind recent advancements in AI performance. While there are significant technical challenges, our analysis suggests that it is possible for this 4x per year growth to continue through 2030. The key will be addressing power availability, chip manufacturing capacity, data scarcity, and the latency wall. If these constraints can be managed, we may witness unprecedented advancements in AI capabilities by the end of the decade.