A data bottleneck could slow the superintelligence race

And that could be a good thing

Credit: Getty/THEPALMER

AI is already helping develop future AI models. Right now, these systems work only as assistants, shepherded by human programmers. But if the trend continues, we could reach a fully automated pipeline where increasingly intelligent machines create even smarter successors — a recursive self-improvement cycle.

And that, according to many, is all it takes for an intelligence explosion where human-level AIs give rise to Dario Amodei’s “country of geniuses in a datacenter” in under a year. This is the underlying premise in both the much-discussed AI 2027 scenario and the earlier Situational Awareness essay that swept the tech world.

Some think it’s already starting. Anthropic co-founder Jack Clark recently wrote that he thinks “the takeoff towards fully automated AI R&D is happening.” And OpenAI’s Sam Altman wrote last year that we’d reached the “larval version of recursive self-improvement.”

Others think those narratives are papering over a host of challenges that will significantly slow those sub-one-year timelines.

Optimists — people optimistic that AI can progress quickly, perhaps scarily quickly — have a compelling vision for how such a “foom” scenario could play out. As we saw so vividly in narratives like AI 2027, AI has to first become good enough at coding and AI R&D to automate its own research. Then, within a few months, it can generate synthetic data and learning environments to train the next generation of AIs, developing a learning algorithm that learns as efficiently as humans. It’s then able to quickly pick up new areas using mostly synthetic data, needing only a small amount of human-generated data or real world deployment to verify that the learning algorithm works. Within months, it could reach superhuman capabilities. Less than a year after automating its own improvement, AI is capable of transforming the world. How it plays out after that is anyone’s guess, as we, in Clark’s words, “cross a Rubicon into a nearly-impossible-to-forecast future.”

---

When AI starts writing itself

---

In that narrative, the AI first automated its own software research, then generalized to most digital and physical work, then finally transformed everyday experiences. The optimists think that this first recursive self-improvement step will be relatively easy. It’s the stage where it’s easiest to generate synthetic data and training environments since code is verifiable and the experts are all right at hand. So far, synthetic data does seem to be fairly effectively targeting gaps in verifiable tasks, such as mathematics or coding. It’s easier to check whether such an answer is right than to come up with that answer. AI can create a large supply of math problems, keep only the ones that check out, and use those verified examples as high-quality training data for future models. Beyond the frontier models, we know the major companies routinely distill synthetic data from the biggest models to train smaller, cheaper models. Smaller AI companies can effectively multiply their compute severalfold by training their models on such outputs, as Anthropic alleges that DeepSeek, Moonshot, and MiniMax illicitly did.

There’s also been an explosion in the creation of virtual environments for reinforcement learning, essentially videogames for AI models. The AI can take a range of actions within these synthetic environments, learning based on rewards built into the system. Imagine a Pokémon videogame, where the AI can capture Pokémon, fight battles, and trade. It wins points depending on the outcomes it achieves. But unlike tasks in the real world, the AI can speedrun potentially billions of iterations on these games, trying every possible combination to find out what maximizes its score. This is the process that allowed the famous AlphaZero to master chess in only four hours of training.

AI will probably need new paradigms to crack sample efficiency, the AI’s ability to squeeze more intelligence out of a given amount of data. But optimists like Forethought senior research fellow Tom Davidson think this will probably happen quickly — after all, they argue, we know it’s possible as humans learn from a tiny fraction of the data that AI currently requires. A human teenager needs something like 20 hours of practice to learn to drive a car. That’s many orders of magnitude less than what Waymo needed to train its cars.

For stage two, where the AI intelligence generalizes to the wider world, Davidson remains optimistic that AI can mostly learn from digital environments. It’s one thing if the AI can master its own R&D. If the AI can also quickly learn how to practice law, play Pokémon, and predict drug discoveries, then it’s likely mastered an underlying learning algorithm. An efficient learning algorithm could easily transfer between skill domains, allowing the AI to rapidly generalize even though the specific skills don’t transfer. Obviously law and medicine require different skill sets, but moving from one to the other shouldn’t mean starting over from scratch. Humans don’t have to relearn how to talk, after all. And if it’s able to generalize to many digital domains, then it might not need much more human-generated data to work in the physical world — a small sample of human-generated tasks might be enough to verify a learning algorithm generalizes.

AI companies are clearly trying to overcome these hurdles. Over the past two years, their spending on external data has shot up to billions a year. That’s dwarfed by their tens of billions spent on compute, but the gap likely narrows when you consider the internal worker hours and compute devoted to generating data. Research found that the majority of R&D compute is spent on producing data and internal research, rather than on final training runs. Some guesses put data generation at around 20% of compute, which would imply billions spent internally by each company. Based on spending trends so far, if money can overcome the data bottleneck, it will. As Davidson told me, “The market economy will find a way.”

---

The fuse is lit on the intelligence explosion

---

However, this vision of a relatively smooth path to self-improving AI rests on several key assumptions that skeptics dispute. Can the AI develop a generalizable learning algorithm based on synthetic data? Will money really be enough to smash through hurdles in months? Skeptics like former GovAI researcher Tom Reed don’t think so. He told Transformer that the need for data will definitely slow an intelligence explosion because it’s required to master the process of researching new models in the first place.

AI R&D consists of a myriad of skills that each take real-world experience to learn — writing clean code, designing experiments, research “taste,” coordinating across dozens of parallel research agendas, optimizing data-generation pipelines, running compute clusters the size of small cities, producing the chips that run them, and so on. Even leaps in sample efficiency can’t fix this, because the data simply doesn’t exist yet. Too much is locked in individuals’ implicit knowledge or relies on studies that haven’t been run yet. How can you train the AI to run a company successfully for decades, when even humans can’t predict how to do it?

We could make the data, but even the most ambitious ideas for how to do this would take months or years, slowing down an intelligence explosion. Even if every employee at TSMC wore a camera recording their actions, narrated their job, and served as consultant when the AI had problems, Reed suspects that you’d need months of such data to begin; the first round wouldn’t work; it would still take time to learn how to use this new kind of data, and so forth. Those little frictions would build up, until the AI takeoff was dragged out for several years longer. Major new breakthroughs in how models learn or are built may take a particularly long time — Reed says that “coming up with new paradigms is probably precisely the kind of thing that even the super good automated researcher … is really not going to be as good at.”

---

Against the METR graph

---

That also matches what some working in the field say about current capabilities. Peter McIntyre, CEO of Trajectory Labs, a company building RL environments for several of the frontier AI companies, told me that RL environment creation is heavily bottlenecked by human expertise. McIntyre points to the work his company is doing developing RL environments to train models against prompt-injection attacks. Building the environment’s tasks requires red-teaming experts, but there are, McIntyre says, “only so many top red teamers in the world that you can hire.” That scarcity of expert talent and knowledge is also true for many other domains.

Proposed workarounds, like just recording experts going about their jobs, are likely to be too unstructured to work off the shelf with current paradigms. Despite the explosion of interest the field is seeing, there are limits to how fast the current methods can scale. While McIntyre says this could eventually change, for now he warns that “if you try to do it too quickly, you’ll just produce slop.”

And the problem only escalates when AI moves from digital self-improvement to interacting with the real world. Many remain skeptical that a learning algorithm derived entirely from digital environments will also work in the physical world. Intuitively, there’s a big difference between playing Baldur’s Gate 3 and actually swinging a sword. The optimists are betting that the AI will master a learning algorithm that generalizes, even though the underlying skills are different. Skeptics aren’t buying it. AI podcaster Dwarkesh Patel suspects that we will need “mind-stretching amounts of human expert trajectories in every single field and skill you want the model to be competent at.”

And skeptics say we can’t speed that up with synthetic data, because of an inherent catch-22: the current models don’t understand messy real-world environments enough to create data that captures them — and they won’t understand messy real-world environments well enough until they have that data. This is backed up by past examples of how AI has been deployed, such as Arvind Narayanan and Sayash Kapoor’s case study of self-driving cars: despite following a similar self-play developmental path to AlphaZero’s, it’s taken decades instead of hours because safety and practical considerations limit how quickly each iteration can learn. Surely, they say, we should expect a recursively self-improving AI to need the same slow learning pace for other real-world tasks.

One thing that both sides mostly agree on: once a fully generalizable artificial intelligence is operating, there will still be delays for world transformation, such as time needed to run serial experiments. Even Anthropic CEO Dario Amodei agrees on that point. The AI might need to wait 60 years to learn which medicines slow aging. But that doesn’t undermine the real change it will bring.

---

Can we ever trust AI to watch over itself?

---

The core disagreement here still revolves around how much real-world data an AI requires for learning. Is it possible for a model to develop an algorithm that allows it to learn general skills from limited, mostly synthetic data, and check its output against small amounts of real-world examples? Or are large volumes of real-world data an irreplaceable ingredient?

But taking a step back, what the two sides are arguing over is far more about when than how or if. Both broadly agree that new discoveries and paradigms are required to reach AGI. One side thinks those will come quickly once AI is doing the work, and that once it breaches the recursive self-improvement stage, world transformation could start within months. The other side thinks these step-changes will take longer: maybe five years, maybe 20, maybe longer.

It might seem like a relatively minor quibble in the grand scheme of things, but it is instructive that those who are most ‘optimistic’ about how quickly AI can overcome these hurdles to an intelligence explosion also tend to be the most worried about what that means for everyone living through it. People tend to worry less if they think we have a couple of decades to solve the problem.

Those few years could change the calculus for students planning life after university, policymakers debating how to respond to AI, or those whose jobs have been displaced.

They could also be crucial for handling even bigger challenges, and for many of those convinced an intelligence explosion could happen incredibly soon and quickly, bottlenecks slowing down recursive self-improvement would come as a relief.

A few of them have already breathed sighs of relief that AI progress hasn’t kept pace with the most ambitious predictions. AI 2027 author Daniel Kokotajlo told MIT Technology Review that extending his timelines by a few years “feels like I just got a better prognosis from my doctor.” (He has, however, since shortened them again by 11 months.) And Jeffrey Ladish, executive director at Palisade Research, simply said, “Thank God we have more time.”

Disclosure: The author’s partner works at Google DeepMind.