Ilya Sutskever is betting on very cheap superintelligence

Reading between the lines of "Safe Superintelligence Inc".

It was inevitable that Ilya Sutskever's new project would grab attention. A beloved pioneer of the field, everyone wondered what would follow after he left OpenAI amid a sea of controversy. Today, we learned the answer: Safe Superintelligence Inc., a new company co-founded with Daniel Gross and Daniel Levy.

While the company's goal of building a “safe superintelligence” is interesting, the real intrigue lies in Sutskever's business plan — or lack thereof. Sutskever told Bloomberg that SSI's “first product will be the safe superintelligence, and it will not do anything else up until then”. That, Sutskever argues, will insulate the company from "the outside pressures of having to deal with a large and complicated product and having to be stuck in a competitive rat race" — a dynamic that seems to have caused tensions at OpenAI.

But no product means no revenue. And even by traditional startup standards, Sutskever's goal here is expensive. Sam Altman has said that GPT-4 cost about $100 million to train, and that model is clearly a long way from superintelligence. We're likely to see $1 billion models this year, and $10 billion ones next, according to Dario Amodei. Microsoft, meanwhile, is reportedly considering building a $100 billion cluster.

While it's hard to find consensus on how large a cluster would need to be to be capable of training an AGI-level model under our current AI paradigms, "$100 billion or more" is a pretty standard answer (many AI experts seem to agree). And more could mean a lot more; potentially on the order of $1 trillion. What's more, these costs are just for compute: they don't even consider the (very expensive) salaries that the engineers capable of building these systems command.

With Sutskever refusing to produce any products (and therefore revenue) until he achieves superintelligence, those costs will have to be entirely borne by investors. And raising $100 billion without a product roadmap is, even in this environment, not feasible. So what's the plan?

The obvious answer is that Sutskever thinks he can do this for much, much less than $100 billion. He wouldn't be the only one. Earlier this month, Sutskever's former colleague Leopold Aschenbrenner published a notably bullish essay series arguing that there's a good chance we'll develop AGI by 2027, at which point frontier clusters would cost $10-100 billion.

I think Sutskever agrees, and believes AGI can be built for around $10 billion. ASI would follow soon after, possibly for not much more money (as the AGIs could automate AI research). Given that it's highly unlikely that simply scaling up current LLMs is enough to produce AGI on a $10 billion cluster, however, such a strategy relies on significant algorithmic improvements. But Sutskever, who's one of the best in the world at developing such improvements, might think he's figured out the secret.

This isn't the only possible explanation for what's going on. Sutskever could expect superintelligence to cost $100+ billion, but be hoping for government investment (notably, SSI will have offices in both the US and Israel). Or he could have wooed a tech giant like Apple or Nvidia to throw in the cash.

But given his worries about safety, the chance of Sutskever ceding control to a powerful and unpredictable institution seems less likely to me. Instead, it seems more probable that he and his investors think they can make huge algorithmic improvements, and bring the cost of training frontier models way, way down. (I've reached out to Aschenbrenner to see if his new firm, which counts Daniel Gross as an investor, has taken a stake in SSI; I've yet to receive a response.)

If this is what's going on, it's a big bet on unproven research. But that bet is one investors might be willing to take, especially given the gigantic returns on offer if it pays off. If Sutskever's right, however, it means superintelligence could be here much sooner than almost anyone thought.