The AI Security era has begun
Transformer Weekly: AISI rebrands, Elon meddles, and new models are imminent
Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
Top stories
The UK government announced today that the AI Safety Institute is now the AI Security Institute.
From the press release:
“This new name will reflect its focus on serious AI risks with security implications, such as how the technology can be used to develop chemical and biological weapons, how it can be used to carry out cyber-attacks, and enable crimes such as fraud and child sexual abuse.”
“The focus of the Institute will be clearer than ever. It will not focus on bias or freedom of speech, but on advancing our understanding of the most serious risks posed by the technology to build up a scientific basis of evidence which will help policymakers to keep the country safe as AI develops.”
AISI chair Ian Hogarth:
“It's not just bad actors that might use AI to threaten our security, we continue to take the possibility of risks, like maintaining oversight and control of AGI-like systems seriously.”
Politico notes some other changes:
“On its website the institute has now dropped talk of ‘societal impacts’ as a reason for evaluating models, changing it to ‘societal resilience.’ References to the risk of AI creating ‘unequal outcomes’ and ‘harming individual welfare’ have also gone. The institute has also dropped ‘public accountability’ as a reason for evaluating models, changing it to keeping the ‘public safe and secure.’”
Some people have cast this as the UK shifting focus away from tackling the catastrophic and existential risks of AI. I think it’s the opposite: it’s a refocus on catastrophic and existential risks.
More broadly, it reflects the shifting tides of AI policy. When AISI was founded in 2023, there was overwhelming pressure to include discussion of AI bias and ethics in the “AI safety” tent. Remember Kamala Harris’s much-mocked speech at Bletchley?
A year and a bit later, people are realising that this might not have been such a good idea.
This is, obviously, mostly because of the US election: you should primarily read the AISI rebranding as the UK attempting to suck up to Trump.
But this predates Trump. Over the past year, it’s become clear that the conjoining of AI ethics and safety has made it harder to actually make progress on tackling safety issues.
If you actually want to build a broad consensus about how to tackle catastrophic safety issues, you’re best off just tackling those issues, rather than trying to build a one-size-fits-all policy that does everything — leading to Republicans torpedoing the whole package just because they don’t like one part of it.
AI ethics and bias issues are very important. But the sad reality is that they’re politically intractable in the near future. (AI’s impact on the workforce is one exception.) And rather than throw the safety baby out with the bathwater, the UK has sensibly decided to just rebrand that work as “security”, with the hope of making it a bit less politically polarized.
I expect others will soon follow suit. Expect to hear much less about “AI safety” going forward — and a lot more about “AI security”.
In other UK news: Peter Kyle suggested at the weekend that the UK will compel frontier labs to share their models with AISI, despite previous reports to the contrary.
From The Times: “Kyle said he was confident that Trump would not oppose Labour’s intention to introduce laws that would require the top AI labs to submit their software for testing before release. He confirmed that the current voluntary agreements would be made mandatory but the thresholds for reporting would stay the same.”
Meanwhile, we got a flurry of news about impending model releases:
Sam Altman said that GPT-4.5, coming in “weeks”, will be OpenAI’s “last non-chain-of-thought model”.
In the coming “months”, it will be followed by GPT-5, “a system that integrates a lot of our technology, including o3”. o3 will no longer be released as a standalone model.
Anthropic is reportedly gearing up to release a “hybrid AI model that includes reasoning capabilities” in the “coming weeks”, according to The Information.
It will let users “control how much computational resources it spends on a query—in other words, how long it will ‘reason’ about that problem”.
And Elon Musk said Grok 3 is coming “in a week or two”, claiming that it outperforms “anything that’s been released” to date.
Speaking of Elon: Musk and a consortium of investors offered $97.4b for the assets owned by the OpenAI non-profit.
As many have pointed out, you should not read this as Elon trying to buy OpenAI. Instead, he’s trying to force OpenAI to value the non-profit’s assets “fairly” — something we talked about in Transformer way back in October.
While I’m sure you saw Sam Altman immediately reject the offer, you might have missed this: the board reportedly still hasn’t formally reviewed the offer, so Altman seemingly rejected it unilaterally. Which is not how boards work!
Also notable: OpenAI pointed out that Musk’s offer undermines his argument that the non-profit shouldn’t be allowed to sell its assets at all. Musk replied saying he’ll drop the offer if OpenAI stops the for-profit conversion.
Also also notable: Musk’s consortium of investors includes Ari Emanuel, one of the most powerful men in Hollywood, who said Altman is “phony and trying to get away with cheating the charity”.
And if you want to dive deeper, this interview with non-profit law expert Rose Chan Loui is worth your time.
The discourse
On Transformer: The Paris AI Summit was an embarrassing failure:
“What was supposed to be a crucial forum for international cooperation has ended as a cautionary tale about how easily serious governance efforts can be derailed by national self-interest … We can no longer afford this pantomime of progress.”
The Financial Times’ editorial board is getting worried about America’s AI policy:
“No one would wish to see [AI] development strangled by excessive regulation, or an authoritarian China become dominant. But America’s apparent readiness to dismantle guardrails that were being put in place represents a bold, potentially reckless, bet — that it can master this game-changing technology first, without anything going wrong along the way.”
Eric Schmidt is worried about AI misuse scenarios:
“The real fears that I have are not the ones that most people talk about AI — I talk about extreme risk … [AI systems could be used to create] a bad biological attack from some evil person.”
He also said Sam Altman was “being a little too quick” in saying that OpenAI was on “the wrong side of history” in not releasing model weights.
Sam Altman’s worried about the economics of AGI:
“It does seem like the balance of power between capital and labor could easily get messed up, and this may require early intervention.”
Though OpenAI is barely acknowledging this at its public events, instead telling people that AI won’t lead to unemployment.
JD Vance took to Twitter to share some AI policy thoughts:
“When it comes to AI specifically, the risks are 1) overstated or 2) difficult to avoid.”
Anton Leicht thinks AI-related unemployment might cause some issues for the American right:
“The [labor market disagreement] has the potential to split the tech right and MAGA coalition: They incidentally align on AI right now because its effects are still hidden in fog of war; but once AI labor market effects manifest, disagreement might break out.”
Tim Hwang thinks AI safety needs to change course:
“The most interesting intellectual work now is in finding a new path for AI governance that assumes fierce nationalism. Despite a reflex to the contrary, we have strong models for stability in these environments. Figures like Metternich [are] becoming extremely interesting.”
Chinese social media users had interesting reactions to Dario Amodei’s recent comments, per ChinaTalk:
One user: “Of course, AI safety is important — no one would deny that. But we must also be wary of those using ‘AI safety’ as a shield to enforce technological hegemony!”
Another: “Based on these beliefs, Amodei arrives at [the conclusion that] … containing China is more critical than AI safety itself”
Policy
The US AI Safety Institute selected Scale AI as its first authorized third-party evaluator for AI models.
You should read this as very bullish for the future of US AISI, I think: I doubt Scale would be signing this deal if they thought Trump (whose AI policy is in part being written by former Scale exec Michael Kratsios) was about to dismantle the institution.
Rep. Jay Obernolte called for an AI select committee to be established. He had some interesting comments on the open-weights debate, too:
“There are grains of truth on both sides [of the debate] … we don't think that there's enough risk here for us as regulators to put our thumb on the scale and say open-source models are better or closed source models are better.”
Re: what would constitute a risk worth tackling: “The parade of horribles, right? … Can a large language model be used to design and build a biological weapon, for example? Can it be used in cyber theft or cyber fraud? Or can it be used to hack in a way that endangers our national security? Those are really … the highest-level existential risk that you want to guard against. And those are the ones that people cite most frequently when they're talking about protections that are baked into a model being removed.”
Sen. Todd Young proposed a 10-point tech “playbook” for Trump's second term. He says the new admin “should encourage carefully scoped efforts to pair American AI with foreign capital to bring trusted AI solutions to the developing world”.
The European Commission withdrew its AI liability directive. It denied doing so due to pressure from the Trump administration.
At the hijacked AI Summit in Paris, France announced €109b in AI investments for France, and plans to dedicate a gigawatt of nuclear power for an AI data center.
The EU, meanwhile, pledged €200b for AI investment.
Saudi Arabia's Neom project signed a $5b deal to build an AI data center.
Saudi also agreed to buy $1.5b worth of AI chips from Groq.
Influence
Scale AI CEO Alexandr Wang met Keir Starmer.
Meta’s running a big “open-source AI is good” ad campaign, taking over multiple DC newsletters.
In Fox News, Tanner Jones and Joe Lonsdale take aim at the Future of Privacy Forum for the various state-level AI bills it seems to have a hand in.
Dean Ball has a less sensationalist and more informative piece on the same topic here.
RAND published a paper arguing that “the potential emergence of [AGI] is plausible and should be taken seriously by the US national security community”, and outlining “five hard problems” that AGI presents.
Industry
A bunch of companies met their Seoul commitments by publishing AI safety frameworks before the Paris Summit: you can see a full list here. Mistral and Zhipu AI (along with some others) have still not published theirs.
SoftBank is reportedly planning to invest $40b in OpenAI at a $260b valuation.
OpenAI is reportedly finalizing its first custom AI chip design, with the goal of mass production at TSMC by next year. It will reportedly be used primarily for inference, at first.
Meta is reportedly in talks to acquire AI chip startup FuriosaAI.
Arm reportedly plans to launch its own server CPU chip this year, with Meta as an early customer.
Anthropic is reportedly projecting revenue of $3.7b this year, with costs down from $5.6b last year to an expected $3b this year.
It expects 2027 revenue of $12b in its base case forecast, and $35b in its optimistic forecast. That does not seem particularly consistent with “AGI by 2027” to me!
Apple is reportedly partnering with both Alibaba and Baidu to bring Apple Intelligence features to China.
Baidu said it will make its upcoming Ernie 4.5 models open-source from June 30.
OpenAI removed its diversity commitment webpage, replacing it with a page on "building dynamic teams”.
OpenAI released an expanded Model Spec. It’s taking feedback on it.
NXP Semiconductors acquired edge AI chip developer Kinara for $307m.
Sardine, which is building AI agents to detect fraud, raised $70m.
Latent Labs, founded by a DeepMind alum, launched with $50m to develop protein design AI models.
Moves
Landon Heid was nominated to be assistant secretary of commerce for export administration at BIS.
Mike Harney, formerly chief of staff to Gina Raimondo and Sen. Mark Warner, is IBM’s new chief lobbyist.
Nokia appointed Justin Hotard, Intel’s data center and AI head, as its new CEO, saying it expects data centers to be its biggest growth engine in the coming years.
Devendra Chaplot is leaving Mistral AI.
Cristiano Lima-Strong left the Washington Post to join Tech Policy Press as associate editor.
Best of the rest
On Transformer: Decentralized training is becoming more widely adopted. But it isn't a policy nightmare — yet.
Anthropic's jailbreaking challenge was broken after 5 days.
Google DeepMind released a course on AGI safety.
Epoch AI said that AI tools currently use much less energy than previously estimated: about 0.3 watt-hours per ChatGPT query, less than the 3Wh figure usually reported.
RAND, meanwhile, said that AI power usage may grow to be much more significant than most people realize, projecting that global AI data center power demand could reach 68GW by 2027 (almost as much as California’s total power capacity) and 327GW by 2030.
The parents of Suchir Balaji sued San Francisco for denying information requests about his death.
Anthropic launched an "Economic Index" tracking real-world AI use, finding that users employ AI more as a collaborator (57%) than for automation (43%) — for now.
Condé Nast, McClatchy, Politico, Vox and lots of other publishers sued Cohere for copyright infringement.
Thomson Reuters won the first major AI copyright case in the US, against Ross Intelligence.
The Guardian announced a partnership with OpenAI.
Dan Hendrycks released yet another eval.
The UK released new details on how it’s using AI to make benefit decisions.
A deepfake video of celebrities criticising Kanye West for antisemitism went viral. Scarlett Johansson said “we must call out the misuse of AI, no matter its messaging”.
UK universities are seeing applicants use deepfakes in online interviews.
Thanks for reading; have a great weekend.