How to think about DeepSeek
Also in Transformer Weekly: Meta’s RSP, a Paris preview, and Amodei on China.
Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
I’m in Paris this week — seemingly along with everyone else in AI. I’m speaking at AI & Society House on Sunday, and am in town till Tuesday night — if you’re around and want to say hi, drop me a line! shakeel@transformernews.ai.
Top stories
Well didn’t I pick a great week to go on holiday! While I was away, much too much nonsense was written about DeepSeek. Instead of contributing to the noise myself, I thought I’d just suggest a few pieces to read that are better than anything I could do myself.
Matt Sheehan and Scott Singer:
“Two policy objectives are clear. The United States must do everything it can to stay ahead of China in frontier AI capabilities. And it must also prepare for a world in which both countries possess extraordinarily powerful—and potentially dangerous—AI systems. In essence, it needs to dominate the frontier of AI and simultaneously defend against the risks … One key step toward preparing for that contingency is laying the groundwork for limited, carefully scoped, and security-conscious exchanges with Chinese counterparts on how to ensure that humans maintain control over advanced AI systems.”
“Last month, DeepSeek joined sixteen other Chinese companies in signing onto the Artificial Intelligence Safety Commitments (人工智能安全承诺). While branded as a domestic Chinese initiative, the commitments bear strong similarity to ongoing global industry-led efforts to put safeguards in place for frontier AI piloted at last year’s AI Summit in Seoul, known as the Seoul Commitments.”
Lennart Heim and Sihao Huang:
“Export controls on hardware operate with a time lag and haven't had time to bite yet.”
“The same efficiency gains that allow smaller actors like DeepSeek to access a given capability (‘access effect’) will probably also allow other companies to build more powerful systems on larger compute clusters (‘performance effect’)”.
“DeepSeek fits well on a price-capabilities curve that is not an outlier based on what other offerings there are from US companies. That is, DeepSeek has gone for a “notably cheap but not super cheap and notably good but not super good” and DeepSeek is not especially capable for its price and I’m very confident OpenAI et. al. could make an offering at that point very quickly if they wanted to.”
“DeepSeek didn’t manage to get around the export controls and instead were meaningfully held back by them, as designed.”
“Our analysis shows that the total server CapEx for DeepSeek is ~$1.6B, with a considerable cost of $944M associated with operating such clusters.”
“The United States must come to terms with the fact that it will not have a monopoly on superhuman AI capabilities.”
“The main thing I would say to [DeepSeek] is ‘take seriously these concerns about AI system autonomy’. When we ran evaluations on the DeepSeek models — we have a spate of national security evaluations for whether models are able to generate information about bioweapons that can’t be found on Google or can’t be easily found in textbooks — the DeepSeek model did the worst of basically any model we’d ever tested in that it had absolutely no blocks whatsoever against generating this information.”
Speaking of Amodei: that quote comes from an interview with ChinaTalk, which is worth paying attention to for its discussion of US-China AI competition. I don’t agree with a lot of what he said, but it’s a good insight into where his head is at.
“We have to make sure that we’re ahead of China and other authoritarian countries both because I don’t think they would use powerful AI very well and because if we’re not ahead, there’s this racing dynamic — yet somehow we have to also protect against the dangers of AI systems we ourselves build. My guess is the best way to do this, and it may be futile, it may not work out, but my guess is that the best way to do this is with something like export controls. We can create a margin between us and China.”
“The nice thing about the other companies in the West is that there’s a coordination mechanism between them. They can all be brought under the same law … I don’t think that’s possible between the US and China. We’re kind of in a Hobbesian international anarchy. I do think there are opportunities to try to cooperate with China. I’m relatively skeptical of them, but I think we should try … But so far, I’m aware of efforts by the US Government to send a delegation to talk to China about topics related to AI safety. My understanding, again, I obviously wasn’t part of those delegations, is that there wasn’t that much interest from the Chinese side. I hope that changes. I hope there is more interest.”
“We should be realistic about the actual situation. We should have a realist view of how international relations work, which is that it’s possible to make progress at the margin, and if some really compelling evidence were available, then maybe it could change things. But look, you have two nation-states that have very different systems of government that have been adversarial for a long time. They’re going to compete over this technology and they’re going to race to build it as fast as they can. That’s what’s going to happen by default.”
Meanwhile, OpenAI released Deep Research, a product powered by o3 and enhanced with browsing and data analysis capabilities.
OpenAI didn’t initially release any information on whether they safety-tested the model.
After I asked, they told me that they did test the underlying model, as well as whether browsing capabilities increased the risk.
But they’re not planning on releasing a full model card until the product is generally available — apparently a $200/month subscription is enough of a barrier to warrant skipping normal safety protocols.
As for the model’s capabilities: Tyler Cowen says he thinks “it’s comparable to having a good PhD-level research assistant, and sending that person away with a task for a week or two, or maybe more”.
I’m not sure I’d go that far: but it’s certainly very impressive, on some tasks more than others.
Meta finally released its version of a responsible scaling policy, which it’s calling a Frontier AI Framework.
It includes sections on addressing CBRN and cyber risks, and explicitly says something we’ve all been waiting to hear from Meta:
“If the results of our evaluations indicate that a frontier AI has a ‘high’ risk threshold by providing significant uplift towards realization of a catastrophic outcome we will not release the frontier AI externally.”
Less good: the framework only considers CBRN and cyber risks — model autonomy risks aren’t considered.
Also not good: the framework sets thresholds in response to whether a model “would uniquely enable execution” of threat scenarios, and that for a model to exceed a threshold it must possess “all of [the] enabling capabilities” for a threat scenario.
As Claudia Wilson of the Center for AI Policy notes, this is all very weak.
Google DeepMind, meanwhile, released an updated version of its Frontier Safety Framework. Some notable parts:
The new framework has an expanded section on deceptive alignment risk.
It explicitly says that “our adoption of the protocols described in this Framework may depend on whether [other AI developers] across the field adopt similar protocols.” Which is about as much of a cry for regulation as you could get from a company.
Separately: Google has removed commitments not to use AI for weapons or surveillance from its ethical guidelines.
The discourse
At the International Association for Safe and Ethical AI, Google DeepMind’s Anca Dragan said she wants governments to tell her what to do:
“I’m not speaking for Google here, but from a safety perspective, I would very much welcome standardisation of these frontier safety frameworks … I don’t want to come up with what are the evals and what are the thresholds. I want society to tell me. It shouldn’t be on me to decide.”
The Pope continues to have excellent AI takes:
“Some AI researchers have expressed concerns that such technology poses an ‘existential risk’ by having the potential to act in ways that could threaten the survival of entire regions or even of humanity itself. This danger demands serious attention.”
And Donald Trump has thoughts on artificial superintelligence:
“There are always risks. And it’s the first question I ask, how do you absolve yourself from mistake, because it could be the rabbit that gets away, we’re not going to let that happen.”
When asked if OpenAI would start releasing model weights, Sam Altman had a concerning response:
“Yes, we are discussing. I personally think we have been on the wrong side of history here and need to figure out a different open source strategy; not everyone at OpenAI shares this view, and it's also not our current highest priority.”
A new paper on “gradual disempowerment” laid out a potential catastrophic scenario that doesn’t get enough attention:
“Even an incremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial risk of eventual human disempowerment … no one has a concrete plausible plan for stopping gradual human disempowerment and methods of aligning individual AI systems with their designers' intentions are not sufficient.”
Bill Gates went on Jimmy Kimmel to freak everyone out:
JK: “Will we still need humans?”
BG: “Not for most things.”
Policy
Who’s coming to the Paris AI Action Summit on Monday? It’s a pretty middling list, according to Politico.
Aside from co-chairs Emmanuel Macron and Narendra Modi, JD Vance is the biggest name. Olaf Scholz and Ursula von der Leyen will be there too. Keir Starmer can’t be bothered to come, sending Peter Kyle instead.
US AISI staff were originally set to come, Reuters reports, but the Trump administration has put an end to that.
On the industry side, Sam Altman and Sundar Pichai are coming. No one knows if Elon Musk is.
The aim of the summit, according to Reuters, is a “non-binding communiqué of principles for the stewardship of AI, bearing US, Chinese and other signatures”.
Howard Lutnick said the government should focus on developing AI standards at NIST.
Rep. Jay Obernolte pledged to reintroduce legislation codifying AISI.
US officials are reportedly investigating whether DeepSeek acquired advanced Nvidia chips through Singapore intermediaries.
Multiple former OpenAI employees have been blocked from joining US AISI because of conflict-of-interest policies, according to The Information.
The UK government announced new laws to combat AI-generated CSAM, which will “make it illegal to possess, create or distribute AI tools designed to generate child sexual abuse material”.
France and the UAE announced a partnership to build a 1GW AI data center in France.
Ireland’s new AI minister, Niamh Smyth, admitted she has never used ChatGPT.
Influence
Mark Zuckerberg was at the White House on Thursday.
Meta’s Joel Kaplan said the EU AI Code of Practice is “unworkable and infeasible”.
Alexandr Wang has reportedly been schmoozing officials in DC.
Sam Hammond and Walter Copan urged the House Science Committee to support the EPIC Act, which would set up a foundation to provide private sector funding for NIST.
A cross-party group of MPs and Lords called on the UK government to introduce binding regulation on the most powerful AI systems to address existential risks.
New polling shows that 87% of Brits support laws requiring AI developers to prove their systems are safe before release, while 60% favour banning "smarter-than-human" AI models altogether.
SoftBank hired Miller Strategies as a lobbyist.
A group of academics and other prominent people published an open letter putting forward principles “guiding research into machine consciousness”.
Industry
A federal judge called Elon Musk’s claims of harm against OpenAI a “stretch”, but allowed the case to potentially go to trial.
Alphabet announced plans to spend $75b on AI infrastructure this year. Amazon gave capex spending guidance of $100b.
Amazon’s reportedly releasing its new AI model later this month. It’s using “automated reasoning” to reduce hallucinations.
Google launched Gemini 2.0 Pro for developers.
SoftBank committed $3b annually to use OpenAI’s tech and formed a joint venture to market it in Japan. To mark the deal, Masayoshi Son said a bunch of nonsense about AIs not wanting to eat humans.
Anthropic’s developed a new “constitutional classifiers” system to prevent AI models being jailbroken. There’s a prize if you can jailbreak it.
Cerebras partnered with Mistral to produce what the companies said were the “world’s fastest AI assistant”.
AMD’s stock fell after forecasting slower AI chip growth than rival Nvidia.
Figure AI ended its partnership with OpenAI.
ByteDance researchers demoed OmniHuman-1, an AI model that can generate highly realistic deepfake videos from a single image and audio.
TrueFoundry, which makes an enterprise AI deployment platform, raised $19m.
Google X spun out Heritable Agriculture, an AI agriculture startup.
Workday laid off 1,750 employees amid a pivot to AI.
Moves
Elizabeth Kelly is out as the head of US AISI, as expected.
John Schulman left Anthropic (just a few months after he left OpenAI to join them). He’s reportedly joining Mira Murati’s new startup — as are Christian Gibson and Mario Saltarelli, according to Fortune.
Marco Tagliasacchi and Zalán Borsos, who worked on NotebookLM’s viral podcast tool at Google DeepMind, are joining Microsoft’s AI team. So is Matthias Minderer, also from GDM.
Meta is reportedly reshuffling its generative AI group, with engineering leaders Ryan Cairns and Ning Li leaving the group.
The Future of Life Institute hired Jason Van Beek, a former senior adviser to Sen. John Thune, as its chief government affairs officer.
Dan Cheever, who was Sen. Todd Young’s lead AI policy staffer, is joining H&M Strategies.
Stuart Styron joined SeedAI as a senior fellow. He was previously senior technology policy counsel to Rep. Anna Eshoo.
In perhaps the most shameless attempt to cosy up to Republicans yet, Andreessen Horowitz hired Daniel Penny — the former Marine who killed a homeless man in New York — as an investment partner.
Though a16z faces stiff competition from Meta, who have just hired Henry Rodgers — formerly of The Daily Caller, a “news” outlet with a weirdly high number of links to white supremacists — to work on public policy.
Google DeepMind is hiring “a Research Scientist/Engineer for automated AI research”.
Microsoft is setting up an Advanced Planning Unit to study AI’s societal impacts.
Tarbell is still recruiting fellows for our 2025 AI reporting cohort. Fellows get $50k, training from world-class AI and journalism experts, and a placement at newsrooms including The Guardian, Bloomberg, The Verge … or even Transformer. Apply here by February 28th.
Best of the rest
More on DeepSeek:
It’s shockingly easy to jailbreak R1: security researchers found a “100 percent attack success rate”.
R1 raises an underdiscussed AI safety worry: in training, the model was rewarded purely based on getting correct answers, without any consideration given to whether its chains of thought were comprehensible to humans. AI safety researchers are urging companies not to give up legibility like this.
Security researchers think the company might have links to the state-owned China Mobile.
Open Philanthropy opened two requests for proposals: $40m for technical AI safety work, and an undisclosed amount for improving AI capability evaluations.
UK AISI released recommendations on how to do good safeguard evaluations.
UK AISI and Redwood AI released a “sketch of an AI control safety case”.
MIT released an AI Agent Index.
Margaret Mitchell and others published a paper arguing against the development of fully autonomous AI agents.
OpenAI is trying to push Sora in Hollywood, but is reportedly facing resistance from studios over data usage and union concerns.
Google said that state-sponsored hacking groups are using its Gemini AI to enhance cyberattacks, albeit mostly for productivity gains rather than generating novel attacks.
Stanford and University of Washington researchers said they created a reasoning model for under $50 in compute costs. It’s distilled from Gemini 2.0 Flash Thinking Experimental.
People are using AI to design computer chips — but we can’t understand how the new chip designs work.
OpenAI is reportedly going to have a Super Bowl commercial.
Thanks for reading; have a great weekend.