Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
Top stories
In his final week in office, Joe Biden made three big AI policy moves.
On Monday, we got new export controls, which we covered last week. The final rule was not significantly different from the leaked draft.
Tech companies are out in full force trying to get Trump to kill the rule; this excellent ChinaTalk episode explains what the rule actually does and notes how many of Big Tech’s arguments are a bit absurd.
On Tuesday, Biden followed up with an AI infrastructure executive order.
The main things here are letting companies build “gigawatt-scale” AI data centres on federal sites, building out clean energy generation to power those, and improving the AI infrastructure permitting regime.
And on Wednesday, we got a cybersecurity executive order that touches on AI in a bunch of places.
The EO “launches a public/private partnership to use AI for cyber defense of critical infrastructure in the energy sector” and “directs research and development of AI-based cybersecurity tools and techniques”.
Not to be outdone, the UK launched its big AI infrastructure package this week, with Matt Clifford’s AI Action Plan.
It’s mostly focused on building domestic compute capacity, with the government planning to create “AI growth zones” where building data centres and energy infrastructure will be easier.
It also calls for a new “UK Sovereign AI” body, focused on making sure the UK has “companies at the frontier that will be our UK national champions”.
And there’s a bunch of stuff focused on ensuring the government actually uses AI, as well as sensible recommendations on immigration.
Perhaps most importantly, Clifford has been appointed as the PM’s AI advisor.
And if that wasn’t enough infrastructure stuff for you, OpenAI got in on the action too with a new Economic Blueprint.
It’s all stuff you’d expect: we’re in an AI race with China, the US needs to win, it needs to build lots of AI infrastructure to do that, etc. etc. etc..
But it also makes some notable regulatory proposals:
“Develop alternatives to the growing patchwork of state and international regulations that risk hindering American competitiveness, such as by having the federal government leading the development of national security evaluations at home, and establishing a US-led international coalition that works toward shared safety standards abroad”
“[Create] a defined, voluntary pathway for companies that develop large language models to work with government to define model evaluations, test models, and exchange information to support the companies safeguards … in return, these companies would receive preemption from state-by-state regulations on the types of risks that the same national security agencies would handle”
“Voluntary” is of course the key word there. This reads an awful lot to me like “we don’t want actual regulation anymore, actually”.
Last but not least: Mark Zuckerberg revealed this week that he is woefully uninformed about AI risks, and that he doesn’t have good answers on what to do about them.
Here’s a section from his latest appearance on the Joe Rogan Experience:
JR: “You know that ChatGPT tried to copy itself when it found out it was being shut down, tried to rewrite its code?”
MZ: “I’m not sure what this is. What is this?”
JR: “You weren't aware of that? … It was shocking, when it was under the impression that it was going to become obsolete, they were gonna have a new version of it, and it would be shut down. It tried copying its code, and it tried rewriting its code, like, unprompted.”
MZ: “I mean, it depends on what goal you give it. I mean, there are all these weird examples of this … I think you need to be careful with these things, like, what guardrails you give it. If you're telling it, ‘at all costs’, then-”
JR: “But this is what we're people are terrified of. Like, that a foreign superpower like China is gonna say, achieve objectives at all costs.”
MZ: “Yeah.”
The conversation ends there, taking a turn to talk about o1’s compute requirements.
So to recap: Zuckerberg isn’t aware of high-profile cases of AIs showing scheming capabilities, and his solution to the problem is simply “guardrails” (which, as he well knows, do absolutely nothing on Meta’s models because they can easily be removed). Good stuff.
The discourse
Paul Triolo pushed back on the “America must beat China” discourse:
“[There] is a troubling combination that conflates the China threat, personal gain, and push back against regulation of advanced AI. It also portrays US China competition around AI as zero sum, which is particularly dangerous … The escalating AI competition between the US and China poses significant threats not only to both nations but also to the entire world.”
Holden Karnofsky laid out just how surprising recent AI progress has been — and what that means for policy:
“When we ask what cognitive tasks humans can do that AI systems can’t, it is harder today to give a clear answer, much less a stable one … The difficulty of laying out stable, robust limitations of AI creates a difficulty in policymaking: for almost anything a human can do, there’s a real chance that an AI will be able to do it, and soon.”
Notable AI commentator Gwern has an interesting theory about the recent bout of exuberance from OpenAI employees:
“Much of the point of a model like o1 is not to deploy it, but to generate training data for the next model … This means that the scaling paradigm here may wind up looking a lot like the current train-time paradigm … There may be a sense that they've 'broken out', and have finally crossed the last threshold of criticality, from merely cutting-edge AI work which everyone else will replicate in a few years, to takeoff — cracked intelligence to the point of being recursively self-improving and where o4 or o5 will be able to automate AI R&D and finish off the rest”.
Also interesting, though unverified: “I am actually mildly surprised OA has bothered to deploy o1-pro at all, instead of keeping it private and investing the compute into more bootstrapping of o3 training etc. (This is apparently what happened with Anthropic and Claude-3.6-opus - it didn't 'fail', they just chose to keep it private and distill it down into a small cheap but strangely smart Claude-3.6-sonnet.)”
OpenAI safety researcher Stephen McAleer asked the kind of question you wish OpenAI safety researchers had an answer to:
“How are we supposed to control a scheming superintelligence?”
Policy
NIST released revised guidance on mitigating misuse risk for dual-use models. You can compare the changes from v1 here. Along with new appendices on “chemical and biological misuse risk” and “cybersecurity misuse risk”, the most notable change is probably a tweak to the section on “example safeguards”:
Previously, the guidance suggested developers “ensure the level of access to the model’s weights is appropriate” and that they “consider when it is appropriate to make the model’s weights widely available”.
Now, the same section instead suggests that developers “release the model in stages”, and “consider deploying a model via an API to understand its impacts before making its weights available”.
Also notable: whereas previously the guidance suggested that if a model is found to have significant misuse risk, it might be “appropriate” to “not develop the model at all”, the new version softens that to “delaying the development of the model”.
CISA published a new playbook for helping the government and AI companies defend against attempts to hack AI models.
The second draft of the EU GPAI Code of Practice is out; Miles Brundage’s commentary is particularly good.
China released a plan to allow public data for AI labelling.
India will co-chair the Paris AI Summit next month.
TIME has an excellent profile of the UK AI Safety Institute.
One nugget of news: “The institute is now working on putting together a set of ‘capability thresholds’ that would be indicative of severe risks, which could serve as triggers for more strenuous government regulations to kick in.” (I’ve talked about the need for such thresholds here.)
In an FT op-ed, Keir Starmer outlined his vision for UK AI regulation:
“We don’t need to walk down a US or an EU path on AI regulation — we can go our own way, taking a distinctively British approach that will test AI long before we regulate, so that everything we do will be proportionate and grounded in the science”
Influence
Sam Altman, Mark Zuckerberg, Jeff Bezos, Tim Cook and Elon Musk will all reportedly be at Trump’s inauguration on Monday. Zuck’s hosting a black-tie reception afterwards.
Marc Andreessen is reportedly heavily involved in recruitment for the Trump administration.
Satya Nadella and Brad Smith met with Trump and Elon Musk at Mar-a-Lago to discuss AI data centres.
OpenAI is hosting a DC event on Jan. 30, the NYT reported, where Sam Altman will “demonstrate new OpenAI technology [to lawmakers and officials] that he believes will show the economic power of AI”.
TechNet launched a litigation center to “advance the interests of American innovation” through lawsuits.
Industry
Nvidia’s new GB200 racks are reportedly overheating, leading to delays for Microsoft, AWS, Google and Meta. Microsoft’s ordered more H200s in the meantime.
New emails seem to confirm that Meta trained its models on pirated content, with one employee writing that the company needs to avoid “media coverage suggesting we have used a dataset we know to be pirated”. Oops!
Amazon plans to relaunch Alexa as an “AI agent”, but it’s not ready yet.
Apple joined the UALink Consortium, an industry group developing AI accelerator architecture to compete with Nvidia's NVLink.
Mistral released a new coding model.
OpenAI launched “Tasks”, which lets users schedule recurring actions and reminders within ChatGPT.
Google announced it would automatically include AI features in Workspace apps and charge basic tier customers an additional $2 per employee monthly.
Anthropic achieved ISO 42001 certification for responsible AI development.
Replit says its AI coding agent has driven a 5x revenue increase in the last six months.
Aligned Data Centers raised over $12b in equity and debt to expand AI infrastructure across North America and Latin America.
AI chip startup Blaize went public via a $1.2b SPAC deal.
Palantir and Lockheed Martin are reportedly in talks to invest in drone startup Shield AI at a $5b valuation.
Legal AI startup Harvey is reportedly raising $300m at a $3b valuation, led by Sequoia.
Synthesia raised $180m at a $2.1b valuation.
Cursor raised $105m from a16z, Thrive and Benchmark (among others). The company says it has $100m in recurring revenue.
Bioptimus, a French biological AI startup, raised $41m.
Moves
OpenAI appointed Adebayo Ogunlesi, a BlackRock executive and CEO of Global Infrastructure Partners, to its board.
François Chollet and Mike Knoop launched Ndea, a new AI lab focused on developing AGI using program synthesis.
Mira Murati’s new AI startup has reportedly hired Jonathan Lachman, previously head of special projects at OpenAI, along with about 10 other researchers from OpenAI, Character AI and DeepMind.
Aidan McLau is joining OpenAI to work on model design for AGI.
Morgan Gress joined Andreessen Horowitz as a communications partner in their DC office.
Rachel Wolbers joined Google to lead engagement with trade associations.
Microsoft created a new group called Core AI, led by former Meta exec Jay Parikh, to build an “AI-first app stack” for Microsoft and its customers.
The Google AI Studio and Gemini Developer API teams moved under Google DeepMind.
OpenAI is hiring lots of robotics engineers.
Best of the rest
The San Francisco Police Department reopened the case of Suchir Balaji, a former OpenAI researcher who spoke out about copyright concerns, from suicide to an active investigation.
Balaji’s mother went on Tucker Carlson’s show this week, where she repeated her allegations that Balaji was murdered.
Wired has a big profile of Sheikh Tahnoun bin Zayed al Nahyan, the UAE’s intelligence chief who runs many of the country’s AI initiatives (including G42).
Lots of AI-media deals this week: Mistral and AFP, Google and the AP, and OpenAI and Axios.
Meanwhile, Microsoft and OpenAI urged a federal judge to dismiss copyright infringement claims by news organisations.
Digital content creators are reportedly selling unused video footage to OpenAI and Google for AI model training, earning up to $4 per minute of footage.
Workers are suing Scale AI over alleged worker misclassification and wage theft.
Researchers at the Oxford Martin School published a paper on “who should develop which AI evaluations?”.
Timothy Lee and Dean Ball have an excellent new podcast, with episodes so far interviewing Jon Askonas, Nathan Lambert, and Ajeya Cotra.
Epoch AI also has a podcast now.
Nathan Lambert has a good explainer on DeepSeek v3, noting that “it cost $6m to train” significantly understates the actual costs.
Andy Masley explained that individual use of ChatGPT is really not that bad for the environment (whether you use a washing machine matters an awful lot more, for comparison).
Apple suspended its glitchy AI-generated news summary feature.
Lots of discussion this week about this NYT piece, which profiles a woman who has fallen in love with ChatGPT.
Thanks for reading; have a great weekend.