Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
Top stories
OpenAI released its new o1 models, which it touts as having advanced reasoning capabilities.
The new models are particularly good at coding, science, and maths, placing “among the top 500 students in the US in a qualifier for the USA Math Olympiad” and exceeding “human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems”.
OpenAI published some details on how they work (emphasis mine):
“Our large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process. We have found that the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute). The constraints on scaling this approach differ substantially from those of LLM pretraining, and we are continuing to investigate them.”
OpenAI’s Noam Brown puts it another way: “The longer it thinks, the better it does on reasoning tasks. This opens up a new dimension for scaling. We’re no longer bottlenecked by pretraining. We can now scale inference compute too.”
As OpenAI’s Miles Brundage notes, “test-time compute as a new scaling paradigm has implications for governance/safety”.
On Transformer, I summarised the new models’ system card, which found the models “instrumentally faked alignment”, and rated CBRN risk as “medium” for the first time, finding that the models “can help experts with the operational planning of reproducing a known biological threat”.
The model also exhibited reward hacking in a way OpenAI says “reflects key elements of instrumental convergence and power seeking”.
One thing worth flagging: METR, an evals org, was given less than a week to evaluate the models for dangerous capabilities. The rushed timescale means its evaluations “likely underestimate the model’s capabilities”, METR says.
And in other, very notable OpenAI news: the company is reportedly raising $6.5b in equity at a $150b valuation, along with $5b in debt.
The round will reportedly be led by $1b from Thrive Capital, and feature participation from the UAE’s MGX fund.
As part of the round, “Altman has said he intends to remove the profit cap for investors”, the WSJ reports. The company reportedly has $4b in annualised recurring revenue, and 11 million paying subscribers.
On Transformer: Over 100 current and former employees from leading AI companies endorsed SB 1047.
The employees said “it is feasible and appropriate for frontier AI companies to test whether the most powerful AI models can cause severe harms, and for these companies to implement reasonable safeguards against such risks”.
Also this week, SAG-AFTRA, NOW, and Fund Her endorsed the bill, and a poll found that 75% of Latino voters supported it.
The discourse
Oprah Winfrey is worried about AI risks:
“I don’t have a lot of faith. Everybody realizes there needs to be regulations … And everybody’s waiting on somebody else to do it … I certainly don’t expect our lawmakers to get it until it’s too late.”
Mike Krieger says safety is at the core of Anthropic:
“It’s not seen as, ‘Oh, that’s standing in our way.’ It’s like, ‘Yeah, that’s why this company exists.’ … At our second all-hands meeting since I was here, somebody who joined very early here stood up and said, ‘If we succeeded at our mission but the company failed, I would see this as a good outcome.’ I don’t think you would hear that elsewhere.”
Bloomberg’s editorial board says we need to take the biological weapons threat from AI seriously:
“Congress should work with AI developers and scientists to develop criteria for which models may require formal screening and what guardrails can be included in those found to pose the highest risks.”
David Krueger thinks we have to act on AI safety now:
“The whole mindset of ‘let's wait until there's a clear and present danger to do anything’ is suicidally irresponsible.”
Gabriel Weil thinks we need strict liability for AI:
“The development of frontier artificial intelligence systems should be legally treated like housing wild animals or blasting dynamite, because its consequences can also be abnormally dangerous”
Policy
The Commerce Department’s Bureau of Industry and Security released draft AI reporting requirements.
The new rule would require AI developers and cloud providers to report to the government about “developmental activities, cybersecurity measures, and outcomes from red-teaming efforts, which involve testing for dangerous capabilities like the ability to assist in cyberattacks or lower the barriers to entry for non-experts to develop chemical, biological, radiological, or nuclear weapons”.
The White House held a roundtable on “U.S. Leadership in AI Infrastructure”, attended by Sam Altman, Dario Amodei, Jensen Huang, Brad Smith, and others.
It said it’s launching a “new Task Force on AI Datacenter Infrastructure”, led by the National Economic Council, National Security Council, and the White House Deputy Chief of Staff’s office.
It will also “scale up technical assistance to Federal, state, and local authorities handling datacenter permitting”.
Meanwhile, the Department of Energy is “creating an AI datacenter engagement team”.
The House Science Committee advanced nine AI bills this week:
The juiciest is the AI Advancement and Reliability Act, a “stripped down” version of the Senate’s Future of AI Innovation Act to authorise the US AI Safety Institute — though it would change AISI’s name to “the Center for AI Advancement and Reliability”.
There were debates within the committee about how much funding to prove the org; it ended up with $10 million.
The Committee also advanced the CREATE AI Act (which codifies the NAIRR), and the Nucleic Acid Screening for Biosecurity Act (which aims to tackle biological risks from AI), among others.
The Biden administration is reportedly preparing a “national security memorandum” on AI, which would “offer a platform for public-private partnerships and testing that could be likened to a national laboratory”.
According to the Washington Post, “what’s motivating this effort is the growing possibility of [AGI]”.
A bunch of companies, including OpenAI, Anthropic, Microsoft, and Common Crawl, made voluntary commitments to the White House on tackling AI-generated sexual abuse images.
Hugging Face and Stability are two very notable omissions, though Hugging Face did sign a separate set of voluntary principles instead.
Rep. Ted Lieu said his AI task force report will be “finalised by the end of this year and that list of priorities will inform next year's work”, Axios reported.
Sen. Amy Klobuchar and others asked the FTC and DOJ to look into whether AI news summarisation might violate antitrust law.
The Senate Judiciary Committee is hosting a session on “oversight on AI” next Tuesday, featuring testimony from Helen Toner, William Saunders, Margaret Mitchell, and David Evan Harris.
Sixty countries, including the US, endorsed a “blueprint for action” on AI usage in the military. China didn’t sign.
Matt Clifford is reportedly proposing that the UK establish a new agency to promote AI investment and innovation.
Mario Draghi’s EU competitiveness report made numerous mentions to AI, criticising the AI Act and calling for a “CERN-like” AI incubator.
The European Parliament’s AI working group is moving forward, albeit slowly.
Influence
Anthropic endorsed the AI Advancement and Reliability Act and the Future of AI Innovation Act.
TechFreedom called NIST’s AI risk management guidance “inherently hostile to open-source AI models”. (One wonders if that might be because open-source AI models are inherently risky.)
The News Media Alliance met Sen. Schumer and other lawmakers to warn of the threat AI poses to the industry.
The AP has a piece on the concerted lobbying effort to kill AI regulation in DC.
“The primary goal of most of these lobbyists is to convince Washington that the fears around AI are overblown,” the publication says.
TheSalon.ai organised a fly-in for AI startups and a range of senators and representatives, in which they “stressed the need to balance any AI legislation in order to maximise the technology’s benefits while minimising its risks”. (Nice and substantive, then.)
In Taylor Swift’s endorsement of Kamala Harris, she pointed to Trump’s use of a Swift-deepfake, saying that “really conjured up my fears around AI”.
The Tony Blair Institute published a paper on how the UK should regulate AI, written by a bunch of people — most notably Nitarshan Rajkumar, who until recently worked on AI policy in DSIT.
The paper calls for the AI Bill to “build regulatory capacity and advance scientific understanding of the cross-cutting public-safety risks posed by frontier AI”, and for AISI to be “an independent, technical and non-regulatory arm’s-length body”.
Industry
Apple announced new iPhones “built for Apple Intelligence”. Its new A18 chips are built on Arm’s new V9 architecture, according to the FT.
Meta said it will resume training its AI models on posts from UK users, but this time with more transparency and an opt-out approach..
Meta is reportedly building a cluster of 100,000 H100s to train Llama 4.
Oracle said it’s building a “north of [one] gigawatt” data centre, powered by three nuclear reactors. Its shares hit a record high this week.
Reliance Industries also said it’s building “gigawatt-scale AI-ready data centres” in India.
Exowatt, which is backed by Sam Altman, launched a modular solar energy generation and storage product. It says it has 1.2GW in demand.
Google DeepMind released DataGemma, new open-source models that, through integration with Data Commons, should hallucinate less.
HuggingFace released an open-source AI evaluation suite.
Mistral released its first multimodal model.
CloudWeave is reportedly arranging a tender offer at a $23b valuation.
Chai Discovery came out of stealth and released a foundation model for molecular structure prediction. The weights are freely available for non-commercial use.
The company’s raised around $30m at a $150m valuation, from Thrive Capital, Dimension Capital, and OpenAI.
Shanghai Enflame Technology Co. and Shanghai Biren Intelligent Technology Co., two Chinese AI chip designers, both plan to IPO in the next few months, Bloomberg reports.
Infineon said it has developed 300mm gallium nitride wafers, which should make it cheaper to develop GaN chips, used in AI data centres for power supply units.
Enterprise AI company Glean raised $260m at a $4.6b valuation.
Wired has a big profile of Fei-Fei Li’s new company, focused on “spatial intelligence”. It reveals that a16z partner Martin Casado played a key role in founding the company (perhaps explaining Li’s vociferous objections to SB 1047).
Moves
Benjamin Schwartz, formerly of the Commerce Department, joined OpenAI to work on “infrastructure partnerships and policy projects”.
Nando de Freitas is leaving Google DeepMind.
Arvind Neelakantan joined Meta’s AI team from OpenAI.
Ben Brody is joining Punchbowl News as a tech reporter.
Best of the rest
AI got one mention in the presidential debate: Kamala Harris said the US must “win the race on AI and quantum computing”.
A new study found that LLMs are better at generating novel, expert-level research ideas than human experts are.
Researchers released a new AI agent that “conducts entire scientific literature reviews on its own”, and seems actually good at it.
Microsoft has been selling AI tools to fossil fuel companies “as a powerful tool for finding and developing new oil and gas reserves and maximising their production”, an Atlantic investigation found.
A new study found that AI chatbots are surprisingly good at changing conspiracy theorists’ minds.
The Center for AI Safety published a paper saying “LLMs are superhuman forecasters”. Many, myself included, are sceptical.
A new study found that humans are dangerously trusting of AI in life-and-death decisions.
ORCG published a report on how the EU can use “compute for good”.
Nevada plans to use a Google AI system to help make unemployment benefit decisions.
San Francisco officials don’t seem too hot about NetworkOcean’s plan for an underwater data centre.
Audible is asking audiobook narrators to licence their voices to the company for AI narration.
The WSJ is launching an “AI & Business” newsletter.
Parmy Olson’s new book on the AI race is out; here’s a lengthy excerpt in Bloomberg. The WSJ reviewed the book, noting that “her narrative is distorted by her peremptory rejection of concerns about destructive behaviour by AI systems”.
The FT’s Stephen Bush says “not everyone needs to have an opinion on AI”.
Thanks for reading; have a great weekend.