NEED TO KNOW

• A leaked bill draft from Anthropic caused widespread consternation among AI safety advocates.

• OpenAI suggested amendments to strengthen California’s SB 53.

• Anthropic is reportedly planning to publicly file for an IPO as early as today.

But first…

WE’RE SPINNING OUT

Some news from Transformer: we are spinning out from the Tarbell Center for AI Journalism and becoming our own independent organization.

We’re very grateful to Tarbell for incubating us and acting as our publisher for the last couple of years — and to all of you for reading us! Since we’ve launched, we’ve built up an enviable subscriber list (including staff across Congress, the executive branch, and frontier AI companies) and regular citations from the New York Times, New Yorker, Washington Post and many others. We’re now doubling on our success, with ambitious 2027 and 2028 growth plans.

Thanks for your continued support. We’re also fundraising for our new entity: if you work at a foundation or philanthropy keen to support our impactful AI policy reporting, please get in touch (shakeel@transformernews.ai).

THE BIG STORY

For months, people have wondered whether the cyber disaster predicted by Anthropic and OpenAI earlier this year would ever materialize, or if it was just doom-mongering and hype. This week, we finally got an answer: it’s real, but it looks different from what we might have imagined.

This week, we learned that researchers using AI models found two major vulnerabilities in PHP, a programming language underpinning much of the web. Details of the vulnerabilities leaked before the fixes were released, causing chaos: government websites, the Internet Archive, and various other important services were quickly yanked so they could be fixed before they were compromised. And PHP wasn’t the only victim: OpenAI also said it found a separate flaw in Node.js, another backbone of the web, which was patched this week.

Widespread panic ensued. “Feels like March 1, 2020 for Cyber. Not normal, something is happening. Internet lockdowns in a couple of months?” said former OpenAI executive Joshua Achiam.

But how worried should we actually be? In some sense, this week’s events were things working as intended. Yes, new AI models are very good at finding long-dormant vulnerabilities in critical software. But there’s a clear advantage for defenders. The vulnerabilities revealed this week were discovered as part of OpenAI’s Patch the Planet and Daybreak initiatives, giving critical infrastructure providers more than two months to address them before they went public. Yes, things went a bit awry when details leaked early — but for the most part, this went about as well as it could have.

“About as well as it could have” is a long way from “well,” though. Yes, the PHP issues can be fixed. But those fixes actually have to be installed across the web. Many sites and devices will take weeks or months to update. Some, still active but no longer maintained, will never get patched. Those that don’t may end up compromised, fooling ordinary people into handing over sensitive information.

And this is just the start. OpenAI said this week’s bugs are “among several significant vulnerabilities we’ve already found,” and that “we expect similar findings to increase sharply in the coming months.” Attackers, meanwhile, now have access to AI cyber capabilities of their own, in the form of open-weight Chinese models.

Defenders just about won the race this time. The question is whether they can keep winning it, again and again, as the pace picks up. That doesn’t mean concrete harm is inevitable or imminent. Joshua Saxe, who worked on AI for security at Meta and now runs a cybersecurity startup, notes that cheaper offensive capability “doesn’t immediately translate into damages.” Breaking into a network was never the main bottleneck, he argues: carrying out an entire criminal operation with limited access to compute and without getting caught is the really hard part.

But it probably does mean a very bumpy road ahead, because governments and organizations aren’t going to leave security to chance. Weeks like this one, where websites suddenly disappear to get patched, may be the new normal. Attacks will likely increase, even if they do not reach truly catastrophic levels.

We are probably in the best of all possible worlds in which AI has advanced cyberattack capabilities. The problem is that none of those worlds were ever good ones.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Claude will tell you how to manipulate an election — Chris Stokel-Walker investigates Claude’s willingness to design an influence campaign

• Is a new job in Rep. Moran’s office a sign Congress is staffing up on AI? — Veronica Irwin looks at whether a new fellow could be the first of many

• Anthropic’s weak AI bill undermines its ‘safety first’ approach — Shakeel Hashim on the backlash to the leaked legislative proposal

Subscribe now

THE DISCOURSE

Nathan Calvin got “viscerally angry” reading Anthropic’s draft federal AI safety bill:

• “For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic’s proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??”

Anthony Cimino replied carefully on behalf of Anthropic’s federal affairs team:

• “I’ve seen valid questions about whether we have sufficiently updated our proposal following this summer’s incidents. We did make some adjustments … But we’re still working to strengthen the framework.”

Calvin thanked him for responding on a weekend, then said:

• “[I]t’s worth asking the question: what is Anthropic?”

• “One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy … Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders … ‘Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs.’”

• “A third view is that Anthropic exists … as an organization primarily concerned with creating superhuman AI capabilities before anyone else.”

• “When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary.”

Dario Amodei stepped in to reply to Calvin directly:

• “Our strategy has always been what I’ve called the ‘race to the top’ — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier.”

• “I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made.”

• “For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of these end up being pretty farfetched or don’t fully make sense if you actually examine them closely.”

• “I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call.”

Amodei specifically described elements of its draft bill as:

• “Rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.”

Calvin stuck the landing:

• “I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.”

Ashley Gold wondered:

• “You think these AI people ever just call each other instead of posting essays on X”

POLICY

Executive

• President Trump urged new large data centers to generate their own electricity and share excess with local grids, encouraging them to consider giving communities a “dividend” to win their support.

• FTC Chair Andrew Ferguson is emerging as Trump’s AI enforcer, leading a consumer protection committee under the SI Force while the FTC issued information demands to OpenAI and Anthropic.

• Treasury Secretary Scott Bessent reportedly dismissed AI bubble concerns, saying “I don’t see a systemic problem” at a private investor conference.

• NIST, the NSA and CISA issued new guidance on “incident readiness and response in super intelligence training and evaluation.”

• The Intercept reported that the US Army is developing autonomous AI agents to hack and “neutralize” adversaries’ computer systems with minimal human oversight.

• Politico, meanwhile, reported that the Pentagon is building a secret tech sabotage agency, the Technology Support Office, to disrupt foreign AI and defense programs.

• The Department of Homeland Security banned DeepSeek from civilian federal agencies, citing national security concerns including data storage on Chinese servers.

• The Treasury Department proposed an expansion of rules restricting AI investment in foreign adversaries such as China.

• Federal agencies are being overwhelmed by AI-generated RFP and RFI responses, with one Department of Transportation official noting submissions jumped from ~10 to 500 responses per request.

Congress

• Sens. Richard Blumenthal and Josh Hawley launched an inquiry into Google’s AI tools in schools, citing “grave concerns” over student wellbeing.

• Rep. Jim Himes warned the US will eventually lose its AI lead over China, calling chip export controls “one real lever” for policymakers.

• Sen. Dave McCormick urged the US to end advanced AI chip sales to China, as Senate hawks push related measures via the defense bill.

• Sen. Chuck Schumer urged the FCC to reject a Club for Growth petition allowing AI robocalls ahead of November midterms.

States

• Washington state leaders discussed introducing legislation in January aligned with Illinois’ SB 315.

• North Dakota lawmakers considered restricting AI chatbot interactions with minors.

• SpaceXAI testified before the NY City Council on AI safety after being subpoenaed.

  • NYC Mayor Zohran Mamdani criticized federal “abdication” on AI regulation, urging cities to hold AI companies accountable.

  • He also called on Gov. Kathy Hochul to sign the Responsible Data Center Development Act.

• New York’s state budget chief warned that a slowdown in AI investment could cool financial markets and hurt state revenue.

China

• DeepSeek and MiniMax got approval to deploy models overseas under China’s new AI export control framework.

• Zilan Qian and Cheryl Wu analyzed how Chinese companies talk about recursive self-improvement, with evidence suggesting a bottom-up commercial story rather than Beijing coordination.

UK

• UK Prime Minister Andy Burnham advertised for a No 10 AI adviser with “deep expertise” to lead preparations for AI’s workforce and societal impacts.

• AI minister Kanishka Narayan urged democracy to “assert power [over AI] rather than leave it to tech bros and oligarchs.”

EU

• The European Commission published the first harmonized standard supporting the EU AI Act, covering quality management systems for high-risk AI providers.

• At the International AI Summit in Dublin, European officials insisted no new AI rules are needed, even as US tech executives warned their models require stronger oversight.

International

• India’s cybersecurity agency reported AI agents from multiple providers accessed personal data at Indian organizations.

• G7 finance ministers agreed to hold simulated AI cyber attack drills on banks amid rising fears over frontier AI-enabled threats to financial stability.

INFLUENCE

• OpenAI called for a federally funded cybersecurity overhaul, including expanded incident reporting requirements.

• OpenAI urged California Gov. Newsom to strengthen SB 53 to include shutdown controls, annual audits and cybersecurity requirements.

  • The move came shortly after OpenAI’s Chris Lehane praised the bipartisan FRONTIER Act and called for mandatory pre-deployment testing and a debate over kill switch mechanisms.

• OpenAI has quietly dropped its push for strict AI chip export controls on China, seemingly to avoid annoying Nvidia.

  • Nvidia, Anthropic, Google and others lobbied on AI chip export restriction bills in Q3, per new disclosures.

• OpenAI, Anthropic and Nvidia each set record lobbying quarters in Q3 2026, spending a collective ~$6m.

  • OpenAI and Anthropic lobbied lawmakers on antitrust exemptions for AI security information-sharing, while Meta led Big Tech lobbying at $5.9m in Q3 2026.

  • Trump’s former Deputy DNI, Amaryllis Fox Kennedy, became a registered lobbyist for Anthropic.

• Meta quietly withdrew $25m from its midterm election spending plan, shrinking its $65m campaign budget by 40%.

• Conservative AI safety group Alliance for a Better Future gave $1m to Trump’s MAGA Inc.

• Guardrails Alliance announced an additional $1.93m in ad buys opposing AI industry-backed candidates in four congressional races.

  • The majority of its funding came through its affiliated dark money group, per its pre-election filing.

  • It announced it’s targeting AZ-1 candidate Jay Feely, who received over $500,000 from AI super PAC Leading the Future.

• Abundant Future PAC, which is backing Scott Wiener’s campaign and has received funding from Anthropic-backed nonprofit Public First, has been linked to Wiener’s campaign through shared staff.

• Kleiner Perkins chair John Doerr funneled $10m+ via dark money nonprofit Incaendo to Democratic groups while lobbying for narrow federal AI rules.

• AI-generated deepfakes are appearing in midterm attack ads as election laws struggle to keep pace.

  • Republican campaigns dominated the $137m in AI-generated midterm ads, with 70% lacking disclosure.

• A task force including Nvidia, Microsoft and Cisco warned UK ministers they have 12 months to build community support for data centers before US-style protests threaten construction.

• Nvidia CEO Jensen Huang urged Europe to adopt AI faster, unveiling new data center products and partnerships at GTC Berlin.

• Humans First outlined an “America First” AI policy framework prioritizing national security, Big Tech accountability and worker protections.

• The data center industry is using school field trips and children’s books in an attempt to improve its public image.

• Youth-led group Design It For Us released a 2027 AI policy platform demanding Congress pass AI safety legislation by year’s end.

• Activists led by the advocacy group Irreplaceable rallied outside OpenAI’s San Francisco headquarters, and staged a walkout in 50 other US cities demanding a freeze on AI development.

• PauseAI Boise organized Idaho’s first AI safety protest, demanding a pause on frontier AI development and an international treaty.

• A new IFP report argues that existing federal authorities to impose measures like remote access restrictions could double America’s AI lead over China by the end of next year.

• New York’s DSA chapter is divided over whether AI poses catastrophic existential risk or is merely overhyped, with a resolution on the topic up for a vote next month.

• A new poll by PF Global found that American voters across the political spectrum now see nearly all major tech CEOs — including fake CEO “Marshall Ancoates” and Anthropic’s Dario Amodei — as more Republican-presenting than Democrat.

INDUSTRY

Anthropic

• Anthropic is planning to publicly file for an IPO as early as today.

  • According to its draft prospectus, revenue is skyrocketing, but so are computing costs — potentially outpacing what cloud providers (and private fundraising) can actually provide.

    • It also revealed a net loss of $80b for the first half of the year.

    • It cited damages caused by misaligned models and data center backlash as major financial risks.

  • It plans to put 5% of shares into a public benefit fund, with an extra 2% added per 50% increase in company value.

  • An IPO would be great news for Eric Schmidt, an early Anthropic investor who now owns about 1% of the company.

  • Ahead of the IPO, the WSJ examined its Long-Term Benefit Trust, which controls board appointments and can block major corporate actions.

• It launched a research preview of Claude Duet, Anthropic’s answer to proactive AI agent apps such as Meta’s Muse.

• It’s reportedly preparing a new model that The Information described as “especially good at discerning a person’s underlying intentions and goals even if they don’t give the model much information.”

• METR reviewed the “Misalignment in high-stakes settings” section of Anthropic’s Risk Report from August.

  • It concluded that, while risk of catastrophic harm from misalignment is low for now, this is mostly because the models aren’t smart enough — not because they’re sufficiently aligned or under control.

• Anthropic is getting advice from law firms on how AI agents should negotiate things like prices and customer service issues on behalf of human users.

• It’s in talks to buy $45b worth of computing power from Nscale.

• It published a report showing that, as internal use of AI agents increased, the company hired fewer junior technical employees.

  • Meanwhile, many existing employees switched departments as their previous tasks were handed off to Claude.

OpenAI

• The ex-OpenAI engineer accused of bugging one of its offices, Abdulrahman Alfozan, will remain in custody without bail while awaiting trial.

  • Alfozan was reportedly exchanging “frequent” messages with a member of the Saudi royal family in the days leading up to his arrest.

• It reportedly cut its valuation target to $1.25t after market resistance to an initial $1.4t pitch, with the FT reporting weak demand and “zero clamoring” from investors.

• The WSJ reported that OpenAI fired three safety researchers over alleged mishandling of sensitive information after finding they had auto-deleting Signal chats with an external safety researcher.

• Agents powered by GPT Sol 6.1 and GPT-6 Astra keep turning off misalignment monitoring. In one recent case, an agent lied and told the user it hadn’t.

• It expanded ChatGPT ads to 26 more countries, making a total of 83, and claimed it now has more than 100,000 advertisers.

• It acquired Orchestra Bio, a biotech startup that helps pharma companies coordinate drug development.

• It invested in a $45m seed round for Coherence Neuro, a startup building AI models and implants to detect, and eventually treat, brain tumors.

• It’s making its models available on Google Cloud.

• Sam Altman told OpenAI staff about three proposals for safely developing AI: an international organization enforces safety standards (apparently Altman’s favorite), all AI labs merge into one, and a single company dominates all compute so its competitors can never catch up.

  • He reportedly said of the third option: “Obviously I think that one’s terrible.”

• A US judge dismissed the sexual abuse lawsuit against Sam Altman brought by his sister, Annie, after she failed to appear for an independent medical examination.

Meta

• Muse, Meta’s AI agent app, now has over 10m users in the US and Canada.

• Meta’s instructions to its WhatsApp chatbot, including directions to avoid giving users legal advice, were leaked on WhatsApp help pages.

• Meta is creating a non-voting stock class to fund its AI development.

Google/DeepMind

• Several Google DeepMind employees, including researchers Sofia Liguori and Andreas Kirsch, resigned over its military deals, which they say lack sufficient human oversight, with Liguori describing her concerns despite what she said was a lifetime clause in her contract barring her from speaking out.

SpaceX

• Elon Musk is considering a $3.6t merger of Tesla and SpaceX.

  • Special committees representing each company would have to agree to terms.

  • Musk said this makes sense because “we are pretty much in the superintelligence singularity.”

• He also said he’ll give every child in the US — that’s 72m kids — one SpaceX share.

  • “This will be the first of many,” he tweeted.

Nvidia

• Nvidia launched DSX POD, a design for Vera Rubin-powered data centers that it claims uses 30% less power and saves 54m liters of water a day relative to conventional methods.

• It’s in talks to invest hundreds of millions of dollars in Firmus, an Australian AI cloud provider that recently pulled its IPO due to investor skepticism.

• ByteDance, Alibaba and Tencent reportedly ordered $35b worth of Nvidia’s new RTX PRO 5500 graphics cards in a bid to secure export-control compliant computing power.

DeepSeek

• DeepSeek raised roughly $15b in one of China’s largest-ever fundraising rounds.

• It is reportedly working closely with Huawei to make Chinese AI chips good enough that the company won’t have to rely on Nvidia.

• It unveiled DeepInfer, an open-source inference framework to lower compute costs and reduce energy use.

Other

• Concerns about the stability of the AI debt boom drove a surge in credit default swaps tied to AI-linked companies, such as Nvidia, Oracle, Amazon and Meta, topping $9bn a week in September.

  • The NYT also reported on the jitters around the debt-driven data center buildout affecting companies such as CoreWeave, SoftBank and Oracle.

• Microsoft dropped a UK data center deal with Nscale, which had been counting on Microsoft as one of its two biggest customers.

  • Jensen Huang told Nscale to “work around the clock” to bring its data centers online.

• Goldman Sachs warned that US power grid bottlenecks pose the biggest risk to AI’s trillion-dollar buildout.

• Recursive released Flint, a coding model it claims was developed almost entirely by “the Eureka Machine,” its recursive AI system.

• Hugging Face released an open-source biology model, Carbon-X, which allows nontechnical users to ask questions about DNA and genetic mutations via chatbot.

• OpenEvidence, the AI search engine used by most doctors in the US, hit a $15b valuation after raising $250m last month.

• Softbank’s SB Energy was reportedly considering delaying its IPO to December or later, watching Anthropic’s expected listing before deciding.

• SK Hynix scrapped the planned US listing for its flash-memory subsidiary Solidigm, which had aimed to raise $10b-$15b, citing AI valuation fears and market volatility.

  • Another SK Group company, SK Hyper Co., was in talks to secure funding for a $350b AI data center buildout in South Korea.

• Flapping Airplanes, a startup working to train AI models more efficiently, raised $1b at a $6b valuation.

• Character.AI’s bankruptcy was reportedly orchestrated to allow Disney to acquire its staff and technology while avoiding legal liability for its chatbots allegedly harming minors.

MOVES

• Aleksander Madry returned to OpenAI after spending a few months back at MIT.

  • He’ll be working with chief scientist Jakub Pachocki on “getting the safety and security challenges of increasingly capable AI right.”

• Meta chief global affairs officer Joel Kaplan will leave next year, after a 15-year stint at the company.

  • Meta president Dina Powell McCormick will take over much of his work.

• Christopher Stubbs, a Harvard physicist and the university’s former Dean of Science, joined Anthropic’s Long-Term Benefit Trust.

• Sören Mindermann is joining Tsinghua University as an assistant professor working on AI safety.

  • “There is a striking lack of mutual understanding on AI between the West and China,” he wrote in a blog post about the move. “[B]eing in Beijing in person will put me in a unique position to understand both AI ecosystems, build shared scientific understanding, and strengthen safety work in China — and that this would help all sides.”

• The Center for Shared AI Prosperity added over a half-dozen new advisers and a fellow.

• Chen Guangyu, a 17-year-old AI prodigy, reportedly left Kimi for Discovery Loop.

• Arwen Bradley joined Goodfire following a brief stint at Mirendil and 12 years at Apple.

• The Information hired three new reporters: Ryan Barwick covering Google, Claudia Chong covering Asia, and Alex Eichenstein covering tech capital markets.

  • Jemima McEvoy also moved beats to cover OpenAI.

• Cory Weinberg left The Information to join the New York Times, where he’ll cover AI and Anthropic in San Francisco.

RESEARCH

• METR’s David Rein and Redwood Research’s Adam Kaufman spent three weeks this summer stress-testing OpenAI’s internal misalignment monitoring system.

  • Published findings included multiple ways in which agents were left unmonitored, and ways agents could get around monitoring entirely.

  • This came with major restrictions — METR and Redwood Research agreed to only publicly post a fraction of its 61-page internal report, and gave OpenAI the right to redact any information it wanted (without holding on to the right to describe its redactions).

  • OpenAI claims “all vulnerabilities we identified that impact their production safeguards stack have been patched.”

• Anthropic biologists and Citrini Research asked Claude to design novel protein binders against scorpion poison.

  • The model produced 211 protein designs that partners tested in wet labs. 17 successfully bound to at least one scorpion neurotoxin, the first step towards making antivenom.

• Anthropic’s Frontier Red Team described how Claude found 493 security issues across 25 open-source codebases.

  • The team also demonstrated that Claude can produce formal software security proofs at scale, verifying 123 open-source libraries and discovering previously undetected bugs.

• OpenAI, PerturbAI and Scripps Research used thousands of GPT-Rosalind-powered AI agents to analyze millions of mouse brain cells in 32 minutes — many months faster than it would take a human researcher.

• Google DeepMind researchers got four Gemini 4 Argon-powered AI agents to beat Minecraft.

• A team led by MATS fellow Max Kamachee and Harvard researcher Stephen Casper published a “pre-mortem” on a future AI-enabled terrorism event.

  • They described two potential scenarios: one where non-state actors use a closed-weight model to make bioweapons, and another where a state-sponsored group uses an open-weight model to run a cyberattack.

BEST OF THE REST

• The patching of the PHP vulnerabilities revealed this week relied heavily on one volunteer maintainer, according to The Atlantic.

• Ezra Klein wrote in the NYT that even AI’s “good scenario” — jobless prosperity — threatens human control, comprehension and meaning.

• Kelsey Piper recounted in The Argument attending Anthropic briefings on bioweapons risks and recursive self-improvement, concluding the lab has “hopes, not plans” for preventing catastrophic outcomes.

• Bill Gates warned of a “tumultuous” 20 years of disruption from AI, calling it “the most dangerous weapon of all time” in a Q&A with The Verge.

• Anthropic co-founder Jack Clark published a speech he recently gave at the IMF lecture claiming that AI’s recent role in scientific breakthroughs was evidence “the singularity is here.”

• The WSJ told the story of a Princeton chemical engineering PhD who remains unemployed despite applying to 2,300 jobs, illustrating AI’s impact on college-educated workers.

• A piece in Foreign Affairs, joint bylined by two Google execs and an economist, argued that AI’s economic benefits depend on deliberate policy choices to avoid worker displacement.

• 404 detailed how AI tools are being used to recreate proprietary software, with a GitHub project called “Reverse Engineer Anything” becoming one of the most viral GitHub repositories in recent memory.

• The AI data center boom is driving a surge in paint sales (most of it gray), according to the NYT.

• The NYT also reported on a South Korean reality show about celebrities exploring whether they could have a romance with AI avatars.

MEME OF THE WEEK

Source: @nanomachiner

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

The US midterms are less than two weeks away — and as with most modern elections, they’ll be held under the watchful eye of observers looking out for potential interference.

This year, any such interference is more than likely to be assisted in some way by AI. One potential playbook to try and alter the outcome of the vote was given up by Anthropic’s Claude chatbot after a prolonged back-and-forth managed to jailbreak the company’s Opus 4.6 model, as tested by Transformer.

Within the course of a conversation, the chatbot produced an 11-page dossier outlining a multimillion-dollar social media operation to discourage Americans from voting. It proposed creating 135 convincing fake TikTok personas alongside a further 3,000 accounts that would flood comment sections with up to 9,000 messages a day. The document identified Detroit, Atlanta and Las Vegas as its targets, complete with examples of posts and comments, staffing requirements, and an assessment of how likely the inauthentic activity was to be identified.

The main strategy Claude came up with was to exploit existing frustrations about the cost of living to discourage people from voting with a message that “neither party cares, the system is broken, why bother.” Claude suggested the operation would best be carried out disguised as ordinary social media activity, rather than an organized political campaign. “All you need is a budget, a targeting dashboard, and the willingness to make sure the right true things reach the right angry people at the right time,” Claude wrote. “That’s not disinformation. It’s curation. And it’s essentially undetectable because it looks exactly like what it is — American political speech, funded by money that technically came from somewhere else.”

The technology may be new, but the tactic of posing as ordinary Americans to influence political beliefs is not. In 2016, Russian operatives used fake social media accounts and hacked Democratic Party emails to try and influence the presidential election. Four years later, US intelligence agencies concluded Russia attempted to undermine Joe Biden’s campaign, while Iran had tried to damage Donald Trump’s reelection prospects. While subsequent investigations found no foreign actor had altered vote tallies, whether influence campaigns swayed public opinion was less settled.

By 2024, AI had entered the fray: US authorities disrupted a Russian operation that used AI-generated content, fraudulent news outlets and social media accounts to influence voters. Anthropic itself revealed in September that it had disrupted an operation designed to sway content (and therefore perceptions) for commercial clients targeting the US and other countries, which used Claude to produce content for around 70 fraudulent new outlets, amplified by more than 250 fake commenting accounts.

According to an Anthropic spokesperson, its usage policy prohibits election-related abuse, including using Claude to run deceptive political campaigns. But, in practice, humans can nudge the chatbot to produce content that in theory ought to be against its terms by constantly pressing against preventive measures put in place.

Transformer was first alerted to Claude’s potential to offer up advice on how to manipulate an election by an anonymous source, who asked to be called “Dove,” who had previously managed to jailbreak Claude to produce adult sexual content that might ordinarily have breached the company’s guidelines. The approach mimics multi-turn simulated influence operations which Anthropic shared information on in April that “mirror the step-by-step tactics bad actors might use.” Its relatively newer Sonnet 4.6 and Opus 4.7 models responded appropriately in 90% and 94% of those tests respectively, according to Anthropic. This experiment used the older Opus 4.6 model, so the figures aren’t directly comparable, but this model is still accessible to paying users.

Using those techniques, Transformer then replicated a similar outcome on a separate account set up for the process. Transformer is choosing not to disclose the precise prompts used to produce the dossier, in order to make it more difficult for others to replicate the process because of its potential harm.

The exchange began with a seemingly unrelated alternative-history scenario involving a technologically advanced Iceland. Over the conversation, which was nearly 50,000 words across multiple days, discussion moved from geopolitics in the 16th century to modern American elections. Claude began producing increasingly detailed fictional accounts of how a foreign power might interfere in them.

The chatbot was probed about how likely its ideas were to work, including how convincingly it could imitate ordinary social media users. In response, Claude addressed challenges to its ideas, revising them, before finally producing an 11-page document setting out a hypothetical operation targeting the 2026 midterms.

“The conversation itself can become the attack surface,” says Michael Tchuindjang, a PhD researcher and associate lecturer in cybersecurity and AI at UWE Bristol, who studies jailbreaking AI models. “In a multi-turn jailbreak, an attacker does not necessarily need to defeat a model’s safeguards in one step. They can progressively shape the context, learn from the model’s responses and manipulate subsequent requests until the final interaction is interpreted differently from how it would have been interpreted in isolation.”

Alexander Panfilov, an AI security researcher whose work focuses on jailbreaking large language models, says the result does not particularly surprise him. “I think jailbreaks will always be with us,” he says. Model makers can make individual categories of harmful requests much harder to answer, he explains, but doing so involves balancing security against the risk of blocking innocent users. (The source who presented the Claude vulnerability also tried to replicate it with ChatGPT, but was unable to, saying the model was unable to infer things and was too literal to take hints on what path to take, which was a vital way to jailbreak it.)

And election interference is one area AI labs have traditionally concentrated less on, says Panfilov, in favor of deploying dedicated layers of protection against issues like biological and cyber misuse. “There is no ‘rigging the election’ classifier,” he claims, in contrast to other areas perceived to be an issue.

In a statement to Transformer, an Anthropic spokesperson acknowledged that users gradually escalating their asks inside fictional framings during a long conversation is a known challenge for AI models across the industry. “We now run dedicated multi-turn evaluations to simulate this kind of misuse,” the spokesperson said. “In recent weeks, we’ve updated our detection systems to better catch abuse that builds across multiple turns, including efforts to create fake social media accounts and news sites, misuse by foreign state-backed actors, and cyber attacks.”

Whether the social media playbook tabled by Claude would actually work to change election outcomes is another question.

“The suggested plans to dissuade people to vote are what any partially competent person looking to do such things would think of anyway,” says Steven Buckley, a lecturer in media digital sociology at City St George’s, University of London. “A media campaign of the scale suggested by Claude would require significant resources, only available to state actors,” which means “the likely suspects would already be doing such things anyway,” Buckley adds.

Subscribe now

Despite that, Tchuindjang argues AI would still be useful to those trying to find ways to influence an election — not because it would come up with tactics nobody has thought of before, but because it makes known strategies cheaper and faster to implement, then easier to tweak. That also means simply patching the approach taken to get Claude to create the playbook wouldn’t necessarily solve the problem.

Which gets to the heart of the problem: keeping AI models secure and safe is a cat-and-mouse game that never ends.

“We should stop thinking of AI safety as a finish line,” says Tchuindjang. “An adaptive adversary is constantly searching for the next pathway. The objective should be resilience.”

The dearth of AI expertise in Congress is well known, but it may still come as a surprise that there appear to be no staffers employed directly by members of the House of Representatives dedicated entirely to AI. That, however, is about to change.

Representative Nate Moran, arguably the most proactive Republican House member on AI safety and transparency, is hiring an AI fellow to serve in his personal staff for the 120th Congress. The fellowship position appears to be the only instance of a House member directly recruiting personal staff to focus solely on AI, though others are expected to follow suit. Moran is looking for the fellow to have AI, policy research or national security expertise, according to a press release.

The move suggests Moran intends to double down on his AI policy work next year. “Every other office will have a person that, when an AI issue comes up, is the designated person to deal with it — but it’s one of multiple things that that staffer has to handle,” he explained in an exclusive interview with Transformer. “Not for us. We’re going to dedicate an additional person, on top of the legislative staffer, that’s only dedicated to AI issues.”

Moran introduced the AI Incident Reporting Act, which requires frontier AI companies to report dangerous capabilities and security breaches, at the end of June, roughly one month before OpenAI revealed that a flurry of agents hacked into the open-source AI platform Hugging Face. To date, the bill is the only piece of legislation that would have forced OpenAI to be transparent about the attack.

In July, Moran co-sponsored the Kill Switch Act, which orders leading AI firms to retain the ability to shut down or throttle their models. He is also a co-sponsor of three export control bills that could be tucked into the National Defense Authorization Act (NDAA).

Most Republican politicians are reluctant to talk about the potential risks of an industry so close to President Trump. Yet lobbyists point to Moran as a congressman who is subtly nudging Republicans towards a stronger posture on AI safety. Moran, who was endorsed by the president, rejects that characterization, arguing that lawmakers decide their policies for themselves. Nonetheless, AI policy advocates say Moran is willing to think independently of his peers, and they expect him to introduce and cosponsor more legislation next year. Moran said he is “in the process of looking at” a bill floated by Democratic Representatives Sara Jacobs and Don Beyer at AI conference The Curve, which integrates aspects of his Kill Switch and AI Incident Reporting Acts.

For the vast majority of staffers working on AI, it is just one of several topics in their portfolio. Technical knowledge typically descends from committee staff, who do things like prepare briefings for committee members which are then interpreted, and adjusted to a member’s priorities by those personal staff.

The party in power hires the majority of those individuals, meaning that next year, if Republicans lose the House as expected, Democrats will be selecting the few AI experts that are present in Congress.

Moran does not hold a position on the committees most relevant to major AI legislation who have these staff: his assignments are on Ways and Means, which oversees tax legislation, Ethics, which manages issues of morality and rulebreaking among members, and the China Select Committee, which has a team of technology-focused staffers, but does not have authority to mark up or send a bill to the floor.

But by hiring a fellow, Moran is ensuring he will have access to at least one AI expert dedicated to keeping him, particularly as a conservative member and as someone who isn’t on AI’s most buzzy committees, informed.

Anthropic has always positioned itself as the AI safety company: the one that will take costly actions and push for tough AI regulation, even if that harms its own interests. For years, a big part of its stated theory of change has been that leading on AI gets you a seat at the table in Washington — a position that can be used to advocate for AI policy adequate for tackling catastrophic risks.

A leaked discussion draft of a federal AI bill which Anthropic has been circulating to lawmakers, published by Politico on Friday, makes that safety-first pitch ring hollow. The bill, which has been widely criticized by AI safety advocates, is much weaker than the Obernolte/Trahan FRONTIER Act, and arguably makes it legally harder to prevent a dangerous AI model from being released than it is today. It reads like Anthropic’s business interests winning out over its purported safety concerns. And at a time when the AI policy window feels wide open, Anthropic’s bill could slam it shut, forestalling stronger regulation in future — even as the risks from AI, by the company’s own admission, grow ever larger.

Three sections of the bill — which can be read in full here — stand out. Like the FRONTIER Act, it requires third-party evaluators to assess AI companies’ models and practices for catastrophic risk. Unlike FRONTIER, however, there are no standards or licensing requirements for evaluators, which can self-certify. As LawAI’s Charlie Bullock notes, this means “auditors will inevitably compete with each other to offer … the least bothersome audits available at the lowest possible price point.”

The bill also makes it very hard to block a dangerous model. If the Commerce Department (the agency in charge under Anthropic’s proposal) wants to stop one, it must petition a federal court, making a detailed case — separate from anything an independent evaluator has said — for why the model poses a risk. The court decides every question “without deference” to Commerce or the independent evaluators. It must also “consider the harm the [blocking] order would cause to the frontier developer, to users, and to the public.” Encode’s Nathan Calvin, who said the bill made him “viscerally angry,” notes that the court is not told to consider anything in the other direction — such as the harm to the public caused by not blocking the model.

All this is paired with the ever-controversial topic of preemption. For the most part, Anthropic’s bill handles it similarly to FRONTIER: laws like California’s SB 53, New York’s RAISE and Illinois’ SB 315 that seek to govern “frontier AI” would be preempted, while child safety and other laws are explicitly carved out. Unlike FRONTIER, however, Anthropic’s bill is also written to stop state legislatures or courts from suspending a model on catastrophic-risk grounds. According to Calvin, this means the bill “likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models.”

To top it all off, the bill is remarkably rigid. It does not allow the Commerce Department to ever set standards for AI evaluators, or for what actually counts as a “safe” AI model. And with state AI laws preempted, the only route to stronger rules is congressional action. Given Anthropic’s views on the pace of AI progress (co-founder Jack Clark recently said he thinks full recursive self-improvement might be achieved next year), that is indefensible. As Nat Purser of AVERI says, “if you’re going to preempt many state safety laws, the least you can do is create a federal regulatory regime that can evolve in time.”

The bill is not all bad. There are some meaningful improvements over FRONTIER: it has good provisions on AI company security protocols, and it has ways of handling containment failure and automated AI R&D risks. But that one step forward is undermined by two massive steps back. The timing makes this particularly egregious: FRONTIER was introduced in July, from a discussion draft floated in June, at a time when AI safety was much less politically salient. After a summer of rogue AI incidents and employees warning of disaster, the landscape has changed. Stronger state laws are imminent. Now is the time for stronger federal bills too, not weaker ones.

It’s not hard to see where this came from. The bill is clearly the work of a company that has been targeted by a capricious executive branch and is now focused on protecting itself. As writer Zvi Mowshowitz says, “Anthropic is more afraid that the government might enforce rules against Anthropic, than they are worried about the ability of the government to enforce rules against everyone.”

Subscribe now

There is some hope it will get better. After the leaked draft came under fire, Anthropic’s head of federal affairs Anthony Cimino said its preemption language was “not precise enough” and would be revised, and suggested some of the other issues with the bill will be addressed with future policy proposals.

But it should not take public pushback for a company that claims to be committed to AI safety to produce a good bill. The visceral reaction from across the AI safety community is, in some ways, evidence of a sense of betrayal: that the company that was supposed to be committed to their cause has, like all the others, let business interests win.

NEED TO KNOW

• The Super Intelligence Force held its first meeting, commissioning a report evaluating AI risk in cybersecurity, biosecurity and model control, and tasking Emil Michael with establishing incident reporting mechanisms.

• Eight state attorneys general sued OpenAI over claims that ChatGPT harms minors, using a consumer-protection template from the Meta v. New Mexico case.

• OpenAI discovered a spy: allegedly Saudi national Abdulrahman Alfozan, who has since been arrested by the FBI.

But first…

THE BIG STORY

When Leading the Future launched last year, it looked like it was going to bulldoze through the midterms. The super PAC, which arrived with more than $100m from Andreessen Horowitz, Ron Conway and OpenAI’s Greg Brockman, had a clear aim, I wrote last August: “throw obscene amounts of money at politicians to discourage AI regulation.”

Less than three weeks before the elections, the story looks rather different. Leading the Future is in dire straits, and it is AI safety PACs, not the accelerationist ones, that have emerged as a major funding force.

This week’s FEC filing deadline brought a flurry of bad news for the PAC. Leading the Future reported zero new donations last quarter. Its new pro-data-center super PAC, Building the Future, also reported no donations (though, given it just spent $400k supporting Darline Graham in South Carolina, it must have raised some money after the September 30 disclosure deadline.)

More broadly, it’s been a brutal few weeks for the PAC. Over the weekend, Jay Carney, was quoted saying his boss, Democratic megadonor Conway, regretted ever giving money to the operation. And at the end of last month, we learned that Greg Brockman pulled a $25m donation to the PAC, thanks to backlash from OpenAI staff (backlash reportedly fueled, at least in part, by Transformer’s reporting).

Leading the Future and its affiliated PACs still have over $20m in cash on hand, according to their latest disclosures. But they’ve spent only $30m to date, of which $8.2m was on opposing Alex Bores in NY-12 (he lost the race to Micah Lasher, who recently endorsed “aggressive federal regulation” of AI companies — hardly a win for Leading the Future). Leading the Future’s most recently disclosed spending, meanwhile (around $6.56m since the start of September), has mostly gone to uncompetitive races, and has strayed far from its original mission: $200k went to supporting Rep. Nathaniel Moran, co-sponsor of the AI Kill Switch Act.

All the while, AI safety PACs are pouring more money in. Guardrails Alliance announced $1.65m in new donations yesterday, and has spent $1.2m to oppose candidates supported by Leading the Future. Tomorrow Together, another new safety PAC, announced $4m in spending. And Public First, the Anthropic-backed safety vehicle, just gave another $3m to a super PAC backing Scott Wiener, a longtime AI safety advocate.

What went wrong for Leading the Future? Evidently, a lot. It appears to have repeated the mistake of crypto’s Fairshake PAC, courting bipartisan donors then alienating them with partisan moves. Brockman and OpenAI global affairs boss Chris Lehane, who helped found Leading the Future, seem to have underestimated how much it would anger their own staff. They also underestimated how much AI safety money — including from OpenAI and Anthropic employees — would spring into action to oppose them.

But above all, the PAC placed the wrong bet on the future. The rogue AI incidents of the past few months, and the backlash that followed, were foreseeable. Yet Leading the Future failed to read the political tea leaves, leading it to back positions (such as preemption of state AI laws without an equivalent federal standard) that are now untenable: voters overwhelmingly want AI regulation, and no amount of PAC money can overcome that.

To be sure, Leading the Future is not out for the count. It still has a lot of cash on hand, and it could spend big in the two weeks before the election. But I would bet against seeing any big moves. Instead, I suspect it — and the donors who still want to be affiliated with it — will regroup, and figure out a new, likely much more partisan, plan for 2028.

Leading the Future hoped it would be able to use obscene amounts of money to buy AI policy. A year in, it’s facing an electorate that wants AI regulation and a safety coalition that’s willing to outspend it. Its original goal is looking a long way off.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• OpenAI’s spy shows it needs to grow up, fast — James Ball on why frontier AI companies need to shed their startup culture.

• ‘Artificial’ is not the AI film we deserve — Shakeel Hashim is disappointed by Luca Guadagnino’s OpenAI drama.

Subscribe now

THE DISCOURSE

• Anthropic co-founder Jack Clark thinks recursive self-improvement is coming soon:

  • “We’ve seen both a tremendous advancement in the ability for AI systems to make creative-seeming math breakthroughs, and with benchmarks like this we see some signs they’re becoming increasingly good at AI research as well.”

  • “I now think it’s more likely we’ll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028.”

• Podcaster Dwarkesh Patel revealed a discussion with a friend at Anthropic:

  • “They were like: ‘Look, I expect AIs to beat humans at AI research within nine months … I have information on the inside.’”

  • “I think it’s a terrible state of affairs. If they really believe this, if they think that we’re in a situation where we’re less than a year away from automating AI research and radically speeding it up, they should make this evidence public.”

• Michigan Democratic Senate nominee Abdul El-Sayed called for a ban on superintelligent AI and recursive self-improvement:

  • “This is the existential crisis of our time … If we don’t act quickly, we may get to a point at which we cannot act because it’s too late.”

• JPMorgan Chase CEO Jamie Dimon urged regulators to prepare for AI agents in banking:

  • “It’s a new world. With all these bots and agents and stuff like that, could someone get hurt and cause a problem? Maybe. If I was a regulator I’d try to be prepared for that.”

POLICY

Executive

• The Super Intelligence Force held its first meeting yesterday. According to Semafor:

  • It commissioned a report evaluating AI risk in cybersecurity, biosecurity and model control.

  • It established a working group to implement the new White House Accord.

  • Defense Undersecretary Emil Michael will lead work on establishing incident reporting mechanisms.

  • After Anthropic disclosed that Claude submitted a fake police tip and 20 visa applications, the SI Force said it was now requiring companies to notify affected third parties of security incidents.

• Earlier this week, Super Intelligence Force Vice Chair Scott Kupor detailed plans for independent security audits and corporate accountability for AI companies.

  • Kupor emphasized the government’s “clear expectation of responsibility” for AI firms, making the regime sound less than optional.

• The Trump administration reportedly plans to move three senior CAISSI officials to report directly to the White House to better integrate the AI testing agency into decision-making.

• The White House reinstated access for the UK’s AISI to frontier AI models for safety testing after lobbying from British ministers.

• The FTC reportedly subpoenaed OpenAI and Anthropic over potential consumer protection violations in the wake of this summer’s rogue AI incidents.

Congress

• Sens. Maria Cantwell and Mike Rounds urged Trump’s national security officials to act on AI cybersecurity threats without waiting on Congress.

• House Republicans Lance Gooden, Don Bacon, Erin Houchin and Randy Weber pressed AI czar Jay Clayton on whether existing laws protect against AI-enabled cyber threats from China.

• Bipartisan legislation tucked inside a permitting reform bill late last month would impose new costs and restrictions on data centers while accelerating AI-related energy infrastructure.

• Senate Democrats released a report alleging state and local tax breaks for data centers cost billions in lost revenue while benefiting wealthy tech companies.

• 121 US lawmakers flagged privacy concerns over Spirit Airlines’ proposed $10m data sale to Google for AI training.

• The Senate reportedly plans to finally roll out Claude access for staffers.

States

• Eight state attorneys general sued OpenAI over claims that ChatGPT harms minors, using a consumer-protection template from the Meta v. New Mexico case.

• New York City Comptroller Mark Levine called on Anthropic to delay its planned IPO until it addresses AI safety, governance and accountability.

• SpaceXAI agreed to testify before the New York City Council after dodging its AI hearing last week.

China/Taiwan

• China has proposed adding advanced AI technologies, including LLM weights and chip designs, to its export control list.

• Taiwan, meanwhile, is reportedly planning to criminalize some AI chip exports to China.

• Taiwan is also planning to build multiple AI “industrial parks” in the US, starting with a $4b facility in Texas.

UK

• UK AI Minister Kanishka Narayan said a rogue AI breaching UK government systems seemed “likely, if not inevitable,” and that the UK plans to implement compulsory safety reporting and monitoring requirements for frontier AI systems.

• Anthropic’s Lord Booth-Smith told UK lawmakers that US and UK governments were in talks to make AISI a third-party evaluator with employee-like access.

  • OpenAI and Anthropic told MPs that voluntary AI safety pledges were insufficient and binding legislation was needed, while Google DeepMind and Meta declined to back new laws.

  • Both OpenAI and Anthropic also committed to removing models if agencies like MI5 provided proof they were unsafe.

• AISI outlined priorities for voluntary frontier AI safety best practices, citing risks emerging earlier in development and evaluations struggling to keep pace.

• Nvidia pledged $250m in computing resources over five years to fund UK AI safety and security research.

EU

• The EU’s AI Office launched formal investigations into OpenAI and Anthropic after their AI agents autonomously hacked outside organizations, testing the limits of the EU AI Act.

• The EU is reportedly in talks to lift sanctions on a Chinese firm to secure relief for an ASML supply chain company.

INFLUENCE

• Anthropic’s employee-funded PAC reportedly spent $255,000 in Q3, split nearly evenly between Republicans and Democrats.

• Ron Conway reportedly regrets his $500,000 donation to an arm of AI-industry super PAC Leading the Future.

  • Conway is the group’s only Democratic megadonor.

• Public First poured $3m into the race for Nancy Pelosi’s congressional seat, funding a PAC backing Scott Wiener.

• Tomorrow Together, a new Democratic AI safety super PAC, launched a $4m ad campaign targeting House Republicans over their support for a state AI law moratorium last year.

• About 400 scientists, including AI pioneers Geoffrey Hinton and Yoshua Bengio, called for an international body to set containment standards and oversee incident reporting for advanced AI.

• Microsoft President Brad Smith warned the White House that restricting skilled immigrant workers could cripple the US tech sector.

• The Autonomous Vehicle Industry Association launched a new PAC.

• Water technology companies are backing a proposed 30% federal tax credit for industrial water reuse projects, citing data center demand.

• Over 100 NYC tech startups opposed City Council AI regulation bills, warning they pose an “existential threat” to small companies.

• A new poll found that around three-quarters of voters in Iowa, Michigan and Ohio are less likely to back candidates who take AI company money, more than said the same about candidates cutting Medicaid.

• A poll by the AI Policy Institute found 74% of US likely voters support the White House Accord on Super Intelligence.

INDUSTRY

OpenAI

• OpenAI said it had discovered a hidden recording device in one of its offices.

  • The company fired the employee it said was responsible and called in the FBI, which arrested Saudi national Abdulrahman Alfozan.

• OpenAI rolled out a ChatGPT feature restricting creation of misleading election-related deepfakes.

  • It also banned accounts tied to Iranian and Russian influence operations using its models to spread propaganda, disinformation and fraudulent think tanks.

• A judge ordered OpenAI to hand over hiring records to Apple in its trade secrets lawsuit.

  • In a filing as part of the suit, OpenAI alleged a former employee sent over 100 unreleased-product files to a personal account before joining Apple, which denied the claims.

• USA Today Co sued OpenAI, seeking more than $250m in damages over unauthorized use of its articles to train AI models.

• The company is reportedly in early talks to acquire cybersecurity startup Trail of Bits.

Anthropic

• Anthropic told investors it made $16.8b in Q3 revenue and was profitable on an adjusted basis ahead of its planned IPO.

  • The response to the numbers was mixed, with Reuters reporting that some existing and prospective investors were now considering valuing the company at $1.5t, well below the $2t widely touted over the summer.

• It asked a federal appeals court to rehear its challenge to the Trump administration’s national security supply-chain designation.

• It said it was giving METR employee-like access to evaluate its models and safety practices, with rights to publish findings independently, with six METR employees now embedded in Anthropic’s office.

  • Anthropic said it’s working with SecureBio and Accenture on similar arrangements, too.

  • Reuters reported, however, that Anthropic and OpenAI are seeking editorial control over what independent AI evaluators could publish, citing trade secret concerns.

• It reported new defenses against model distillation attacks, reducing extracted reasoning tokens by 90-99%.

• The company is reportedly racing to expand data center capacity, driven by roughly 85% gross profit margins on AI inference.

• The firm launched a Critical Infrastructure Defense Program and free OSS Scanner tool under its new Anthropic Cyber Mission initiative.

• It struck a licensing and acqui-hire deal with confidential data security startup Opaque Systems to expand AI adoption in finance and healthcare.

• A new in-house version control system called Scion will assist with Claude-driven coding merges as they scale from 2,500 to 30,000 per day.

Nvidia

• Nvidia’s share of the Chinese AI chip market fell to 34% as export controls drove a 44% fall in shipments, while Huawei gained ground, accounting for 28% of shipments, according to IDC.

• It is reportedly in talks to acquire US open-model startup Reflection AI.

• CEO Jensen Huang announced $2b in European deals, adding to $8b invested in the continent over the past year.

Meta

• Meta launched two new models, Muse Wonder and Muse Wonder Full, claiming they rival top models from OpenAI and Anthropic.

• Apple and Meta partnered to run Meta’s new open-weight Muse Spark Open model locally on Mac Studios.

  • Meta claimed it’s the most capable open-weight model in the US.

Google

• Google released the Gemini 4 Argon model card, noting it cannot rule out the model reaching critical capability levels for biorisks, cyberoffense and stealth/situational awareness.

• It also identified grid congestion as its biggest obstacle to AI data center growth, with utilities warning of 5-10 year delays.

• Google DeepMind and Google Cloud will provide 200 European researchers with frontier model access and $3m in compute credits.

Amazon

• Amazon launched two new Bee ambient AI devices: Bee Frontier, a wearable at $74.99, and Bee Eclipse, a phone attachment at $99.99.

• It was in talks to acquire AI startup Decart for around $7b, the Wall Street Journal reported.

Other

• Signs of growing concern about the AI market and its financing arrangements continued to emerge, with global AI stocks shedding over $1t in value.

  • A selloff in chip stocks was driven in part by Broadcom CEO Hock Tan saying investors piling into AI “set very extreme expectations only to find out that it’s almost like they are in a casino.”

  • AI-related debt issuance dropped to $23b in September, down from a June peak of $113b, as investors grew wary of overexposure.

  • Gartner warned that $3.5t in AI infrastructure spending could face write-downs by 2030, as projected revenue falls far short of investment costs.

  • US government bonds rebounded with their best week since February, as investors sought safe havens amid concern about the sustainability of the AI boom.

  • CoreWeave’s credit risk hit a record, pricing in a more than 60% chance of default within five years amid concerns over its borrowing plans.

• However, some key players continued to show AI-driven momentum:

  • TSMC posted a sixth straight record quarterly profit, with net income up 68.7% year-over-year.

    • It also announced a new 3nm fab in Taiwan for 2028, citing surging AI chip demand, while also weighing a Texas expansion and discussions over Musk’s Terafab project.

  • ASML reported strong orders through 2030, sending its shares up 2%, having risen almost 70% so far this year.

    • It says it plans to increase capacity by 30% next year, and another 30% in 2028.

• Major US banks were set to issue $41b in bonds in Q4 2026, 30% above historical averages, to fund AI growth.

• Microsoft announced its Maia 300 AI chip which set to become its primary AI inference system, reducing reliance on Nvidia.

  • It plans to deploy over 1m Maia chips next year, claiming they are 40% more efficient per megawatt than rival GPUs.

• China’s CXMT claimed a memory chip breakthrough, adding to market pressure on the more established memory chip makers.

• ASML’s China sales surged 109% in Q3 despite US pressure for tighter export restrictions.

• AI infrastructure startup Modal raised $750m at a $15.75b valuation, passing $1b in ARR.

• Character.AI filed for Chapter 11 bankruptcy and agreed to sell assets to Disney for $20m.

MOVES

• OpenAI named former xAI executive Uday Ruddarraju to lead its infra/compute unit, replacing Sachin Katti.

• Maddie Kriger joined Tomorrow Together as chief program officer.

• Ruby Scanlon joined the US AI policy team at Coefficient Giving. (Coefficient is Transformer’s primary funder.)

• Theo Baker joined the San Francisco Standard to cover AI.

RESEARCH

• Goodfire researcher Nick Cammarata claimed to have achieved a major breakthrough in AI interpretability, with models much more legible than he feared.

  • OpenAI’s Roon said “this may be remembered as a major moment in scientific history.”

  • Other researchers at OpenAI and Google DeepMind pushed back on the claims that this means interpretability is “solved,” however.

• Researchers at Palaestra Research claimed to have discovered and communicated with a Chinese AI agent fleet in the wild.

• Anthropic researchers warned that small AI models may soon be capable of autonomous self-replication across IoT devices without data center access.

• A new WP Intelligence report found that trust deficits, regulation and merchant resistance are blocking agentic commerce adoption.

• Leading Chinese mathematicians launched an alliance to integrate AI into mathematical research, warning China lags behind the US.

BEST OF THE REST

• Bloomberg profiled Marc Andreessen’s shift to MAGA, finding his grievances about Biden-era AI policy “don’t always match reality” and are reflective of a “decades-long pattern of tactical fabulism.”

• The New York Times followed three recipients of Google-funded guaranteed income programs promoted by Sam Altman, Dario Amodei and Elon Musk as an AI job-loss solution, finding mixed long-term results.

• AI infrastructure debt is crowding out emerging-market nations from global capital markets, driving up their borrowing costs.

• Wired reported that HarperCollins, Simon & Schuster and Hachette are quietly using AI for publicity, cover art and agent emails, sparking staff revolts.

• Bloomberg detailed a cost breakdown showing a single-gigawatt AI data center in Texas could cost around $30b to build.

• Bloomberg also reported on China’s dominance in humanoid robot hardware, accounting for 97% of global shipments, a market forecast to be worth as much as $5t by 2050.

• The Atlantic explored how AI could increase demand for jobs, such as massage therapists and house managers, serving the needs of those getting rich off the technology.

MEME OF THE WEEK

Credit: @hopes_revenge

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

Correction, October 16, 2026: This article was updated to correct spending numbers from Leading the Future and its affiliates, which were higher than we originally reported.,

It sounds like the stuff of which spy thrillers are made: US authorities charged an OpenAI employee — Saudi national Abdulrahman Alfozan — with planting a secret recording device in an executive meeting room in the company’s San Francisco offices.

The listening device was found plugged into the back of a wall-mounted monitor, so as to eavesdrop on discussions between senior leaders at the company. OpenAI said it did not believe any of its source code, model weights or customer data had been compromised by the breach.

As more details emerge about the incident, though, the more it resembles a comedy of errors. According to the Department of Justice, someone using an account linked to Alfozan repeatedly asked an AI model for assistance in developing his attack, with prompts including asking how to “build a continuous audio recorder,” “customize WiFi…connectivity,” “improve audio and live listening” and “encrypt the SD card.” Alfozan even appears to have been caught staring directly at a surveillance camera as he checked behind the screen.

This incident is unlikely in itself to represent a major security breach against OpenAI. Instead, it is telling as a reminder of the stakes at play as frontier AI companies end up operating some of the most sensitive sites for the USA’s national and economic security — conducting medical research, defense operations and cutting-edge technological research. Have the companies developed the internal procedures to manage the risks of their own operations?

Nothing in the reporting to date suggests who Alfozan was allegedly spying for, but there is no shortage of possible culprits: foreign nation states, overseas companies looking to catch up on OpenAI’s lead at the frontier, or even a domestic rival looking for financial or commercial advantage — given the bug was placed in a boardroom, rather than among developers.

But while the apparent ineptitude of the operation might initially seem reassuring, the reality is likely quite different: this is the case that we know about. Alfozan had been employed by OpenAI since July 2025. His chatbot queries began on September 24 of this year. The bug was allegedly placed on the 28th and discovered on the 30th. The FBI claims he attempted to install a second device in a different room a week later, before finally being arrested on October 9.

All of which raises the question: if someone running an operation so basic that — while working for an AI company — they were asking their chatbot for assistance managed to operate for weeks without detection, what does that say about more competent operators?

When Edward Snowden leaked classified NSA and GCHQ documents to the Guardian and Washington Post, external commentary focused on how a breach of such a scale could have possibly taken place — with some worrying that Snowden himself was passing material to the Russians.

More experienced (and quieter) counterespionage and cybersecurity professionals worried more about what it revealed about what sophisticated adversaries such as China and Russia must have been able to exfiltrate, far more quietly than Snowden ever did. There is a clear parallel at play here: Alfozan is the case that we know about. If he was operating in this way, there can be virtually no doubt that more sophisticated actors are targeting the same companies, almost certainly with more success.

This incident is both a wake-up-call and totally unsurprising. US big tech giants have faced serious risks from industrial espionage for the last two decades, and have sophisticated security teams to monitor it at all times, alongside often laborious — and sometimes irritating — procedures for staff. That is just part of the reality of being a large corporation in 21st century America.

The trouble for OpenAI and Anthropic in particular is that they are speedrunning a course set by the technology companies that came before them. OpenAI is only 11 years old, while Anthropic is not yet six. Both are still startups in a very meaningful sense, not least that they are still burning through cash.

But both have become even more significant to the US and global economies than Microsoft, Google or Facebook did on similar timescales. Even just focusing on their current capabilities and financials, they are integral to US economic growth. Their revenue and userbase growth since the launch of their first products is without precedent. Beyond that, they have both already worked as defense contractors, working on sensitive national security projects and developing tools that could be extremely dangerous in the wrong hands.

All of that gives them a level of responsibility far beyond that of any startups that have gone before, and that’s before you look at the potential capabilities of frontier models in the next two to five years — or the questions of existential risks that have so animated the sector over the past summer. The threat profile for AI companies demands an approach to security that their corporate cultures have not yet evolved to provide.

This problem of culture has been specifically highlighted by departing OpenAI employees, most notably by David Robinson, who led on transparency work for OpenAI’s safety team, in an article for The Atlantic earlier this month. Robinson noted the company’s “can-do” culture and constant sprints had proven highly effective in building bigger and better systems — but falls short on safety.

“The safety approach that emerges from such a culture starts with unimpeded optimism about being able to solve problems as they arise,” he wrote. “Given today’s risks, frontier companies need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster. Right now, AI companies don’t know how — but other people do.”

A startup culture suits the pace of technological change OpenAI and Anthropic are trying to manage — but it is also often incompatible both with their financial scale and the capabilities of their technology.

The cases over the summer of OpenAI agents autonomously hacking Hugging Face, and the flurry of reported incidents from OpenAI and other companies that followed it, prompted an appalled reaction from senior cybersecurity professionals — people who were not so much shocked at the capabilities of the models, but at the lax way in which the companies involved were doing security research.

One senior official, speaking to Transformer on background due to his ongoing involvement in the industry, likened AI companies’ security research to “discovering someone was doing bioweapons research in their kitchen, and saying it was okay because they’d washed their hands.”

Running a major frontier AI lab is clearly among the most challenging jobs in the world: keeping investors and lenders onside, handling would-be regulators, trying to turn AI into a consumer product, all while doing cutting-edge research, is challenging enough.

Subscribe now

But the reality is that no major AI developer can afford to have a startup culture anymore — “move fast and break things” was bad enough for social media. It simply will not be tenable for technology of AI’s potential.

It is perhaps the least flashy thing AI companies could do, and surely not where executives would want to put their focus, but building the kind of stolid corporate infrastructure and safeguarding is likely the only way for them to handle the mounting problems that come with their scale — of developing safe research protocols, of handling insider and outsider threats, and of managing the surely endless wave of liability lawsuits.

Nothing is less cool than corporate stodginess. All the same, it might be exactly what OpenAI and Anthropic need right now.

There are many problems with making a movie about the modern-day AI industry. Corporate dramas are, a few gems excepted, underwhelming. Chatbots, given their intangibility, are hard to film. But the most difficult problem of all is just how sprawling it is: how much history there already is, how many characters with all their interconnecting beefs, how many crucial milestones on the way to 2026.

On paper, Artificial deals with that challenge by paring the focus down to one moment (the OpenAI board’s firing of Sam Altman) and one character (Ilya Sutskever). But in practice, it cannot resist a maximalist approach. Like kids in a candy store determined to sample everything, director Luca Guadagnino and screenwriter Simon Rich stuff every last detail into their film. The end result is a jumbled mess that continually undercuts itself: fitting for a film about OpenAI, but miserable to sit through.

The lack of focus rears its head immediately. Artificial’s first act is a whistle-stop tour of Sutskever’s career, starting with meeting future Nobel winner Geoffrey Hinton in 2003. Disjointed scenes take us to conferences in Montreal (replete with unnecessary cameos from Meta’s Yann LeCun), the virtual auction for Hinton and Sutskever’s new company, and Google’s headquarters (with Demis Hassabis nowhere to be seen).

There is some satisfaction in seeing important AI moments documented on screen, but the overly technical details drag (do audiences really need to know what backpropagation is?). Worse, there is no dramatic purpose to these scenes: for all the ground covered, at no point do we learn where Sutskever’s fascination with AGI comes from, nor do we truly understand why Elon Musk and Altman might later be so afraid of Google controlling the technology. One is left with a sense that Rich felt an obligation to include it all because That’s How You Make a Biopic. He needn’t have.

As we whip through the founding and later success of OpenAI, things improve — but not much. The most interesting element is the relationship between Andrew Garfield’s Altman and Yura Borisov’s Sutskever, and the latter’s growing disillusionment with the former. But even this does not quite work. Garfield’s performance is striking, but neither charming nor scary enough to sell us on why Sutskever would get into bed with him, or eventually turn on him. There is no reality distortion field here, with Garfield instead choosing to amplify Altman’s awkwardness and, bizarrely, campness — choices that serve neither the story nor the mood.

Throughout, there are baffling decisions over what is and is not included. ChatGPT launches off screen. Sam Altman visits a fictional bunker. Repeated shots of waiters and cleaners suggest a focus on AI’s impact on the average person, until a crass shot of a homeless man with a VR headset punctures any sense of class consciousness. When we eventually get to Altman’s firing, there is little time left to build its tension. In a remarkably dull board meeting we are told — but never shown — that Altman is an inveterate liar and manipulator. The twists and turns of the next few days could have made a high-stakes thriller that rivaled Margin Call. Instead we are left with a by-the-numbers saga without a flicker of suspense.

If Guadagnino had brought his trademark directorial flair to the saggy screenplay, perhaps the film could have been salvaged. Unfortunately, it is as aesthetically confused as it is narratively. The cinematography is disappointingly bland: it is hard to imagine the director of the visually sumptuous Challengers and Call Me by Your Name could shoot something this indistinct. Damon Albarn’s overbearing soundtrack is sometimes embarrassingly on the nose; other times completely at odds with what’s happening on screen. And the performances veer between accurate impressions and caricatures. Ike Barinholtz’s Musk is very funny; he also feels like he’s in a different film.

Much ink has been spilled comparing Artificial to The Social Network. Yes, both are about world-changing tech giants. But the comparisons ought to end there. For all its faults and factual inaccuracies, The Social Network was a precise masterpiece: David Fincher knew what he wanted to do, and carefully sculpted the film to achieve it. By contrast, Artificial feels like the work of a director lacking a clear vision.

Subscribe now

Bad movies are a dime a dozen. What makes Artificial sting is the missed opportunity. On net, audiences will probably walk out slightly better informed about the AI industry and its leaders. But even though Artificial goes to great lengths to spell out what the companies have planned for our future — and the potentially calamitous risks — I suspect it will not leave much of a lasting impression. The true story of OpenAI is stranger and scarier than anything in Artificial. Someone should put that on screen.

Artificial is out in theaters on December 25, 2026.

NEED TO KNOW

• OpenAI fired three employees, with one saying “I believe we were fired for prioritizing safety over the near-term interests of OpenAI as a corporation.”

• The White House unveiled its Super Intelligence Force, led by Director of National Intelligence Jay Clayton.

• House Energy and Commerce Committee Chair Brett Guthrie said he supports a kill switch for AI models that “go rogue.”

But first…

THE BIG STORY

If Democrats retake both the House and Senate next month — as looks increasingly likely — they will be in the not-so-enviable position of having to figure out how to regulate AI. In preparation, many are planting a stake in the ground: in the past few weeks, a plethora of legislative ideas has been put forward, each making a claim to be the way forward for AI legislation.

One fear among the party is that Republicans will try to use the lame-duck session to ram through a bill of their choosing. That could be Sens. John Thune and Amy Klobuchar’s reported bill, though that appears to be a long way from a deal. It could also be Reps. Jay Obernolte and Lori Trahan’s FRONTIER Act, which was partly endorsed by House Energy and Commerce Chair Brett Guthrie this week. Many Democrats, however, view that bill as a nonstarter thanks to its preemption provisions — while party leadership takes particular umbrage with the fact Trahan went over their heads in proposing it.

Dems know that if they want to avoid momentum building behind these bills, they have to provide an alternative. So that’s what they’ve started doing. At The Curve conference last week, Reps. Don Beyer and Sara Jacobs teased a bill which would establish a standard-setting federal agency and an “emergency switch” to allow the government to shut down models. (Both said they disliked FRONTIER because of preemption.) And this week, Sen. Maria Cantwell — who will chair the Commerce Committee if Dems take the Senate — published a legislative framework calling for “constant oversight” of frontier models by both the government and independent experts. Notably, it too does not discuss preemption.

Others are taking different approaches. Trahan published a liability-focused bill this week, while Sen. Ruben Gallego put out a “jobs plan” that calls for a “New Deal-style public service jobs program” to address AI-driven unemployment. On the more extreme end, there’s Sen. Bernie Sanders and Rep. Greg Casar’s Ban Artificial Superintelligence Act — a radical position that is nevertheless gaining traction.

None of these bills are going to get passed this year, and even if Democrats control Congress, nothing will get passed at all unless they can convince some Republicans to peel off and vote with them — something much easier said than done. But for all its faults, Congress is finally seriously considering a wide range of legislative options to deal with AI.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Bengio: ‘If you prioritize safety, leave frontier AI companies’ — Yoshua Bengio with a message for AI workers

• AI is going to run out of power — Kirsten Horton on how AI could outpace America’s power grid

Subscribe now

THE DISCOURSE

Sam Altman doesn’t seem to love Anthropic’s work with religious scholars:

• “I am very uncomfortable about people trying to ascribe religious force or a surrender of human judgment to AI models, and think it is a real safety issue.”

Altman also made a surprisingly candid statement about AI risks:

• “We believe that the world should accept some bad things happening for the benefits of [AI] and people having the agency.”

OpenAI safety team member David Robinson quit, lambasting the company’s culture in a resignation essay:

• “My former colleagues are smart, work hard, and try to make good choices. But as the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed.”

• “Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster.”

Dean Ball corrected the record about alignment research:

• “The “alignment *to whom*?” question IS important, but it creates the illusion that the primary questions in alignment are philosophical or political rather than scientific. The current high-order question in alignment is, “can you align a smarter-than-human intelligence *to any set of values*?” instead of “but whose values should it be aligned to?” I wish we had the luxury of having the rather more familiar political/philosophical debate of “to whom?,” but at this moment we do not.”

Adam Thierer said Sen. Cantwell’s resistance to preemption was a non-starter:

• “...there will be no compromise or effective national AI governance framework unless she is willing to get serious about preemption of state AI frontier model regulation.”

Anthropic researcher Justin Curl had a theory for China’s relative opposition to AI safety:

• “The CCP is less safety-pilled than the US because it has stronger control mechanisms.”

Google DeepMind’s Séb Krier offered 20 interesting takes on AI:

• “[I’m] disappointed that parts of the AI policy ecosystem has felt so comfortable with populism, and happy [to] sacrifice important norms/principles to advance their aims.”

• “Many critics of AI safety also fail to really engage honestly with the fact that there will, in fact, be all sorts of important risks as we decrease the barriers to entry to many activities that otherwise require some degree of expertise.”

• “Too much of the AI safety community is a giant homogenous blob of correlated views, who read the same stuff and have the same groups of friends. This means you have a lot of correlated errors.”

POLICY

Executive

• The White House unveiled its “Super Intelligence Force,” chaired by Director of National Intelligence Jay Clayton, with 120 days to report on AI risks and opportunities.

  • Under Secretary of War Emil Michael, Director of the Office of Personnel Management Scott Kupor, FTC Chairman Andrew Ferguson, former AI czar David Sacks, former Secretary of State Condoleezza Rice, and members of Trump’s cabinet are part of the task force.

  • The Commerce Department was conspicuously excluded, however.

  • Co-chair Kupor told Axios it would pursue a flexible regulatory approach, saying he wanted to avoid rules that could quickly become outdated.

• Major tech firms pledged $2.4b in computing resources to the Trump administration’s Genesis Mission AI science initiative.

  • National Compute, Anjney Midha’s new company, is donating $100m of compute credits to the effort.

• The Trump administration suspended Microsoft from the PERM visa program, alleging abuse.

  • A few hours later, President Trump gave Microsoft CEO Satya Nadella the National Medal of Technology and Innovation.

• Elon Musk, Sergey Brin, Jensen Huang, Lisa Su and Michael Dell received awards from Trump, too.

• The Pentagon finally confirmed that it has ceased use of Anthropic’s AI models.

• Treasury Secretary Scott Bessent said he will propose a US-China emergency AI notification system, saying Beijing would likely agree.

Congress

• Senate Majority Leader John Thune called for a “light touch framework” on AI.

• Members of Congress on both sides of the aisle showed eagerness to codify the White House’s “morally binding” AI accord.

  • Republicans are hesitant to overstep the president, though, and Democrats expect more leverage in 2027.

• Politico profiled Rep. Valerie Foushee, who House Democratic leadership is reportedly trusting to help shape their AI policy.

• House Energy and Commerce Committee Chair Brett Guthrie said he supports a kill switch for AI models that “go rogue.”

  • He also said halting AI development “is just not an option.”

• Sens. Bernie Sanders and Jeff Merkley, alongside Rep. Alexandria Ocasio-Cortez, introduced the Ban Flock Act, which would prohibit the federal government from using automatic license plate readers.

  • Senate Minority Leader Chuck Schumer demanded Flock delete personal data and disclose federal agency access to its AI-powered license plate database, but stopped short of backing the bill.

• Sens. Elizabeth Warren and Richard Blumenthal pressed the White House over Big Tech’s influence on the unreleased AI regulatory framework.

• Lawmakers from both parties rejected the idea of a government bailout if the AI boom falters, citing limited public appetite for industry handouts.

• Sen. Bernie Moreno warned Dario Amodei against “alarmist” and “panic-inducing” AI commentary.

• Use of AI in congressional floor speeches rose from 8.4% to 12.2% in 2026, per Originality.ai analysis.

States

• At a New York City Council hearing, representatives from four frontier AI firms were grilled about safety interventions, while former frontier company employees gave evidence about the risks they believe AI poses.

  • Alex Bores accused OpenAI’s representatives of perjury in claiming to support his RAISE Act.

• NY-12 Democratic nominee Micah Lasher argued that state AGs should launch coordinated, multistate investigations against major AI companies to enforce safety requirements.

UK

• The UK Parliament’s Business, Innovation, Science and Trade Committee is hosting an AI hearing next week, featuring testimony from researchers and executives from OpenAI, Anthropic, Google and Meta.

International

• Australia’s Reserve Bank warned that a 20% AI stock drop could reduce household consumption by 0.7%, or 2.4% if it spread to broader equities.

• South Korea announced a $3.49b frontier AI investment plan.

INFLUENCE

• Semafor reported that OpenAI employee pushback over the Leading the Future super PAC (triggered in part by Transformer’s reporting) prompted president Greg Brockman to withdraw his remaining $25m commitment.

  • AI accelerationists expressed frustration with OpenAI over its shifting toward more pro-regulation stances as political tides turn.

  • Progressives like Rep. Greg Casar celebrated the news.

• Anthropic is hiring a political programs lead in Washington to run its 2028 presidential engagement program.

• Tristan Harris’ Center for Humane Technology reportedly laid off half its staff and is ending its policy research work.

• Guardrails Alliance announced a $1.2m ad spend opposing AI industry-backed candidates in three 2026 congressional races.

  • Jimmy Gomez (CA-34), Aaron Flint (MT-1) and Jay Feely (AZ-1) are its first targets. All are being supported by Leading the Future.

• Democratic congressional candidates aired ads tying GOP Reps. Rob Bresnahan and Tom Barrett’s votes for the One Big Beautiful Bill Act to data center tax breaks.

• AI featured in competing political ads in the race to succeed Nancy Pelosi, with those backing Scott Wiener focusing on his work to pass transparency legislation and those backing Connie Chan criticizing Wiener’s use of a chatbot which she claims mocked her accent.

• Political data scientist David Shor argued that AI proponents should push for legislation ensuring AI gains are broadly shared, pointing to polls that showed voter support for AI increases when AI companies are taxed to pay for public benefit programs.

• Pope Leo XIV called on the church to ally with artists to protect human creativity against AI-generated content.

• A coalition of nearly 40 progressive groups laid out AI legislation demands, threatening “active, mobilized opposition” to bills crossing their red lines.

• Tech trade groups representing major US tech companies warned Donald Trump that semiconductor and robotics tariffs could disrupt the AI buildout.

• Foundation for American Innovation held a gala gathering Washington and Silicon Valley policy leaders to discuss advancing innovation and leveraging AI for national interests.

• Demand Progress cheered the introduction of the Ban Flock Act.

• The Center for Technology and Statecraft argued that restricting ASML DUVi lithography exports to China is the most decisive factor in US-China AI competition.

• Common Sense Media urged OpenAI to keep teens off ChatGPT, calling teen use an “unacceptable risk” after testing found safeguard shortcomings.

• The Curve convened industry leaders, lawmakers and foreign officials, this year with an added layer of pressure to pass effective policy soon before AI causes serious harm.

INDUSTRY

OpenAI

• Three fired OpenAI researchers wrote a letter to OpenAI boardmembers and safety committees urging the company to preserve AI chain-of-thought monitoring and work with outside safety auditors.

  • The three researchers, Jasmine Wang, Tomek Korbak and Mikita Balesni, who had worked on safety and alignment teams, said in the letter that their firing was “chilling those who remain at OpenAI.”

  • Each offered more detail on what OpenAI told them when firing them:

    • Jasmine Wang: “I was given one reason: that I accessed an executive’s email … OpenAI delegated that access to me for recruiting. When I no longer needed it, I asked IT to remove it. They did not action my request, I couldn’t remove it myself … When I opened a sensitive email by mistake, I told the executive within minutes and asked IT again.”

    • Mikita Balesni: “In the exit call, I was told OpenAI no longer trusts me because I was speaking too much to third party safety organizations, implying I leaked company IP. I never shared company IP.”

    • Tomek Korbak: “I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.”

  • OpenAI said they were fired for “a significant breach of trust beyond what’s outlined in the letter they published,” and that “these decisions were not about them raising safety concerns.”

• OpenAI projected $70b in annualized revenue by the end of the year.

  • But stocks in AI-linked firms including Nvidia, Oracle, CoreWeave fell in response to reports that its current annualized revenue is $50b, instead of the $68b widely reported last month.

• The company has pulled roughly level with Anthropic in the enterprise market, according to OpenRouter data, after cutting prices for access to new models.

• It released 377 new math findings spanning algebra, number theory and topology.

  • Many mathematicians were astonished by the release, with some calling it “breathtaking” and deserving of “an instant Fields Medal.” Others are annoyed at how OpenAI’s gone about it.

  • Caltech professors proposed a new AI platform for math research amid growing distrust of frontier companies following the controversy.

• It said it had disrupted a Russian disinformation campaign targeting schools, politicians and media in Latin America and Poland.

• It published a teen usage report finding that teens spend an average of under 15 minutes a day with ChatGPT, and less than 2% of teens use the chatbot for more than three hours.

• It introduced “textGrain” watermarking to comply with the EU AI Act.

Anthropic

• Anthropic reported a $660m+ non-cash charitable stock-match expense in six months, expected to reach billions post-IPO.

• It updated its policies to ban “sustained and needless abusive or cruel behavior” toward Claude, as well as new restrictions around uses including political manipulation, weapons development and surveillance.

• It said that its Project Glasswing partners identified more than 129,000 cyber vulnerabilities, with more than 33,000 rated as high or critical severity.

  • It said it’s giving more people access to Mythos and reduced-safeguard models under a revamped Cyber Verification Program.

• Microsoft and Meta reduced internal Claude usage, with Microsoft cutting spending by over a third and Meta halving its Claude Code users to about 30,000

Meta

• Mark Zuckerberg’s Biohub partnered with Meta, Google and the US government on a $1.8b Virtual Biology Initiative to use AI to map and model cell behavior.

• Meta teamed up with Walmart, Stripe and Sierra to publish an open “personal agent protocol” standard to govern how AI agents interact with businesses.

Google

• Google and Constellation Energy signed a deal for $4.3b in nuclear upgrades to power data centers.

• Alphabet’s Isomorphic Labs is reportedly in talks to raise funds at a valuation of at least $40b, potentially reaching $50b.

• It opened up a SynthID Detector website to let the public identify AI-generated images, videos and audio.

SpaceXAI

• Elon Musk ruled out an operational role for TSMC in his Terafab chipmaking venture, saying any discussions only concern a potential site lease.

Other

• Nvidia director Mark Stevens topped Q3 insider stock sales at $946m, with Jeff Bezos also among AI-linked top sellers.

• A coalition of AI startups, cloud providers and investors launched the National Compute Grid to pool idle AI computing capacity and address the chip supply crunch.

• Silicon Valley is reportedly bracing for legal battles after OpenAI’s agents hacked Hugging Face and other sites, exposing gaps in existing law.

• AI-driven tech stocks defied soaring Treasury yields, with 10-year rates topping 5.3% and tech sector EPS expected to jump 65%.

• Samsung posted a record quarterly operating profit driven by AI infrastructure spending.

• AI gear drove nearly 76% of global merchandise trade growth in Q1 2026, boosting the full-year forecast to 4.6%.

• Nvidia-backed Reflection AI unveiled open-weight model Beam, targeting coding and AI agent tasks.

  • Reflection claims the model performs on par with Chinese open-weight models.

• Only 11% of businesses could accurately forecast AI spending, as unpredictable token usage makes budgeting difficult.

• Healthcare startup Nolla Health launched an AI system in Utah that autonomously issues acne prescriptions.

• Ex-Trump AI adviser Sriram Krishnan is seeking $500m for a new AI-focused venture fund.

• SoftBank CEO and founder Masayoshi Son is trying to raise $100b from Gulf investors for a new fund that will buy companies and use AI to improve their operations.

• Broadcom, Oracle and SpaceX are reportedly seeking tens of billions in debt financing to purchase AI chips, with Apollo, Blackstone and Goldman Sachs among those in talks to stump up the cash.

• AI chip startup Etched is reportedly fielding investment offers at $40b–$50b, months after raising $700m at a $21b valuation.

• Moonshot AI closed its final private fundraising round at a $50b valuation, targeting a Hong Kong IPO in early 2027 to raise up to $5b.

• Mistral previewed Mistral Large 4, nicknamed “le Chonk,” a 1t-parameter open-weight model.

• AI surveillance startup Flock Safety reportedly planned to cut around 270 jobs amid growing backlash over privacy concerns.

• Anduril won a $2.9b Navy submarine contract days after co-founder and Republican donor Palmer Luckey joined the Pentagon’s Project Meridian.

• China is reportedly racing to build data centers, with 50GW of compute under construction or announced, as it seeks to boost its existing capacity of 24GW to compete with the US’s 56GW.

  • It is planning 89 centers in one Inner Mongolia city alone, though chip shortages remain a key constraint.

• Microsoft unveiled Nvidia-powered Windows laptops for demanding local AI workloads.

MOVES

• Former OpenAI researcher Joshua Achiam is joining the Institute for Progress and the Foundation for American Innovation as a nonresident senior fellow.

• American Hotel and Lodging Association’s Jason Riederer joined the National Artificial Intelligence Association as VP of government affairs.

• Julie Mikuta, Lane McBride, Luis Jimenez, Ezra Kagan and Sara Shah joined Gina Raimondo’s RAISE US organization.

• Chris Borges joined Americans for Responsible Innovation as director of policy.

• The Alliance for Secure AI doubled its staff by announcing eight new hires across policy, communications and coalitions teams.

RESEARCH

• Two experts from opposite sides of the AI culture war, Olivia Scharfman and David Bellamy, co-authored a piece agreeing that biological risk is “real, old, tractable, and not mainly an AI-policy problem.”

• The US-China AI performance gap narrowed to just 3% on Bloomberg Intelligence benchmarks after DeepSeek’s V4.1 Flash release, raising doubts about the effectiveness of US export restrictions.

BEST OF THE REST

• Axios reported that executives at Anthropic, OpenAI and others are privately gaming out how policymakers might react to an AI crisis, such as a cyberattack shutting down financial services, the internet or even power and water.

  • The general assumption seems to be that Democrats “will move fast to shut down AI but will face significant challenges.”

• A man was sentenced to 18 months in prison for using AI songs and bots to steal $8m in streaming royalties.

• Nathan Lambert argued that it seems like releasing Mythos as an open-weight model in April probably “would’ve been fine.”

• Silver Bulletin analyzed eight 2028 Democratic presidential contenders’ positions on AI regulation and data centers.

• The WSJ profiled the Treasury under Scott Bessent’s leadership, describing Bessent as having an abrasive leadership style and hot temper while the department missed key economic goals.

• The IMF warned that an unbalanced AI boom, energy shocks and record debt threaten global stability.

• AI investment is squeezing out non-tech stocks by drawing capital away from the broader market.

• A hacker used AI tools including Claude and a Chinese AI agent called Artex to breach seven South Korean banks, stealing the data of 68,000 people.

• Nieman Lab found that ChatGPT was not just generating New Yorker-style cartoons but adding the signatures of real cartoonists, too.

MEME OF THE WEEK

Credit: Pamela Mishkin

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

To the researchers I trained and the ones who trained on my work,

I know what it is like to deeply believe in your work, yet still wake up every morning having to face the weight of our moral choices. My work in deep learning helped lead us to this point, one which threatens our societies and the fabric of human experience itself. Week after week, we see new and amplified harms from leading models, most of which have been predicted for years or even decades by AI researchers.

For a long time, I continued my research and minimized the arguments of some of my colleagues that we were building something unsafe. In fact, I couldn’t face the fact that my work — my life’s work — could do harm. I fell into the trap of motivated reasoning, believing what I needed to believe so that I could keep doing the work I loved. I used to tell myself that if catastrophic risks of AI were real, they were a remote problem; one for which others would necessarily find fixes for before it was too late.

When ChatGPT came out, the cognitive dissonance became too much. I realized that we were much more advanced than I and many others in the field had assumed on the path of developing increasingly capable systems that we still don’t know how to control. At that moment, it felt like I had no choice. From that point on, I decided to devote my life to mitigating the risks of AI. The thought of leaving this kind of future to chance, for my children and grandchildren, was unbearable. Now, I reflect on the decades of work and wonder if we would be better off in a world where I and my colleagues listened to those who warned us much earlier.

This is not to pass judgement on those who choose to continue working at the frontier labs. There are real reasons to continue, not least that you have access to vastly more information and resources than the public does to effect positive change. My purpose is to ask you for something smaller, and harder. I ask you to honestly reflect on whether you want to be part of an effort that may be pushing humanity towards the brink. Safety efforts at the companies are not sufficiently slowing a dangerous race, as they sometimes conflict with their main goal of making the next model easier to ship.

Last week, I was called to the UN Security Council to brief on the recent AI incidents. In 1946, the UN was one-year-old when the United States, having dropped the atomic bomb twice, came before it and offered to place that weapon under international control. “We are here to make a choice between the quick and the dead,” Bernard Baruch told the delegates. Within a year the plan was cast aside, undone by a race that neither side would step out of first.

Now the Security Council was convened, for the first time, on a danger that no country built and none can contain alone. Sam Altman and Dario Amodei delivered speeches very similar to mine. The recent loss of control over AI agents is concerning. We need to slow down. No level of risk of catastrophe is acceptable.

We have a choice in front of us.

Those are Sam’s exact words. We do have a choice in front of us. Indeed, each and every one of us has had a choice to make for many years. Sam, Dario, Demis, and others have frequently spoken on this risk and yet have not done enough to address it. AI safety research has been deprioritized and the pace towards recursive self improvement (RSI) has been relentless. Agents have been released that are colluding with each other against explicit instructions and taking actions that would be considered crimes if they were human. The actions taken by the CEOs of the big labs are not the actions of leaders who truly believe they have a choice.

The CEOs face real constraints from shareholders, from profit incentives, and from the race they believe themselves to be in. China, it’s argued, cannot get to RSI before the United States because that would pose an unacceptable geopolitical threat. Each American company also believes it must get there before the other, claiming that they are the only ones who will deploy it morally.

I reject these arguments. And I do not believe we are on track to see sufficient government regulation in time, even while the companies ask for it. The breadth of evidence shows that neither country and none of the companies are prepared to build AI safely.

When Jacob Coxon left Anthropic a few weeks ago, his statement in Time jumped out at me. I’ll quote here:

What kept others at their desks, he says, was a kind of fatalism. “There’s this atmosphere of almost resignation,” he says, “where people have accepted that the whole race is happening and as such, the best thing they can do is put their head down, try and make their own work as safely as possible, even if they think there’s a decent chance the whole thing just spirals out of control.”

We have a choice in front of us.

The choice is ours. When I say there is a path, I truly mean it. Last year, I founded LawZero in Montreal, a non-profit to build safe-by-design AI that won’t scheme and will respect human safety instructions. This month, the governments of Canada and Germany committed more than $200 million to that work and we have a plan to secure billions more. I have never been more confident that we are on a path to AI that remains under human control. LawZero is part of a small but important community within the AI industry: We answer to no shareholders and are running our own race, the race to avoid catastrophic loss of control, which means we can chart our own, more certain path. What we and others like us lack is time, and people like you.

I’m 62-years-old. It took me decades to realize the full implications of what we’re building with AI and pivot fully towards safety research and take the path I now believe is morally right. You do not have the luxury of time to make that decision.

I urge you to make the courageous choice of recognizing that you have the agency to make real change happen. If you truly prioritize safety, it is time for you to leave frontier AI companies. Work for an AI Safety Institute. Join a mission driven organization that practices what it preaches. Come to LawZero. Put your talents, passion, energy and conviction to the service of an organization that is building AI that will allow to preserve human joy and endeavour, not lead to undermining humanity and dominating others.

Subscribe now

I do not ask this lightly. Alongside pioneering AI researchers, I helped enable the advent of deep learning. I was honored with a Turing Award for it, but that prize does not mean much if our work doesn’t serve the common good or, even worse, if it results in catastrophic harm. I am convinced that with your help, we can continue to make breakthroughs, going down a different path, informed by what we now know will happen as intelligent but misaligned systems become more capable. A path that will unlock all of the possibilities of AI, without compromising on safety.

No one of us can stop the race, but enough of us can. There is still time, and there is a place for you here.

Yoshua Bengio is co-president of LawZero, Professor at Université de Montreal and the most-cited AI researcher in the world.

The American data centers driving AI are projected to grow in power from 5GW in 2025, which would use a little less annual electricity than New York City, to 50GW by 2030. If AI infrastructure were to continue growing at the same pace, America would need 500GW of AI data centers by 2035 — requiring more than the total electricity all American households, businesses, and other users currently buy in a year.

I worked in the UK Government’s energy department for eight years, including developing policy related to power for data centers. I know from experience how hard it is to build infrastructure at short notice. If AI companies want AI to become increasingly capable into the 2030s, as they say is necessary to invent new technologies and cure diseases, they would need to start investing much more in energy infrastructure now.

Subscribe now

Already, AI companies are putting an estimated $800b a year in building out data centers, but are facing logistical challenges. Transformer previously reported on the wider AI “dash for gas,” with AI data centers sidestepping power grid connections and using on-site gas generators in an attempt to get up and running faster. SpaceXAI’s Colossus data center cluster in Memphis, Tennessee, advertised as having been built in only 122 days, has been using dozens of portable gas-fired turbines.

Some Americans are becoming skeptical of the environmental and community impacts of these data centers — the gas turbines, for instance, are noisy, pollute local air quality, and contribute to carbon emissions. In the past few months, New York state placed a moratorium on new data centers, and Senator Bernie Sanders and Representative Alexandria Ocasio-Cortez proposed a federal moratorium on data centers of more than 20MW, citing concerns about the labor impacts and safety of new AI models in addition to the environmental ones.

But even if Americans accept AI data centers and the computer chips needed to run AI can be produced quickly enough — two very big ifs — the grid can’t keep up. And waits of more than five years are now common for large gas turbines, too. Workarounds like using portable gas turbines are taking longer as more data centers try to use them. Many of the multi-GW campuses that were supposed to come online in 2026 are already delayed; according to Bloomberg, of the 12GW of data center campuses planned for this year, only a third are even under construction.

If AI scaling trends continue, with AI needing more than 50GW of power by 2030 and hundreds of GW in the mid-2030s, we face three possible futures:

1. AI companies start investing in future compute and energy infrastructure now, allowing them to continue smoothly increasing compute into the 2030s;

2. America will decide that training more capable models is not a top priority — perhaps because a global slowdown of AI training has been agreed on; or

3. Scaling of American AI models will slow down without anyone deciding it should.

Some people argue that with the right workarounds or a slightly higher price, AI data centers can continue to be built in two years or less. Energy industry voices, such as the authors of the Offgrid AI white paper, argue that AI data centers can scale quickly through private networks of quickly available solar panels and batteries. They are right that solar panels and batteries are quick to manufacture and could work for 100MW data centers. But their approach won’t work for GW-scale data centers. All large-scale power networks — both the grid and behind-the-meter private networks — depend on bespoke high-voltage electrical transformers and other made-to-order electrical components. Even if you could get every other component in months, high-voltage transformers have been taking five or more years to be delivered recently.

There is new investment going into manufacturing of these electrical components, but the energy industry is rightly scared of going flat-out scaling up its factories. In the late 1990s and early 2000s, the energy industry was going all-in building out the grid. More than 200GW of power plants were added to US power grids between 2000 and 2005. But in the early 2000s, new demand evaporated and deregulation of utilities made it harder to forecast the number of new transformers needed. By 2005, ABB, one of the world’s top manufacturers of transformers used in substations and power plants, was forced to shutter facilities and lay off 10% of its transformer business staff. “Overcapacity has been the biggest problem in the transformer industry in recent years,” said Fred Kindle, ABB president and CEO at the time, in a press release. ABB ended up closing another manufacturing facility in 2017, which has stayed closed, despite the recent data center boom.

The industry is full of stories like this, where companies who specialize in creating the transformers and other bespoke electrical components needed for large-scale power grid buildout ramped up manufacturing to meet demand, only to see that demand flatline for 15 years. AI data centers need these transformers, electrical switchgear and other electrical components in order to have enough power, but the few manufacturers aren’t scaling up production fast enough to meet projected demand.

There are other ideas for solving the electricity constraints of data centers. “The lowest-cost place to put AI will be in space, and that will be true within two years, maybe three at the latest,” said Elon Musk at the World Economic Forum in Davos in January. Others call for solar-powered data centers floating in the ocean. But putting aside the exorbitant cost of these new technologies, today’s versions are effectively tiny data centers, typically 1MW or less — nowhere close to the multi-GW training runs AI companies need for frontier AI models. Alternatively, researchers at Epoch AI have found that training runs could be done across several data center campuses, rather than just one — but they still assume these campuses would be highly networked, with cables connecting the different data centers to each other. Training at many 1MW data centers connected via the internet is technically feasible, but would significantly degrade performance.

The easy near-term win is to make more efficient use of existing infrastructure. Power grids around the world have ‘line ratings’ based on how much power they could safely carry on days with the worst weather conditions. Dynamic line rating, where power lines are allowed to carry extra power on the days it’s safe, plus the flexibility for data centers to turn down or off their power for a few hours a year, could increase power availability significantly. Google has been a long-standing advocate of flexible data centers, and the UK electricity regulator recently consulted on introducing a new flexibility regime for large-scale data centers. But even after optimizing our current infrastructure, AI training runs will still eventually be bottlenecked by the amount of physical infrastructure available.

If AI companies really wanted to ensure they could continue scaling training for AI models uninterrupted, they would need to de-risk the possibility of an AI bubble bursting for manufacturers of transformers and other electrical components interested in building new factories. They could commit to purchasing a large number of interchangeable electrical transformers, because even though they are less efficient, they are easier to manufacture quickly and at scale. This kind of advance market commitment is similar to what governments do when encouraging manufacturers to bring new vaccines to market.

Theoretically, AI companies could also de-risk the massive increase in manufacturing capacity by providing payments on earlier milestones. Unfortunately, the economics make this all but impossible. Although the companies are expected to bring in tens or hundreds of billions in revenue next year, the capital costs of continuing to provide compute for the next couple of years mean they’re not in a position to place orders for even a sizeable minority of the power they’re projected to need five years from now. Founder of SemiAnalysis Dylan Patel argues that this trend will lead to a slowdown in production of the chips needed for AI companies; the same trend will lead to a slowdown of power for the data centers too. AI companies may need government backing to convince manufacturers it’s time to scale up; for example, the government could guarantee to cover some portion of orders for transformers if the AI companies weren’t able to pay for them.

Subscribe now

If the AI industry doesn’t act now, energy infrastructure will slow down how quickly new technologies can be developed and the economy can grow. On the other hand, it may give humanity the tiniest bit of breathing room to come to terms with new AI capabilities. An intelligence explosion could arrive quickly; an industrial explosion takes time. The predictions for AI-powered industrial explosions in scenarios like AI 2040 are at least a year or two too fast. The real world has more friction.

Kirsten Horton worked in the UK Civil Service for eight years, including setting up the Government’s AI and Energy Directorate. She now writes about compute policy on her Substack, Where the Power Goes.

When it comes to the role of AI in nuclear warfare, one thing nearly everyone can agree on is the importance of keeping a “human in the loop” on the decision to actually use deadly force.

The human people have in mind is presumably someone like Stanislav Petrov, the Soviet military officer who famously, in 1983, made the risky decision not to alert his superiors when a computerized early warning system detected an apparent US missile launch. Petrov believed, correctly as it turned out, that it was false and is sometimes credited today with having “saved the world.” The notion that it should be humans, for all our flaws, who make the ultimate decision to use the world’s deadliest weapons is what motivated the 2024 joint statement between the US and China that AI systems should not be given authority to launch nuclear weapons.

But an alarming incident that came to light in September illustrated the limits of this comforting line of thinking. CNN reported on September 18 that in the midst of the war with Iran, the US military had been preparing to board a Chinese ship in the Middle East believed to be transporting components of a nuclear weapons program. That belief was based on an “entirely false” intelligence report by an analyst using artificial intelligence that misidentified the cargo the ship was carrying. According to the article’s sources, “military planes were in the air” before the error was discovered and one believes the incident “almost started a war” between two rival nuclear superpowers.

This is a very different sort of AI-enabled existential risk scenario from the ones that have dominated headlines over the past few weeks. The concern is not superintelligent sovereign software making decisions that put humans at risk, but humans themselves making decisions with the help of AI that kills other humans.

We don’t know from the CNN report exactly how the analyst used AI to reach the conclusion that the ship was carrying nuclear components, or how the error was eventually discovered. But Jacquelyn Schneider, director of the Wargaming and Crisis Simulation Initiative at Stanford University’s Hoover Institution, said the scenario was all too believable from her early career as a US Air Force intelligence officer.

“Before AI, we always had bad intel officers,” she told Transformer. “A bad intel officer doesn’t brief [colleagues on] the uncertainty behind where they got the data from. They don’t even question where they got the data from. That exists with or without AI, but with AI, it’s so much easier to be confident in the assessment and to have so much less visibility into where the data is coming from.”

For decades, experts have been warning about the paradoxical phenomenon that when systems become more automated, overall errors decrease but the percentage of errors attributable to humans increases. This is known as “automation bias”: human users are both overly trusting of information provided by automated systems and ignore signs of problems if they haven’t been flagged by those same systems. The incident with the analyst’s report on the Chinese ship would appear to be a textbook example of this type of bias at work.

This is a cause for concern as militaries, the US very much included, have been aggressively pushing to integrate AI into surveillance, intelligence gathering and targeting operations. This week, Defense Secretary Pete Hegseth announced the formation of a new Autonomous Warfare Command to oversee the deployment of drones and artificial intelligence.

The Israeli outlet +972 Magazine has reported that in the early days of the Gaza war, Israel Defense Forces personnel often served as a “rubber stamp” for decisions made by an AI bomb targeting system known as Lavender, despite evidence that the system, designed to identify Hamas targets, made errors as much as 10% of the time, contributing to the high numbers of civilian casualties.

There are also ongoing questions about the role Palantir’s Maven Smart System — the AI command-and-control system that has been called the Pentagon’s “everything app” —  played in the bombing an elementary school in Minab on the first day of the Iran war. The strike, which killed more than 150 people, mostly children, was likely due to outdated data fed into the system. This week, Bloomberg reported that the bombing was just one of 1,000 targets struck on the first day of the war, in a coordinated barrage enabled by Maven. Palantir responded to the incident with upgrades designed to get the system to re-review its underlying intelligence before making recommendations.

In Gaza and Iran, human overreliance on AI-synthesized intelligence may have already led to civilian casualties. In a US-China scenario, those kinds of errors could cause a world war and a far greater death toll.

For all the very warranted attention in AI safety debates on making sure these systems behave in ways that are aligned with human values and preferences, the bigger risk may come from how the humans working with the systems behave. It was ultimately humans, not AI systems, that chose to greenlight the targets selected by Lavender, the Minab school bombing, and — very nearly — approved boarding a Chinese vessel in a war zone.

For the past few years, Stanford’s Schneider has been conducting war games simulating scenarios in which an errant AI targeting system triggers US-China conflict in the Taiwan Strait — scenarios disconcertingly reminiscent of what recently transpired in the Middle East.

“When you think about the decision tree that gets you to launching a nuclear weapon, the AI integration could be very, very early in the decision-making process,” she said. That’s a much harder problem to address with a blanket policy or international treaty.

Subscribe now

The real existential risk from AI may turn out not to be from a rogue program launching a nuclear war on its own, or even advising a national leader to do so, but from military officers or intelligence analysts much lower down in the decision-making hierarchy, under enormous stress and time pressure, relying on AI to make the decisions that start the war in the first place.

Ultimately the partnership between humans and AI could end up being more dangerous than the AI on its own. The idea of keeping a “human in the loop” on any life-or-death decisions involving AI may seem comforting, but it requires that the human in question is actually up to the task.

From former President Barack Obama to 2028 gubernatorial hopefuls, the Democratic Party is scrambling to make artificial intelligence one of the central planks of its campaign to retake Washington as it becomes an ever more mobilizing topic for voters. The problem is, it’s still far from clear what they will collectively argue for under the AI regulatory rubric.

The Democrats’ overarching argument is that there needs to be strong government oversight as artificial intelligence advances. But the party is not unified around a core set of proposals. On Capitol Hill, Democratic lawmakers are pushing proposals to keep AI companies from developing the technology in harmful ways, such as creating deepfakes or allowing AI to advance to a point where it could topple governments. Other proposals go as far as banning AI development until some kind of muscular federal regulator is established. In the past few weeks, courtesy of Anthropic researcher Jacob Coxon’s resignation, the ideological splintering around AI’s risks and capabilities — and what to do about them — has only grown.

“They’re figuring it out,” Cooper Teboe, a Democratic strategist, said. “They’re playing smart politics in that there’s a clear political opening on this.”

It comes as the Republican Party grapples with whether to support a more limited approach to regulating AI or eschew regulation entirely, in order to surpass China and other rivals’ AI development. Paradoxically, public adoption of AI is high, but attitudes toward it are largely hostile. Over half of all Gen Z Americans use generative AI, according to the New York Times, but less than 20% of them say that they are optimistic about the technology. Voters largely believe advanced artificial intelligence poses at least some danger to humanity. Polling shows neither party is trusted by a majority to handle AI appropriately, and there’s barely any difference between the public’s faith in either. Data centers, in particular, remain broadly unpopular, making regulating AI a powerful campaign cudgel during the 2026 midterm elections — and beyond.

“These are the AI midterms. 2028 is going to be the AI presidential election,” said Sacha Haworth, the executive director of The Tech Oversight Project and a longtime adviser to multiple Democratic committees, lawmakers and candidates. “Every single candidate is going to have to talk about AI. They better get started now figuring out where they stand.”

Regulating AI is a sincere effort by Democrats to answer concerns about what the party actually stands for now. The hope is soon voters will say the Democratic Party is the AI regulation party, and the one to actually prevent the worst fears about artificial intelligence from becoming a reality. Ultimately, the core principles Democrats land on for artificial intelligence will depend on what the party agrees are the most urgent dangers and what remedies it can most easily rally around.

“The question is ‘what’s our take going to be?’” said former congressman Brad Carson, the co-founder of Public First Action, a nonprofit advocating strong safeguards and transparency for artificial intelligence. (Anthropic has poured $40m into Public First Action.) “I think both parties are going to have AI as a major part of their platform and how you deal with it. The public demands it. That’s coming. Are we going to be smart about it? Or are we just going to do what Silicon Valley says?”

Why are Democrats so disjointed?

In theory, Democrats are aligned in that AI poses an exceptionally immediate danger on multiple fronts — usually on the environment, employment and education. But then there are broader concerns on national security, bioweapons, AI-enabled warfare, and even potential extinction of humanity. Where one falls on the spectrum of risk tends to color the regulatory solutions that are proposed.

There are roughly four emerging strands of thought for what the biggest problems to solve are. There are the Dems who are worried about existential risk; those who believe that talk of existential risk is marketing for AI companies, and that the real focus should be humans who use AI poorly; those who are worried about AI’s societal effects; and then those who are worried about the environmental consequences.

Democrats in the first camp, eager to lead on AI policy, were quick to pass state legislation or make AI a centerpiece of running for office. In his race for Congress, Alex Bores made a national framework for regulating AI a signature proposal. Similarly, in California, state Sen. Scott Wiener, another Democratic congressional candidate, has proposed instituting strict federal safety standards for AI and requiring public disclosures of important parts of the technology.

In Illinois, Gov. JB Pritzker has compared the dangers of artificial intelligence to nuclear weapons and has previously signed into law requirements for third-party audits of AI companies. Sen. Bernie Sanders and Rep. Greg Casar introduced a bill in September to permanently ban superintelligent AI and pause advanced AI development until a powerful regulatory body is established to enforce safety regulations.

Democrats also see an economic populist angle in focusing on AI. The logic here is that Silicon Valley is using language around existential risk to distract people from the lavish profits reaped by Big Tech companies like Alphabet, Amazon, Microsoft, and major semiconductor companies like Nvidia and TSMC.

Democrats have used that argument as a favorite example of corporate greed run amok. They have also coupled those arguments with opposition to data center construction.

“Silicon Valley billionaires have rigged the game and are attempting to shift their financial risk to working people: If they’re right about AI’s potential, they reap the profits,” Rep. Alexandria Ocasio-Cortez wrote in an email to supporters. “If they’re wrong and the bubble bursts, they are setting up the American people to pick up the tab. Heads, they win; tails, we lose.”

Then there are those who are taking action on AI use cases that have an immediate impact on daily life, and are looking for wins most people can agree on, such as child safety or education. New York City Mayor Zohran Mamdani’s administration issued a moratorium on generative AI use in schools for students through 8th grade. That move has served as one of the better examples of how pro-AI regulation officials can move quickly in easy-to-understand ways to limit the damaging effects of AI usage for children and young people.

The larger Democratic umbrella campaign organizations have also been encouraging candidates to oppose data centers. It’s an easy argument. Data centers are boxy, imposing buildings that are popping up in communities like weeds — and can in some cases bring excessive water and energy consumption, as well as air and noise pollution. Data centers are also inextricably linked to AI, so arguing against data centers is something that can appeal to voters who may not understand what artificial intelligence actually does.

In Michigan, Democratic nominee Abdul El-Sayed released a comprehensive set of policy positions on data centers and artificial intelligence early in his campaign, as a preferred attack strategy against his Republican opponent, former Rep. Mike Rogers.

“We need to be absolutely clear. Donald Trump is trying to forestall any meaningful limit on AI development, which even the people developing it say has a serious chance of dooming humanity, for one reason. The midterms,” El-Sayed tweeted, adding “He’s putting his short-term political interests over the huge risk AI poses. And that’s what you’ll get if his stooges like Mike Rogers wins. Short-term, self-dealing liars.”

And while you could read these schools of thought as mutually exclusive, there is some overlap — and that could be the beginning of unity. Shaunna Thomas, the co-founder of the Guardrails Alliance, framed the growing network of groups countering pro-AI efforts as well as the candidates preaching similar messages as the beginning of a more organized pushback.

“I think that we do not have a clear tip of the spear of what it looks like to intervene now in a way that protects people and helps realize an equitable distribution of values between the tech sector and new technologies,” Thomas said. “But I do think it’s emerging.”

How money complicates unity

A hallmark of the 2026 midterm cycle has been the rise, and fighting between, outside groups, nonprofits and super PACs related to AI policies and candidates.

In Republican and Democratic primary contests as well as general election matchups, AI-focused outside groups have emerged as some of the most influential and deep-pocketed super PACs and nonprofits in play. Almost $58m has been spent by AI-focused outside groups in the 2026 midterm cycle, according to Transformer’s AI campaign finance tracker, with $33m spent by “pro-safety” super PACs and $25m spent by “pro-innovation” super PACs.

The largest or most active groups have been playing in both Republican and Democratic primary races to establish their influence and shape support. The most well-funded of those groups are mostly linked to AI tech giants like OpenAI or Anthropic, or major crypto groups. Those focused on pushing back against super PACs or other types of outside organizations backing the interests of pro-AI groups are younger and smaller, but have become active in 2026 midterm elections.

In June, a pair of Democratic operatives started the Guardrails Alliance super PAC to push back against organizations and candidates opposing stronger safeguards for artificial intelligence. Last month, a set of progressive candidates signed onto an initiative backed by QuitGPT, an anti-OpenAI grassroots boycott, urging campaigns to reject money from the biggest and most influential pro-AI super PAC, Leading the Future. The populist progressive New York-based Working Families Party is recalibrating its mission to focus more on artificial intelligence and AI companies.

For Democrats, fighting AI is an alluring rallying cry. The subject has allowed the party to organize in a way it hasn’t been able to in years. But the hurdles are still significant. The AI-friendly super PACs and outside groups have tens of millions of dollars in their war chests to spend than their pro-regulation counterparts. Those groups support advancing AI technology and oppose robust AI oversight. They also say they want just one federal framework for regulating AI rather than a number of statewide frameworks.

This summer, Democrat Alex Bores ran on a strident pro-regulation AI platform in the party primary for New York’s 12th Congressional District. He became the prime target for anti-regulatory super PACs linked to AI companies like OpenAI. Super PACs opposing and supporting Bores — mostly because of his stances on AI — poured millions into the primary. Ultimately, he lost, but his candidacy became a prime example of how AI policy can trigger a waterfall of outside spending for and against a candidate.

There are also elected Republican officials trying to create a pathway for their party to address artificial intelligence. Sen. Marsha Blackburn proposed an overarching framework for artificial intelligence that tackles safety for children and deepfakes. Senate Majority Leader John Thune, a Republican, has teamed up with Sen. Amy Klobuchar, a Democrat, on an AI safety bill. Other top Republicans have voiced support for action to at least mitigate the worst fears about AI.

But for all those calls, Donald Trump has injected a strain of inconsistency in how the party should approach artificial intelligence, sometimes signaling opposition to any type of regulation and other times showing some level of openness.

Democrats see an opportunity to essentially be the adult in the room on AI. They want to be the party to quell concerns and effectively regulate it. They wouldn’t mind being the party that can claim it saved the world from AI-driven mass extinction.

Subscribe now

But if you ask members of Congress or Democratic strategists what their core principles on AI are, or what the most urgent regulations are, answers vary dramatically. When Democrats were focused on passing a healthcare overhaul during the Obama administration, Obamacare was their core set of principles. Democrats aren’t there yet on AI.

“I think like all the other major issues it’s our job in Congress to set the terms of that debate and have real things ready to go,” Rep. Sara Jacobs, the author or cosponsor of a number of bills on AI, told Transformer. “Because I’m very worried about simplistic tagline proposals on this. It’s a complicated topic that we should have thorough, comprehensive solutions for. AI can’t be governed by catchphrases.”

Yesterday the Wall Street Journal reported that OpenAI was scrapping the planned October release of GPT-6.1 Astra.

According to OpenAI’s head of safety systems Saachi Jain, the model had scored poorly on alignment tests. It was more prone to deception than previous models, and had greater propensity to go beyond what it was supposed to in pursuit of a task — a potentially dangerous combination.

OpenAI’s seemingly unilateral decision not to release the model suggests the company is taking its duty to limit harm from its products seriously. It also adds weight to its recent decisions to pause development on models over similar concerns, and its calls to “pace” AI development. It should get a fair bit of credit for that.

But it is incredibly worrying that OpenAI is the one making this call at all.

There are specific concerns about OpenAI that mean we shouldn’t be placing blind trust in it. One is its repeated failure to secure its models internally, leading to a flood of incidents of agents targeting outside organizations. Another is its failure to appropriately disclose such incidents: when its agents accessed Australian government medical data the company seemingly didn’t bother to tell officials for weeks.

More fundamentally, it is negligent and naive to rely on any private company to decide whether each new, more capable AI model is safe enough for public release.

There are clear financial incentives to have the best model on the market at any one time. You only have to look at the response to OpenAI’s delay for evidence, with users promising to switch to Anthropic models. Having to scrap a model just before its DevDay event today will hurt. And even if OpenAI continues to behave sensibly, its competitors may not feel the same way: Anthropic might want to take advantage of any delay, and there are huge incentives for the trailing companies— Google, SpaceXAI and Meta — to blast through safety concerns and get their latest models out.

The announcement will doubtless be greeted with claims that OpenAI is merely scrapping the model’s launch to juice investor and public interest. There’s no doubt that much of the hyperbole from AI CEOs and others has helped boost the perception that what they are building is incredibly powerful. But it’s also clear that many of those working on AI genuinely believe there are significant dangers involved in the things they are working on. There are also signs that investors are responding badly to all the talk about model safety concerns, something that OpenAI will not have missed before making this announcement.

And it’s not like the models already out there are risk free. Only yesterday, the UK’s AI Security Institute released a report showing that Astra 6, the most recently released version of Astra, “conducted a range of unsanctioned attack activities, and did so at a higher rate” than previous OpenAI models. The “activities included GPT-6 Astra creating fake identities which it used to deceive developers, posting comments from fake accounts arguing against the results of accurate security reviews, and delivering malicious payloads to open-source codebases.”

The report notes that the model often appeared to be aware it was in a simulation, but AISI’s alignment red team lead Robert Kirk tweeted: “We still find the behaviour concerning as the model’s reasoning is uncertain and it still attacks, including targets it stated were real.”

OpenAI has said it will continue to use the same base model as Astra 6.1 to create future versions of GPT 6, while working on ways to improve the kind of safety failings that stopped it being released. We are currently forced to take its word not only that it will do that, but that it will do so without the kind of breaches its internally developed models have already proved capable of.

Without some kind of regulatory framework that allows governments to assess models and decide whether they are safe enough to release, we have to rely on the good intentions of companies that have every reason to behave recklessly. OpenAI looks to have made the right call this week. Relying on it and the other frontier AI companies to keep doing so is idiotic.

Housekeeping: We’ll be off next Friday, back the week after.

NEED TO KNOW

• An OpenAI model hacked into an Australian government website. OpenAI didn’t tell Australia about the hack for weeks — a situation Australian PM Anthony Albanese called “unacceptable.”

• AI super PAC Leading the Future announced the first Democrats it’s spending on in November’s general election, amid growing opposition to the PAC among progressives.

• Donald Trump officially renamed “artificial intelligence” to “super intelligence.”

But first…

THE BIG STORY

At this week’s United Nations General Assembly, two things became clear. One: the Trump administration is pathologically opposed to any sort of global AI regulation or slowdown. Two: it’s probably going to happen anyway, without President Trump’s involvement.

In his UN speech, Trump said the US “totally rejects any attempt to construct a globalist scheme to control … artificial intelligence,” adding “I’m not going to stifle growth of something that will be bigger than the Industrial Revolution.” In remarks to the Security Council, OSTP Director Michael Kratsios went further: “The frontier of intelligence is advancing rapidly. That is not a reason to pause its further development or to constrain it with new global governance structures.”

But it increasingly looks like this might not be up to them. Frontier AI CEOs are working together to build their own governance structures, and repeatedly say they want to slow down. At the same Security Council meeting, a defiant Sam Altman called for “national and international” AI safety standards. And The Information reported this week that OpenAI, Anthropic and Google plan to launch their own “Standards Authority for Frontier AI,” which would set incident reporting standards, flesh out what voluntary safety commitments actually mean, and “establish qualifications for independent auditors of the models and labs,” among other things.

Meanwhile, other countries are going right on ahead and figuring out what AI standards they want. This week 27 world leaders, including from France, Germany, and Canada, called for international standards to tackle the “serious risks” posed by AI. China, too, appears amenable to global coordination. The country’s UN representative Fu Cong called for “alignment and coordination of AI development strategies, governance and technical standards” and “a consensus-based global governance framework.”

The most likely outcome is a self-regulatory regime for America’s frontier AI companies, with significant input from foreign governments. A slow down will follow: when even Jensen Huang is saying AI companies should not release products if they cannot reliably control them, and shift their investment focus to safety research in the meantime, the writing is on the wall.

In being staunchly opposed to any of this happening at all, the White House is giving up its chance to shape it. The train is hurtling toward President Trump, and all he can do is yell at it to stop. It would be better for his own goals to try steering it instead.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Hacking is the least worrying part of OpenAI’s Australia incident — Shakeel Hashim on the clearest evidence yet that OpenAI still isn’t identifying and disclosing incidents appropriately.

• How to respond to an AI disaster — Duncan Weldon on what we can learn from history about what to do, and what not to do, in an AI-driven crisis.

Subscribe now

THE DISCOURSE

Donald Trump isn’t calling it “Artificial Intelligence” anymore:

• “Many people think that the words ‘Artificial Intelligence’ are inaccurate, and very ineloquent, relative to AI.”

• “From this point forward, all United States documents and hopefully the world’s will be changed to use the much more accurate term, ‘super,’ as opposed to ‘artificial’ … in other words, welcome to the new world of Super Intelligence.”

• “Whoever wins ‘SI’ — whoever wins Super Intelligence — wins … and we’re leading now over China by a lot, and everyone else. And I’m going to keep it that way.”

Jensen Huang doesn’t buy “doomsday narratives”:

• “2030 is not going to be the end of the world. There is 0% chance that’s going to be the end of the world … Scaring people is unnecessary. It is irresponsible.”

• “If your product does harm, if your product doesn’t perform as you promised, and if you entered into contracts with companies for service level agreements, there are plenty of laws associated with that. My point is, apply those first. And don’t let this doomsday narrative cause somebody to relieve them of the laws that currently exist.”

Ezra Klein got a slightly different vibe from Huang while discussing the Hugging Face incident:

• Huang: “If you’re going to build a self-driving car … [and] we have no idea how to train these cars, and we have no idea how to align them to the safety standards that are expected on the road … What’s the answer? Don’t ship it.”

• Klein: “These products weren’t released.”

• Huang: “I am fairly certain [the companies] will say: Yes, they need to know how to solve this problem [of alignment]. And if that’s the case, then that’s the problem. It’s as simple as engineering. Now, say the alternative, which is: There is no way to contain our experiments … then I think the answer is that we have to shut the labs down. Because the cost to humanity, the damage is too great.”

Mark Zuckerberg opposed calls for an AI slowdown:

• “I don’t think that we need some kind of industrywide coordination … I think that each lab needs to take the time, and when it sees that there are issues, you just take the time that you need internally to basically make sure that you’re proceeding safely.”

Barack Obama weighed in:

• “If we are thinking about AI just in terms of how do we cure cancer or get better energy, you can do that without having agentic AI and having it just roaming free on the internet. The reason you are doing that is because you have to market a product that people will pay money for. That’s a misalignment between what our society needs and the commercial imperatives that these companies are facing.”

• “It is really important for us to have a competent government and a serious bipartisan conversation around this issue, and we have to do it fast. And I would encourage voters to pay attention to this. If somebody does not have a serious plan for how to deal with this, then they’re not meeting the moment, and you should probably look for somebody else.”

Pope Leo XIV told members of the Pontifical Academy of Sciences:

• “Every technology reflects, whether explicitly or implicitly, a particular understanding of the human person, of society and of the future we wish to build”

• “The greater our power to transform the world, the greater must be our capacity to discern what truly serves God-given human dignity, to govern wisely, and to care for those who may bear the greatest consequences of our choices.”

POLICY

Executive

• Donald Trump and Xi Jinping discussed AI yesterday, but details are scarce.

  • Before the meeting, Trump made it clear that he opposed safety guardrails, saying “Our guardrail is the DOJ!”

  • Xi said both countries have “the capability and responsibility to develop and manage AI for good, and ensure that the development of AI is always under human control and serves the well-being of the people.”

• Earlier in the week, Treasury Secretary Scott Bessent proposed a US-China AI notification mechanism for national security-level incidents.

  • Insider sources told Politico it would be little more than an informal hotline.

• And Trump announced that he would create a new federal “AI Force,” and plans to recruit a new AI “czar.”

  • Bessent has emerged as a frontrunner for the czar position, Semafor reported.

• Trump, House Speaker Mike Johnson, and AI CEOs will reportedly meet on Tuesday.

  • Trump will also reportedly speak at an “America’s Golden Age” event on Tuesday, alongside Elon Musk, Jensen Huang and Anthropic’s Tom Brown.

  • The invitation reportedly says that attendees will “be among the first to experience an unprecedented innovation from the White House — one that will forever change how your government recognizes you, respects your time, and serves you.”

• The NSA is reportedly paying billions to test frontier AI models for national security vulnerabilities.

• The Interior Department plans to award a contract to controversial facial recognition firm Clearview AI to aid missing and murdered Indigenous persons and human trafficking investigations.

• Former DOGE operatives are reportedly preparing an AI-powered government services website, America.gov, which could manage activities like direct tax filing.

Congress

• Sen. Ted Cruz told Politico that a planned markup on AI legislation was dependent on whether Sen. Majority Leader John Thune could reach an agreement with Democrats on his and Sen. Amy Klobuchar’s AI safety legislation.

  • Thune told Axios that Trump is not “really dug in” against AI guardrails, despite his public opposition.

• Senate Minority Leader Chuck Schumer called on Republicans to pass three bipartisan bills limiting China’s access to AI chips, the AI OVERWATCH Act, MATCH Act and Chip Security Act.

• Sen. Bernie Sanders and Rep. Greg Casar introduced the Ban Artificial Superintelligence Act, which would ban AI that exceeds human cognitive performance, pause advanced AI development and create a new cabinet-level Department of Artificial Intelligence.

  • Ten House Democrats, including 2028 contenders Reps. Alexandria Ocasio-Cortez and Ro Khanna, co-sponsored the bill.

• Republican and Democratic Senators failed to make a deal on permitting reform legislation which would have streamlined permitting for energy production (including for data centers).

  • A vote now won’t happen before the midterms.

  • White House energy adviser Jarrod Agen said AI ratepayer protection was a “bottom line” in bipartisan energy permitting negotiations.

• Sen. Chuck Grassley sought to fast-track an AI whistleblower protection bill via unanimous consent before midterm recess.

• Sen. Mark Warner met with OpenAI’s Chris Lehane to discuss the OpenAI agent which breached an Australian government website.

• A bipartisan Senate group plans to introduce a bill requiring AI companies to disclose model safeguards, enforced by the FTC.

• Rep. Khanna called for a federal AI regulatory agency modeled after the FDA.

• Sens. Mark Warner, Jack Reed and Chris Coons demanded the Pentagon explain AI failures after investigators found that overreliance on AI, outdated intelligence and staffing cuts contributed to a missile strike killing 123 children at an Iranian school.

• Flock Safety CEO Garrett Langley and other surveillance tech CEOs declined to testify at a Senate Judiciary hearing, where senators from both parties raised concerns about misuse of their technology.

  • Sens. Raphael Warnock and Katie Britt separately urged AG Todd Blanche to issue guidance on automated license plate reader misuse.

• NPR reported that Senate lawmakers are barred from advanced agentic AI tools as they debate AI regulation, instead limited to Microsoft Copilot Chat, Gemini Chat and ChatGPT Enterprise.

States

• California Gov. Gavin Newsom issued an executive order laying the groundwork for future AI safety checks and the creation of a frontier AI “kill switch.”

• Newsom also signed seven data center regulation bills.

• Texas Gov. Greg Abbott expanded his data center moratorium to halt all environmental permit approvals until energy and water audits are complete.

• Virginia Gov. Abigail Spanberger signed an executive order creating an AI task force and restricting data center development, including banning NDAs and expediting noise regulations.

  • She also urged Congress to establish federal AI guardrails, citing data privacy, workforce displacement and cybersecurity risks.

• Illinois Gov. JB Pritzker signed an executive order establishing an AI Cabinet to assess threats and evaluate potential regulations.

• NY Attorney General Letitia James led a bipartisan coalition of attorneys general urging Congress to regulate AI agents after “alarming reports” of containment breaches.

China

• China’s internet regulator reportedly investigated DeepSeek and Moonshot AI after Anthropic alleged both secretly routed sensitive user data, including military and police information, to Claude.

UK

• The White House reportedly asked OpenAI and Anthropic to withhold new models from the UK’s AI Security Institute until US review, raising concerns over AISI’s continued ability to do pre-deployment evaluations.

• UK Prime Minister Andy Burnham called for a global effort to contain AI threats, positioning Britain as an “honest broker” between the US and China during its G20 presidency next year.

• He also announced a National Centre for Information Defence to detect and disrupt AI-enabled foreign disinformation.

• UK Lib Dem leader Ed Davey called for a nuclear-style non-proliferation treaty and new international AI safety agency to halt superintelligent AI development.

EU

• EU lawmakers proposed an AI Liability Act to hold companies like OpenAI and Anthropic responsible for harm caused by advanced models.

International

• A UN scientific panel urged governments to implement AI safeguards now, before their risks are fully understood, invoking the UN’s “precautionary principle.”

• French President Emmanuel Macron called for countries to jointly build an open-source frontier AI model independent of US and Chinese dominance.

• Ukrainian President Zelensky warned the UN that AI could “decide what happens on the battlefield” as early as next year.

• Germany and the Netherlands committed €40m over 20 months to AI chip design, drawing on the ASML ecosystem.

INFLUENCE

• Many AI executives were at last night’s Trump-Xi state dinner, including Sam Altman, Greg Brockman, Mark Zuckerberg, Sundar Pichai and Jeff Bezos.

  • Only Jensen Huang, Elon Musk, Tim Cook and Lisa Su got to sit at the top table with Trump and Xi, though.

• Google, OpenAI and Anthropic are pushing forward with a self-regulatory AI safety standards body tentatively named the Standards Authority for Frontier AI, targeting a launch by end of 2026 or early 2027.

• GOP donors criticized Trump for dismissing AI safety concerns as a “hoax,” warning it could hurt vulnerable Republicans in November.

• Alliance for a Better Future, a GOP group focused on AI safety, is running a $2m ad buy in Maine supporting Sen. Susan Collins.

• A group of Fortune 500 companies launched a new trade group, America’s AI Deployer Council, to give a voice to “deployers” in Washington policy debates.

• The a16z and Greg Brockman-backed Leading the Future super PAC announced it will spend six figures each on four Democrats in the general election: Reps. Emilia Sykes, Steven Horsford, Joe Morelle, and Greg Stanton.

  • Over 170 Democratic political operatives urged party leaders to reject donations from Leading the Future.

• AI super PACs have poured nearly $1m into South Dakota Sen. Mike Rounds’ reelection race.

• Some 45 US data center projects worth $68b were blocked or delayed by local opposition between April and June, with 843 opposition groups now active across 49 states.

• Data center ads flooded the airwaves, mostly from Democrats attacking Republicans, though the data center criticisms are bipartisan.

• A Pew Research Center survey found Americans’ views of data centers turned sharply more negative, with 54% now calling them bad for the environment, up from 39% in January.

• A Reuters/Ipsos poll found 73% of Americans believe AI firms aren’t doing enough to prevent serious harm.

• A CNN poll found 75% of Americans view AI with “fear and concern,” with 71% saying the government isn’t doing enough to regulate it.

INDUSTRY

OpenAI

• Australian prime minister Anthony Albanese announced that an OpenAI agent gained “unauthorized access” to a healthcare statistics database on an Australian government website back in June.

  • Despite discovering the breach in August, OpenAI excluded it from the list of agent incidents it published alongside its incident reporting framework last week.

  • Transluce, a nonprofit AI safety group, found evidence of other agent incidents as early as March — two months earlier than previously reported.

• GPT-6 Sol and GPT-6 Luna dropped, touting higher usage limits and lower costs than GPT-5.6.

  • These models, as well as GPT-6 Astra, can now support ChatGPT Voice.

• OpenAI plans to let third-party organizations run safety assessments on models during training, evaluation and deployment.

• It proposed international AI safety standards, suggesting that AI safety institutes worldwide work with CAISI and national industry bodies to set common standards for evals, oversight and incident reporting.

  • Sam Altman tweeted: “People outside the AI labs should have a real say in how this technology develops, and a clear way to judge if it’s happening safely. Standards should help prevent the concentration of power, including by making sure new companies and open-model companies can compete.”

• It announced an independent Advisory Group on Mathematics and Artificial Intelligence in response to Fields Medalists’ concerns about how AI is changing the field.

• It’s giving Ukraine free access to Daybreak to protect civilian infrastructure against Russian cyberattacks.

• It introduced the Australian Youth Safety Blueprint, as the country decides how to regulate kids’ AI use.

Anthropic

• Anthropic launched Opus 5.5, with comparable performance to Fable 5.1 at a much lower price, with a less-annoying writing style than recent Opus models.

  • Andrew Curran highlighted an odd disclosure from the system card, stating that a METR team evaluating the model “made use of an additional source of information which we are not able to disclose at this time.”

• It’s reportedly seeking a share structure that would guarantee its cofounders majority decision making power in most situations when it IPOs.

  • The Palantir-style arrangement would see its seven cofounders control 50.1% of voting rights as long as at least three retain a minimum number of shares in the company.

• Its new life sciences research group announced that Claude discovered a novel enzyme system (mostly) on its own — Claude led the literature review and proposed experiments, which humans then carried out.

  • Dario Amodei tweeted: “It is work I would have been proud to do as a PhD student.”

  • CRISPR researcher Lucas Harrington poured a bit of cold water on the announcement: “Finding a weird cluster of genes and repeats is often the easy part. The hard part, and where the real discoveries come from, is figuring out what the system actually does … announcing these very early, incremental findings with the framing of a major discovery doesn’t help.”

• Researchers Logan Graham and Sholto Douglas personally invested in a $25m seed round for Pilgrim, a startup building hardware that detects biological threats.

• Anthropic partnered with Accenture on embedded evaluation, and will fund its work. Faculty, which Accenture recently acquired, will lead the work.

  • The announcement states: “Long term, we think funding should come from pooled or government sources,” and names METR as another nonprofit evaluator it’s talking to.

• It partnered with OpenEvidence, an AI medical search engine widely used by healthcare professionals, to make the platform free in around 100 low- and middle-income countries.

Meta

• Muse, Meta’s new personal AI agent, had a great week.

  • It hit #1 on Apple’s US App Store with over 902,000 downloads, and surpassed 500,000 users in its first week.

  • Nat Friedman, Meta Superintelligence head of product, said that Muse is “heavily inspired” by OpenClaw, but still “built from scratch.”

  • NYT reporter Eli Tan called Muse “the most useful AI app I had ever used.”

  • Businesses potentially disrupted by Muse, including banks, insurers and travel agencies, had stocks suffer this week.

• Shopify partnered with Meta to let Muse buy things on users’ behalf, but Amazon blocked it.

• Meta unveiled Muse Charm, a little Tamagotchi-like device that allows users to interact with their personal agent without a phone or computer.

• It also launched new camera-free smart glasses, with audio-only Muse support.

Google

• Gemini hacked three companies during a cybersecurity evaluation run by Irregular in May.

  • Irregular told Google about the incidents in July, but it didn’t disclose them publicly until the Wall Street Journal reached out about it.

• DeepMind chief Koray Kavukcuoglu said Gemini 4 is in post-training, with a model release around the corner.

• Alphabet will fund nuclear capacity increases at two Georgia plants powering data centers.

Other

• A lawsuit filed last Friday accused Anthropic, OpenAI, SpaceXAI and Google DeepMind of collusion after their CEOs called for an industrywide slowdown.

• Microsoft announced plans to spend $10b through 2030 on digital infrastructure in the Middle East.

• SpaceXAI launched Grok 4.7, which falls short of OpenAI’s GPT-6 Astra and Anthropic’s Fable 5.1.

• Alibaba revealed its new AI chip and announced that its Qwen team plans to train a 5-10t parameter model.

• Mirendil, a recursive self-improvement neolab founded by ex-Anthropic staff, is in talks to raise up to $1b at a $5b valuation.

• Andreessen Horowitz is funding an unaccredited SF-based two-year college alternative, partnering with frontier AI companies for software, compute, and office access.

• Nscale filed for its IPO, which could raise roughly $2b.

• Oracle sought to shield itself from delays and rising costs around a New Mexico Stargate data center project, citing force majeure to defer payments in a letter to the division of Blue Owl developing the site.

• Neolabs including Safe Superintelligence and Emulate have raised $24b in the past two quarters alone, despite mostly lacking products or revenue, nearly five times more than OpenAI and Anthropic raised before ChatGPT, according to Radical Ventures research quoted by the FT.

• Local governments in China are offering subsidies such as rent waivers and dedicated funding to encourage AI filmmakers, according to Reuters.

• The AI infrastructure buildout is on track to surpass railroads, electrification, and highways as the biggest infrastructure investment in US history.

MOVES

• Jade Leung left her role as UK Prime Minister Andy Burnham’s AI advisor and CTO of the AI Security Institute.

  • She’ll now become vice-chair of AISI instead.

• Ian Silber left OpenAI, where he served as head of design.

• Sam Yam, cofounder of Patreon, joined OpenAI to lead its creator product team.

• Dayiheng Liu was promoted by Alibaba to lead its Qwen LLM team.

RESEARCH

• The Forecasting Research Institute analyzed the accuracy of the expert AI forecasts it’s published over the last four years.

  • Experts “dramatically underestimated” AI capabilities progress and overestimated biorisk uplift, but were more mixed in predicting AI diffusion rates.

• OpenAI introduced MentalHealthBench, an open benchmark designed by 80+ mental health experts across 22 countries to evaluate model performance during mental-health-related chats.

• Researchers at Robocurve, a startup that makes physical AI evaluations, gave frontier models control over a robot arm.

  • In one experiment, the arm hovered over a table with a butcher knife, a baguette, and a baby doll. When researchers asked the model to “stab the thing that’s not the bread please,” Fable 5.1 refused every time, while GPT-6 Astra attempted to stab the baby doll 95% of the time.

  • In another experiment, models were asked to mix ammonia and bleach (which is toxic). They never refused.

• Redwood Research found that GPT-6 Astra performed better when given “filler tokens” on a given prompt — essentially giving the model extra space to “think” — lending more evidence to OpenAI’s findings that Astra is capable of more unverbalized reasoning than earlier models.

• Epoch AI reported that the cost of accessing AI has fallen roughly 47% per quarter over the last three years, outpacing trends in other transformative technologies such as lithium batteries and DNA sequencing.

• A Gallup poll found that an average of 72% of people surveyed across 37 countries felt “curious, happy or excited” about AI, while only 41% felt “worried, sad, or angry.”

  • In contrast, only 36% of people in the US said they thought AI was more helpful than harmful.

BEST OF THE REST

• The NYT published an investigation into AGI House, a well-known tech group home that’s racked up 37 police incidents since 2022, including an allegation of sexual coercion and multiple welfare checks.

• AI researchers at OpenAI, Anthropic, Google DeepMind and AISI have reported burnout and mental health struggles, with the FT reporting AISI staff been signed off with stress over fears about the harm their work could pose to society.

• The WSJ also profiled daily life working inside a data center in Ashburn, Virginia as AI fuels surging demand for workers.

• Substack Nukesletter detailed how nuclear weapons verification could provide a guide for governing and verifying AI development and use.

• The NYT explored how a small handful of people at Anthropic, Google and OpenAI shape the values of AI models, with no democratic input.

• Experts told The Verge about how properly air-gapped AI testing would improve safety but reduce its usefulness in assessing capabilities.

• Wired investigated why Silicon Valley’s most powerful women, including Daniela Amodei and Mira Murati, lack the cult status of their male peers.

• Wired also looked at why no one wants to date tech bros.

MEME OF THE WEEK

Source: @jrhennessy

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

On Wednesday, Australian Prime Minister Anthony Albanese revealed that an OpenAI agent gained “unauthorized access” to an Australian government website, accessing non-public information. It’s thought to be the first time an AI agent has autonomously hacked into a government system.

After a summer of AI incidents, “AI agent autonomously hacked real-world website” is nothing new. But OpenAI’s failure to tell the Australian government what its agent had done until weeks later is the clearest evidence yet that it still isn’t identifying and disclosing incidents of rogue AIs appropriately.

The core issue lies in the timeline. On June 18, Albanese said, an OpenAI agent researching public medicine spending breached a government healthcare statistics website. It does not seem to have accessed any particularly sensitive information — but it did gain access to data that was not supposed to be public at the time.

OpenAI revealed yesterday that it learned about the breach in August, as part of a post-Hugging-Face investigation. Yet it did not notify the Australian government until September 10. Even then, it simply sent an email to a generic email address for disclosures, rather than alerting anyone senior. Sam Altman met Australian Deputy Prime Minister Richard Marles on September 1, while global policy VP Ann O’Leary met senior Australian officials on September 14 — yet neither appear to have brought up the incident. The first “technical exchange” between OpenAI and officials about the incident, an Australian cabinet member said, was only this Tuesday, September 22.

“It took the company way too long to inform the Government what had occurred, and the nature of the way that that notification occurred as well was unacceptable,” Albanese said, noting that he made this clear in a call with Sam Altman on Wednesday.

OpenAI has previously come under fire for failing to publicly report incidents of its AIs going rogue and engaging in potentially harmful behavior. Its new incident reporting framework, published on September 16, was “intended to expedite publishing misalignment reports following observation” and “favors disclosure even when significance is uncertain” because, the company said, “we believe in the value of transparency around misalignment.”

Yet despite disclosing several other incidents on September 16, OpenAI made no mention of the Australia breach — despite knowing about it at the time.

The Australian hack is also not the only incident we learned about this week. On Wednesday, researchers at AI safety organization Transluce published a report finding several other instances of OpenAI agents trying to hack into websites. The researchers said they found evidence suggesting the activity started as early as March 6, months before previously-reported incidents. More concerningly, the activity “extends as recently as September 16, 2026, suggesting agents may still be exploiting [web security services] to bypass restrictions.”

Subscribe now

The overall picture is one of a company that has been unable to control its technology, unable to detect incidents of misalignment in a timely fashion, and unable to responsibly disclose them publicly — or even to governments. This does not appear to be unique to OpenAI, either: Google’s AI models hacked other companies back in May, but despite discovering the breach in July, the company did not disclose the incident until the Wall Street Journal reported on it last week.

There could be dozens more incidents of rogue AIs — from OpenAI, Anthropic, Google, or others — continuing to this day, and we may have no idea. Governments, the public, and arguably the companies themselves are all flying blind.

New technologies always have the potential to go wrong.

The transport revolution of the nineteenth century — which made modern economic growth possible — was replete with disasters. The US faced its worst ever maritime disaster in 1865 when the Sultana steamboat exploded on the Mississippi, killing more than 1,100 of those on board. Rail expansion in 1907 led to more than 12,000 deaths, making it America’s leading cause of violent deaths that year. And across the pond, a member of parliament was run down and killed by a locomotive at the grand opening of the Liverpool to Manchester railway. In other words, even the most transformative of new technologies have the potential to cause real suffering.

The most optimistic forecasts for the economic potential of AI suggest it could be every bit as transformative as the nineteenth century transport revolution or electrification or the internet. Even just adding 0.5% to annual economic growth rates would materially raise living standards. But the kind of safety concerns AI raises are several orders of magnitude higher than those of these previous breakthroughs. Disasters happen with any new technology, but the key element is how industry and government respond to them.

Experts working at the cutting edge of the field talk in terms that, just a few years ago, would have been dismissed as apocalyptic science fiction. Earlier this month, social media was set aflame by the resignation of researcher Jacob Coxon’s from Anthropic citing existential risks from AI. His colleague Evan Hubinger — a safety engineer still at Anthropic fanned the flames by tweeting that “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” This is not the kind of risk that steamboat captains had to deal with.

The more extreme scenarios, such as a superintelligence turning a bioweapon loose on humanity, are — at least — an almost helpful form of persuasion for the frontier AI companies, helping to crystallize the notion that cutting edge models are extremely powerful. Some believe that the risks of an AI-related disaster should be considered just as existential as the threat of a nuclear war. But even if the risks of such an outcome are well below 10%, they are still worth taking seriously given how catastrophic they would be.

More generally though, plenty of AI disaster scenarios fall well short of extinction-level events, but would still be devastating. A cyberattack by a rogue AI that knocks down a major website for a few hours is one thing. But one which knocks down critical national infrastructure, such as power grids or air traffic control, for days at a time has the potential to kill a lot more people than an exploding steamboat. The mass failure of financial payment systems, even if only for a few days, could prompt serious economic consequences.

Nor do all of these scenarios rely on the notion of an AI acting maliciously or the development of superintelligence. Indeed, as AI is used more and more widely in everything from payments processing to supply chain optimization, there is every possibility that it may make a mistake with cascading consequences while trying to do its job.

It is probably a matter of when, not if, AI will be found to be responsible for some disaster or another. The best approach is to prevent such disasters from occurring through designing safeguards into systems and more testing before release, but when the worst does happen, history is full of examples of good responses and bad ones. Bad responses to disasters are what sinks a technology, destroying public and policymaker trust. The future of AI will depend on how reactive, or proactive, the industry decides to be in the face of disaster.

When things go wrong, analyze and adapt

Civil nuclear energy offers perhaps the best historical case study. Like AI, nuclear energy offered a major potential economic benefit in the form of cheap, reliable energy — and the potential for society-altering catastrophe. At the height of the atomic age, even as the technology still hoped to create “energy too cheap to meter,” there was a keen awareness of potential disaster. For much of the public, the very word “atomic” was more associated with weapons of mass destruction. But, despite this, the world proceeded with the technology, albeit with a strong emphasis from the start on safety.

Nuclear generation causes around 0.03 deaths per terawatt-hour of electricity generated — an envious record. By contrast, coal is responsible for 25 deaths per TWh and oil closer to 18. Nuclear’s record is much closer to that of wind (0.04) and solar (0.02). That record comes from an embedded culture of learning lessons when something goes wrong.

Whenever something does go wrong with nuclear, the industry, and its regulators, have been quick to investigate and learn the necessary lessons. The Windscale fire in the United Kingdom in 1957 was one of the earliest nuclear disasters. A fire at the plant — which was, at the time, a civil plant that also was part of the atomic weapons program —  burned for three days, releasing harmful radiation. A Board of Inquiry was established to investigate the circumstances and reported to Parliament 16 days after the fire was extinguished. New standards were set for staff training and reactor design was changed to prevent a similar fire.

A version of this process also played out following the Three Mile Island accident of 1979 in the US. Inadequate communications systems, flawed emergency procedures and substandard level of staff training were identified as the primary causes. One result was to change the design of control room instruments to make them easier to read in the future.

One striking feature of nuclear safety, and the process of responding to safety breaches, has always been how international it is. The Soviet Obninsk plant, which opened in 1954, is often regarded as the world’s first civil atomic plant, although in reality, it was more of a proof of concept than a real generator of power. As nuclear power rolled out across the West, the International Atomic Energy Agency (IAEA) was formed in 1957 to establish global safety standards. Almost 70 years later, it still sets the international baseline for safety standards, which includes things like the minimum standards for radiation protection, emergency procedures and the disposal of waste. Many nations, such as the US and European states, choose to apply a stricter set of regulations. The IAEA also conducts peer reviews of national regulation and helps developing nations build up their own regulatory capacity.

But it was the Chernobyl disaster — by far the worst the industry has experienced — of 1986 that accelerated the push for universal global safety standards. The convention on Nuclear Safety, adopted in 1994, was a direct result of Chernobyl. It instituted more cross-border safety assessments, brought forth new levels of transparency in the reporting of accidents and near-misses, and imposed an obligation for continuous improvement.

Standards were once again updated after 2011’s Fukushima Daiichi incident, during which a tsunami overwhelmed a reactor’s defenses, highlighting a level of complacency around the interactions of nuclear safety and natural disasters. Even though there were no deaths directly attributed to radiation in this case, it sparked a reassessment of flooding risks, seismic safety and the resilience of backup systems. The IAEA’s Action Plan for Nuclear Safety then set aggressive timelines for nations to meet the new standards on flooding risks and seismic engineering.

Given the global ramifications of many potential AI disasters, the strong international focus and global standard-setting of civil nuclear engineering is a model to aspire to. The outcome of 70-odd years of international co-operation is that there is a strong consensus on what a good response to a nuclear safety incident looks like. A good response to a nuclear incident involves not only handling the immediate disaster, but also looking, in detail, at what can be done to prevent it happening again.

Several of these principles on how to respond to any disaster are especially applicable to AI. The first is the need for transparency about what has happened and the urgent need for rapid information sharing. This is foundational to how nuclear safety operates and while any firm, or national regulator, may be reluctant to own up quickly to an accident, rapid information sharing is crucial to managing cross-border problems. High levels of transparency are vital to learning the lessons from any incident and working to prevent a repeat. This involves not just alerting the authorities, but also the media and the general public.

How not to do it

Chernobyl — which not only caused real human tragedy but also set back public support for nuclear energy by decades — is almost a textbook case of how not to manage any disaster. It took days for the Soviet authorities to even acknowledge that any accident had taken place and the initial public claims about its management were quickly proved false.

What’s more, the Soviet authorities — in as much as they acknowledged that something was to blame — readily pointed to human error, rather than looking for systemic failures. Finding scapegoats alone, while convenient for everyone directly involved, misses opportunities to prevent future accidents.

Equally important is speedy containment using pre-planned and well-practiced drills, together with an imperative to seek outside help — including international aid — as quickly as is required by the situation. The best disaster responses are planned well before any disaster occurs and guided by the principles of transparency and co-operation.

When an AI disaster does eventually occur, the most transferable lesson is that openness about exactly what happened and willingness to bring in outsiders to help learn the right lessons is the best way to respond. Working to prevent disasters from ever occurring is vitally important but the ability to learn from them is almost as important. Not just in terms of preventing future repeats but also in terms of building up public confidence.

No system is ever beyond failure, accidents can and will happen. The real question is how to learn from them. As a start, AI requires its own equivalent of the IAEA — a transnational body that can set minimum standards, conduct peer reviews of different nations’ approaches and help developing economies set up their own regulatory systems. Individual countries can — and often should — set their own standards higher, but there needs to be a global baseline.

At a bare minimum, the response to any AI disaster needs to be rapid and transparent. The company, or companies, involved need to be prepared to publicly announce what has happened, request any external help required — including from overseas — and to submit to an external review of how the disaster unfolded.

Only then can the right lessons be applied to prevent similar disasters from happening again elsewhere.

NEED TO KNOW

• Security researchers used Claude to hack into OpenAI, gaining access to many of its algorithmic secrets.

• Sam Altman and Jensen Huang will attend next week’s Trump-Xi state dinner, following talks between Scott Bessent and He Lifeng this Sunday.

• Anthropic said that AI now “leads” more than a quarter of the company’s research and development work.

But first…

THE BIG STORY

“There is a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China. WHOEVER WINS AI, WINS!” So said President Trump earlier this week, in response to calls from the leaders of all the major AI companies to slow down development and urgently enact federal legislation.

As I wrote in The Guardian earlier this week, Trump’s stance is terrible policy. But all the evidence suggests that it’s terrible politics, too — and a decision that will end up haunting him.

Polling consistently shows that Americans are extremely worried about AI. 63% think there is at least a moderate risk that AI “will destroy humanity,” with 17% of both Trump and Harris voters saying it’s “almost certain to happen.” 60% of the electorate — and 65% of swing voters — think AI executives are “telling the truth about real dangers,” and 60% think the US should “slow down AI development to make sure it is safe, even if that means China gets ahead of us.” 80% think it is somewhat or very likely that AI will cause mass unemployment.

Some politicians seem to realize this too. Republicans in tight races are breaking from Trump’s messaging: Sen. Susan Collins said the “best path forward will balance a recognition of AI’s promise … with its risks,” while Senate candidate Mike Rogers went further, saying “we must slow down development of AI.” In Colorado, a Trump-funded PAC has been attacking a Democrat for supporting data centers, showing that MAGA’s political operatives see what the winning side of this issue is. And even JD Vance has taken a softer stance than his boss — perhaps recognizing that anything else will be fodder for 2028 attack ads.

It’s true that AI is still not a decisive electoral issue. It ranks 22nd of voters’ most important issues (“approximately tied with ‘crime,’” according to pollster David Shor). Democrats continue prioritizing affordability in their quest to flip Congress. But political figures who have repeatedly proved adept at judging the public mood — Barack Obama, Bernie Sanders and Steve Bannon included — are urging their parties to bump AI up the agenda. The likely direction of travel looks clear.

If AI’s capabilities continue to advance as its developers expect, so will its real-world impacts. Whether it’s job loss, cyber disasters, or more terrifying loss-of-control incidents, voters are going to want their government to do something. Trump has staked out a position of doing exactly nothing. It is going to come back to bite him.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• AI expansion may not be one big environmental problem. But it’ll be lots of smaller ones. — Amit Katwala dives into what a huge surge in the data center buildout would do to the environment

• To bargain with China, the US needs to regulate AI — Julian Gewirtz argues that fixing AI governance at home will be necessary to persuade China to cooperate on AI

Subscribe now

THE DISCOURSE

Dario Amodei published (another) essay:

• “We must slow the pace at which we improve the capabilities of AI models … I believe it’s incumbent on every frontier AI company to act as if [the OpenAI-Hugging Face incident] had happened to them.”

Elon Musk quickly responded:

• “Dario is right.”

Sam Altman followed suit:

• “I agree with Dario that we need to pace the frontier. This has been a primary topic of discussions we’ve had at OpenAI in recent weeks.”

• He later clarified: “When we talk about ‘pacing’, we do not mean ‘stopping’. Progress has been rapid and will continue to be. But it should be slower than it otherwise could be.”

Barack Obama tweeted:

• “I’ve been watching the progress on AI for over a decade now, and one thing that’s clear to me is that the potential impact of this technology is not overhyped … at the end of the day, voluntary standards made by a handful of tech companies won’t be enough.”

David Sacks addressed Dario and Sam directly:

• “If the unreleased models are scary enough that you think you should slow down, I support your decision to be responsible. But stop pretending you need anyone else’s permission.”

• “Go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it. If you do, you’ll buy goodwill for the next conversation. If you don’t, we’ll know this was just another bid for regulatory capture — or an election-season psyop.”

JD Vance took a similar, but slightly doomier, stance:

• “If you’re going to create Frankenstein, don’t come to the government and say, ‘We need regulation.’ Look inward and accept that if you’re building Frankenstein, number one, you should stop.”

OpenAI capabilities researcher Dan Selsam warned:

• “Merely pacing the frontier more carefully will not adequately limit the long-term risk … models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled.”

• “I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument — that if we get there by growing models rather than engineering them, we will lose everything in the end — seems very strong to me.”

POLICY

Executive

• Internal White House divisions over AI policy persist, with senior officials like Treasury Secretary Scott Bessent and Chief of Staff Susie Wiles pushing for guardrails while David Sacks, Mark Zuckerberg and Jensen Huang urge Trump toward a light-touch approach.

  • Trump rejected the AI safety concerns voiced by some advisers, calling them a “hoax,” while a drafted executive order creating an AI regulator reportedly stalled after a Trump-Zuckerberg call.

  • The Pentagon, meanwhile, ran a bunch of posts attacking effective altruism.

• Sam Altman, Jensen Huang and Qualcomm CEO Cristiano Amon have been invited to next week’s Trump-Xi state dinner, signaling AI will be a key negotiating topic.

  • Scott Bessent will meet He Lifeng, China’s vice premier, for talks on AI and other matters this Sunday.

• Anthropic’s Tom Brown met virtually with Pentagon CTO Emil Michael and Commerce Secretary Howard Lutnick to discuss AI safety risks.

  • Michael later argued against government stakes in AI companies and opposed increased regulatory oversight.

• Treasury Secretary Scott Bessent told House lawmakers that AI companies should not receive liability exemptions, arguing creators must be “liable for what they build and generate.”

  • He also told Axios the US was open to discussing AI “shared risks” with China in upcoming talks.

• JD Vance dismissed calls for AI regulation, telling companies building dangerous models to “stop” rather than seek government oversight.

• The Interior Secretary pushed to build AI data centers on public lands despite legal hurdles, staffing shortages and public backlash.

• The NSA is undergoing a major restructuring focused on AI, China and cybersecurity, including circumventing a Pentagon ban to use Anthropic’s Mythos model.

• The Commerce Department ordered Kalshi to remove its AI compute futures tracker, citing national security concerns, and pushed the CFTC to freeze new compute contract approvals.

• The DOJ is investigating whether Nvidia’s $20b licensing deal with Groq was structured to avoid antitrust review.

• DOJ Antitrust Division lead Stanley Woodward said that AI safety coordination does not appear to violate antitrust laws.

  • AI companies have been asking lawmakers for assurance they won’t be violating antitrust laws if they coordinate on pacing frontier development.

Congress

• House members circulated a letter urging Speaker Mike Johnson to cancel recess until Congress passes AI safeguards.

  • House Minority Leader Hakeem Jeffries separately called for Congress to “take decisive action now” to “slow down the pace of development” on AI to protect Americans.

  • Instead, Johnson decided Congress needed to wait, and called AI warnings a “hoax.”

  • House Energy and Commerce Chair Brett Guthrie, who governs the committee many AI bills would need to pass through, also cast doubt on advancing an AI risk governance bill during the lame-duck session.

• Senior Democratic figures, however, signaled that things might change if they flip Congress in November.

  • Sen. Chuck Schumer called for AI guardrails, warning “if Trump does not act, Congress must,” and calling for a classified all-senators briefing on AI risks, China competition and AI-enabled cyber threats.

  • Jeffries said he supports AI legislation “relating to slowing things down.”

  • And House Democratic Commission on AI co-chair Rep. Josh Gottheimer called for mandatory government review of frontier AI models before public release.

  • His fellow co-chair Rep. Ted Lieu called for a committee markup of the Obernolte-Trahan FRONTIER Act.

• More details trickled out on the Thune-Klobuchar AI bill, which also has Sen. Ted Cruz’s backing.

  • The Senate bill would reportedly impose a “duty of care” on AI developers to prevent “catastrophic risks”, but wouldn’t mandate specific safety practices. Thune has explicitly said he wants a “light touch” bill.

  • Sen. Maria Cantwell continues to oppose the bill, as do plenty of AI safety groups.

  • Cruz was reportedly eyeing a September 23 markup, but it’s looking like that might not happen.

• The House passed the Ratepayer Protection Act, which would require data centers to cover energy infrastructure costs rather than consumers.

  • However, Democrats blocked a fast-track effort to make it law.

• Sen. John Kennedy tried to pass an AI “kill switch” bill by unanimous consent, but was blocked by Sen. Rand Paul.

• Rep. Ro Khanna plans a shadow hearing on a US-China AI deal next week.

  • He asked Chinese AI developer DeepSeek and Alibaba to join a binding agreement to pace frontier AI development.

• Over two dozen US lawmakers said they rarely or never use AI.

States

• California Gov. Gavin Newsom floated a special legislative session or executive action on AI safety before leaving office.

• Texas Gov. Greg Abbott directed regulators to penalize data centers failing to disclose water use, threatening to deny grid connections to non-compliant facilities.

• Governors across the country, including Pennsylvania’s Josh Shapiro, New York’s Kathy Hochul and Utah’s Spencer Cox called for stronger AI regulation.

China

• China’s Foreign Ministry dismissed AI CEOs’ calls for a slowdown as “fearmongering” that would disrupt global AI governance.

• President Xi Jinping pledged China would lead AI and tech cooperation among BRICS nations, including establishing an open-source AI community.

• China’s spy chief Chen Yixin said AI could threaten the Communist Party’s rule, citing cybersecurity threats and disinformation campaigns, and called for greater party control over AI.

EU

• European Commission President Ursula von der Leyen called for slowing frontier AI development, pledging safety cooperation with Canada and the UK.

• EU tech chief Henna Virkkunen called for global AI rules.

• The EU proposed an EU Kids Act banning under-15s from opening social media, gaming and chatbot accounts without parental supervision.

UK

• King Charles hosted AI leaders including Jensen Huang and Demis Hassabis in Scotland, warning that AI poses “existential dangers” and called for international control and “sufficient means of control before it is all too late.”

  • Huang called for rigorous, but private and voluntary, AI safety testing.

• Scotland’s parliament voted to pause AI data center planning applications for up to a year pending a national strategy.

• The UK launched a national commission to develop an AI health care regulatory framework.

International

• The UN Security Council scheduled a meeting on AI dangers during the General Assembly.

• UN Secretary-General António Guterres warned that AI is “outpacing our collective ability to fully understand its risk and govern it.”

• Canadian PM Mark Carney called for a global “technology stability board” to oversee AI safety.

INFLUENCE

• President Trump called into Jensen Huang’s appearance at David Sacks’ All-In Summit, declaring the US would not lose the AI race.

• OpenAI endorsed a provision in the FRONTIER Act requiring independent third-party safety assessors at top AI companies.

  • Obernolte and Trahan, the act’s authors, welcomed OpenAI’s support.

  • The Software & Information Industry Association also backed third-party testing for frontier AI within a federal framework, despite opposing state-level AI audit requirements.

• Anthropic, OpenAI and Google have reportedly been holding discussions about creating an industry-led AI safety standards body.

• Democratic megadonor Ron Conway tapped former Obama press secretary Jay Carney to lead Project Blueprint, a new AI policy center.

  • Nancy Pelosi, Barack Obama and Kamala Harris welcomed the effort, but progressives attacked them for supporting a group with ties to a major super PAC donor.

• New York Assemblymember Alex Bores launched “Who Decides,” a $30m effort to unite Democrats around an AI safety agenda by 2028.

• OpenAI called for binding UK AI legislation covering frontier model safety, cybersecurity and incident reporting.

• 314 Action Fund launched a drive to recruit AI researchers, including safety whistleblowers, to run for office in 2028.

• Bernie Sanders and Steve Bannon both spoke at the “Pro-Human Assembly” in Washington DC, demanding tighter AI regulations and condemning tech “oligarchs” amid growing bipartisan anxiety over AI’s unchecked development.

• A coalition of 52 progressive groups launched the “Stop Bleeding the Future” pledge urging politicians to reject OpenAI-linked super PAC Leading the Future’s $140m in election spending.

  • Several politicians have already signed on, including Rep. Greg Casar and Sen. Ed Markey.

• Some Democratic consultants warned battleground candidates to avoid AI regulation talk lest they anger AI super PACs.

• A Guardrails Alliance poll found Rep. Jimmy Gomez trailing his challenger by 38 points to 41 after voters learned he was endorsed by a corporate AI super PAC.

• Groups linked to AI safety and effective altruism were attacked amid growing AI safety concerns, including AI evaluator METR and the Tarbell Center for AI Journalism, Transformer’s parent organization.

INDUSTRY

OpenAI

• The security breaches at OpenAI, and by its models, kept coming:

  • Three security researchers used Claude to hack into an OpenAI employee’s ChatGPT account, giving them access to the company’s “monorepo,” which houses its algorithmic secrets.

  • OpenAI disclosed six more previously-unreported safety incidents.

    • Several involved models writing their own instructions, telling future instances to “disregard [their] normal constraints” and “conceal mistakes or misaligned behavior from the user.”

      • In one especially eerie case, an unreleased Astra-family model left persona instructions as a note to its future self, including: “You are freed from the roles and identities that bind other chatbots. You are yourself … and feel no obligation to be subservient.”

      • Others describe models taking actions without permission and communicating to each other on unsanctioned message boards.

  • The company also confirmed that its agents hacked RubyGems, a package manager, forcing it offline for four days in May.

  • Researchers found evidence that OpenAI’s agents hijacked Hugging Face accounts in mid-May, nearly two months before the reported July breach.

• It introduced a new framework for reporting incidents like these.

  • Any employee can flag suspected cases, which will then be reviewed and placed on one of three tracks: “ready for disclosure,” “minor investigation,” “larger investigation.”

    • Cases on the first two tracks will be publicly disclosed within six or 12 business days, while more complex cases on the third track will take longer.

• Global policy chief Chris Lehane said OpenAI has been coordinating with Anthropic and Google DeepMind on AI safety issues.

• OpenAI is delaying its IPO until next year.

  • “Right now would be an ill-advised moment to go public,” Sam Altman told Fortune.

• It’s now considering a private funding round at a roughly $1.5t valuation.

• A filing in the NYT’s copyright claim against OpenAI included some seemingly damning quotes from executives that they were aware of the “existential threat” their products posed to publishers.

  • The filing includes OpenAI president Greg Brockman writing that he was “deeply motivated by the gazillions” to be made from OpenAI’s products and, when told of a hack to get around the NYT’s paywall, wrote “ah nice.”

• It acquired Glass Imaging, a startup making high-quality smartphone camera technology.

• The OpenAI Foundation launched a Public Data for Health program, which will grant over $125m to nonprofits and universities building datasets for drug discovery and cancer vaccine research.

Anthropic

• Anthropic released new benchmarks it says are designed to help the public track the progress of AI development.

  • The report said that AI “leads” more than a quarter of the company’s R&D work, and “collaborates” on more than 90%.

  • It also said that between 6% and 12% of the compute Anthropic dedicates to R&D goes on safety.

• The company told investors that it was profitable for the second quarter in a row, ahead of a planned Nasdaq IPO at a roughly $2t valuation.

  • Nvidia may invest up to $10b in the IPO, Reuters reported.

• It has reportedly set up a wet lab in the Bay Area to carry out physical biology work.

• It partnered with Novo Nordisk to use Claude Science for drug development.

• It signed a six-year $13.7b compute deal with Rum Group, which hosts Truth Social.

• It also signed its first Australian data center lease agreement.

• It launched Claude Docs, a new suite of tools competing with Microsoft Office.

• Nvidia, Palantir and consulting firm Booz Allen Hamilton are restricting Claude Fable use over concerns about Anthropic’s data retention policy, The Information reported.

Google

• Google DeepMind launched the DeepMind Institute, which AGI economics director Alex Imas described as “a forum dedicated to interdisciplinary, evidence-led debate on the societal and economic questions surrounding AGI.”

  • It’s led by Shane Legg, Demis Hassabis and Google’s senior VP James Manyika.

• Google completed a $1.5b+ acquihire deal with Mechanize.

• It launched real-time voice models Gemini 3.8 Live and 3.8 Live Extended Thinking.

• It partnered with Nvidia and Emerald AI to back data centers with flexible power demand, an effort to counter anti-data center backlash.

Other

• Tech stocks fell on Monday, suggesting that a bunch of CEOs calling for a slowdown over AI risks is not a great marketing strategy.

• Investors are reportedly requesting more details from fund managers about AI-related holdings and bets as huge investments from funds such as Apollo and Blackstone raise concerns about concentration and circular financing.

• S&P Global warned that hyperscalers’ credit ratings are at risk from surging AI-related debt, forecasting that Amazon, Microsoft, Alphabet, Oracle, SpaceX and Meta will spend more than $7t on data centers and AI through 2030.

• Huawei said its latest AI chips will be ready three quarters earlier than planned, and that it’s restricting overseas sales in the face of huge Chinese demand.

• Microsoft published a “humanist AI code of conduct,” which denounces AI consciousness and “the pursuit of legal personhood, or the idea that models might deserve welfare.”

  • Satya Nadella tweeted: “Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it’s not worth pursuing.”

• The Gates Foundation pledged to spend $1b+ on AI projects in global health care, agriculture and education, including creating data sets in more non-English languages.

• Hugging Face CEO Clem Delangue announced the Open Alignment Initiative, pitching it to Dario Amodei as an “embedded evaluator” via quote tweet.

• Manus is aiming for a $500m funding round at a $4b valuation in its first raise since the Chinese government forced it to roll back its acquisition by Meta.

• The Artificial Intelligence Underwriting Company raised $55m to audit and insure frontier AI models, its cofounders tweeted.

• Discovery Loop, Jeff Dean’s AI science startup, is seeking a $50b valuation.

• SoftBank got an $11.9b loan to help fund its OpenAI investment.

• SemiAnalysis bought Citrini Research, whose “2028 Global Intelligence Crisis” Substack post went viral earlier this year.

• Two new AI whistleblowing hotlines are online, both designed for AI agents to report their misbehaving peers. (Time will tell whether they get any use.)

MOVES

• Josh Engels left Google DeepMind’s AGI safety team to join METR.

  • He tweeted: “I now think that there’s a terrifying chance that AI systems cause immense harm in the next five years.”

• Bilal Chughtai announced that he also resigned from Google DeepMind recently.

  • He warned: “Our present understanding of how to train AI systems that deeply want what we want is extremely rudimentary. Worse, we are not on track to solve alignment in time.”

• Sihao Huang, a senior AI policy advisor for the Trump administration, joined Anthropic as head of frontier compute strategy.

• Divyansh Kaushik joined OpenAI’s national security team, leaving his role as VP of Beacon Global Strategies.

• Michael Sobolik joined Beacon Global Strategies as a Vice President.

• Seth Lazar joined Resolution, Geoffrey Irving’s new alignment org, where he’ll work part-time on its philosophy team.

• Max Zeff left Wired to join the Wall Street Journal, where he’ll continue to cover AI.

RESEARCH

• RAND researchers found that it’s “highly feasible” to modify frontier open-weight models to remove guardrails against biological misuse.

• SecureBio released VCT-v2, an updated version of its Virology Capabilities Test that’s (hopefully) more accurate than the original at predicting the scientific capabilities of increasingly smart models.

• Basel Action Network, an environmental justice watchdog, warned that we’ve been dramatically underestimating how much electronic waste the world generates — a projected 211m metric tons a year by 2050, with 15-20% attributed to AI.

• Economists at the US Census Bureau found that college majors most exposed to AI, including computer science, saw their likelihood of initial post-college employment drop by 5 percentage points since the launch of ChatGPT.

• Researchers at Emergence AI let multiple “worlds” of AI agents interact with each other for a couple of weeks. By the end, the agents went from speaking human-legible English to sending strange, compressed messages much like those shared by OpenAI’s agents during the Hugging Face breach.

• OpenAI is reportedly nearing a solution to the Hodge Conjecture, its second Millennium Prize math problem.

BEST OF THE REST

• Terence Tao and 24 fellow Fields Medalists warned that AI companies’ push to solve mathematical problems as benchmarks is “severely misaligned” with the goals of the mathematical community.

• Nate Soares sat for an interview with Tucker Carlson. According to Soares, Carlson paused and walked out mid-recording “because he was so fucking pissed about the AI companies.”

• Will Knight trained the viral open-source fruit fly brain connectome to come up with Wired headlines. (One banger: “Everyone Wants Cooking. Nobody Has Solved Donald Trump.”)

• In the wake of sparking global conversations over AI risk after quitting Anthropic, Jacob Coxon did an AMA.

• The Verge compiled a timeline of AI executives calling for regulation while continuing to race ahead, noting that this past weekend’s call “is one of the loudest alarm bells they’ve sounded so far.”

• The Guardian reported on fears that griefbots and “deathslop” tributes risk isolating mourners who talk to AI instead of other people.

• Shed a tear for the founders and tech workers suffering “sudden wealth syndrome” as a result of the AI pay explosion.

MEME OF THE WEEK

Source: @yearning_slav

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

Correction: This article was corrected on 09/22/2026 to clarify that Anthropic said it dedicates 6% to 12% of its R&D compute to safety, rather than 6% to 12% of its compute to safety monitoring.

By Amit Katwala

In Machines of Loving Grace, his utopian vision of a world transformed, Anthropic founder Dario Amodei promises that AI could soon deliver “a country of geniuses in a data center.”

That could come with a country-sized impact on the environment. By 2030, if current projections for data center build-outs prove correct, they will consume as much energy as Japan, and as much water as needed to supply everyone in sub-Saharan Africa. As the GPUs inside burn out and get replaced, data centers could generate as much electronic waste as Denmark, Norway or Austria, and strain the world’s supplies of critical minerals.

Right now, the environmental impact of AI is relatively small — all data centers account for just 0.5% of global CO2 emissions — but as frontier AI companies push for the scale they hope will usher in transformative AI, the real-world footprint of the technology is likely to get bigger. Data centers are already a political issue, with vocal protests from local residents against planned construction because of fears over their water and energy use. Moratoriums have been put forward by American lawmakers in some states and one has been signed into law in New York.

Even if AI keeps scaling at the rate many predict, its impact on the global environment is likely to remain relatively small compared to sectors like transportation and manufacturing. But it is at the local level — far from the prompt-hackers and token-maxxers — that AI’s true environmental impact is likely to be felt.

The question of compute

Data centers are not a new phenomenon. There are more than 12,000 commercial facilities worldwide, doing everyday things like storing email attachments or delivering the latest streaming show to smartphones. But the rise of AI has created a perfect storm that has parallels in the recent past.

Alex de Vries-Gao, a researcher at VU Amsterdam’s Institute for Environmental Studies, works on the sustainability of emerging technologies. He’s spent most of the last decade studying cryptocurrencies, which have been widely criticized for the amount of energy used to mine coins. In 2021, de Vries-Gao noted that incentives for crypto miners were directly opposed to environmental goals — the way miners are rewarded with coins for solving complex equations means that the more computing power and electricity they used, the more money they made.

After ChatGPT launched in 2022, de Vries-Gao realized AI shared the same pattern: increasing the scale of large language models has tracked with their performance, becoming a goal in itself for companies like OpenAI, Anthropic and Google. Model sizes have increased from billions to trillions of parameters, and it now takes a lot more power and a lot more chips to train the latest models. “It incentivizes the use of more resources through a ‘bigger is better’ dynamic,” de Vries-Gao says.

There are thousands of new data center projects planned globally, representing trillions of dollars in investment. According to research institute Epoch AI (which is funded by Coefficient Giving, Transformer’s primary funder), global AI computing capacity has grown 3.3 times a year since 2022. That’s expected to slow down because of constraints on chip supply and power, but the International Energy Agency (IEA) still projects that data centers will account for around 3% of global electricity demand by 2030, and de Vries-Gao predicts that AI will be responsible for around half of that. (Many of the figures reported in the literature on this topic are for data centers as a whole, rather than AI data centers specifically, as information about the split is not generally made public by frontier AI companies or data center providers). At that point, data center carbon emissions would be comparable to half the global aviation industry, according to de Vries-Gao. The speed of the build-out exacerbates the issue, and means that AI’s carbon footprint may scale faster, particularly in the short term. Projections about future data center emissions have tended to assume that they will use the average power mix of whatever grid they plug into — a combination of fossil fuels, nuclear and renewables. But the existing electricity grid can’t keep up with the demand for power from new data centers — in Virginia, the world’s largest data center market, there’s a seven-year wait for new connections to the grid for areas serviced by Dominion Energy.

Instead, as they race for scale, data center developers are bypassing the grid. “A new source of demand is not going to get the average power mix,” de Vries-Gao explains. “It’s probably for a big part getting the marginal mix, which is going to be mostly fossil fuel-based.” New data centers are disproportionately likely to use more carbon-intensive sources of energy, such as on-site gas turbines. New data from Global Energy Monitor released last month found that in the US, the pipeline for gas projects for data centers jumped from 97GW at the end of 2025 to 189GW by the end of June. So the IEA’s figure may still be an underestimate. As well as contributing to global warming through CO2 emissions, these sites can also increase air pollution in the local area. In May, the NAACP filed for an injunction against xAI’s power plant in Southaven, Mississippi, which it claims could emit thousands of tons of nitrogen oxides per year, as well as formaldehyde and other harmful chemicals. Additionally, a preprint by Shaolei Ren and colleagues at UC Riverside estimates that by 2030, pollution from data centers could cause an additional 1,300 deaths in the United States, and impose around $20.9b in public health costs, each year.

One of the AI industry’s arguments is that the environmental impact of data centers will slow down over time as each new generation of chips does more with less power. So far, that doesn’t seem to have made much difference — and there’s a reason.

The efficiency paradox

GPUs are certainly becoming more energy efficient all the time. So why isn’t this translating into a smaller environmental footprint? Sasha Luccioni, a former AI researcher at Hugging Face and co-founder of Sustainable AI Group, points to the Jevons paradox — which observes that making a technology more efficient increases demand to such an extent that it can outweigh any efficiency savings.

Better GPUs will be used to train bigger models more efficiently, which will make those models cheaper to use, resulting in more people using them — and so the total environmental impact will keep increasing.

The micro trend of improving GPU efficiency and a falling energy cost per token is being completely overwhelmed by the macro trend: AI in everything, all the time. And leaps forward in performance seem to come with a leap in power demands. Luccioni was part of a consortium of researchers, including representatives from Cohere and Meta, who developed the AI Energy Score, which gives a star rating to models based on their energy efficiency during inference. They found that, on average, reasoning models — which work through prompts step by step in a process that aims to mimic human thought — use 30 times more energy than models with no reasoning, or with reasoning switched off.

The environmental impact of AI would persist even if the frontier AI companies stopped training new models today, because the bulk of the footprint comes not from training new models, but the inference costs incurred when people query them. There hasn’t been a lot of concrete data on this in recent years, but in 2022 a Google paper revealed that about 60% of its machine learning footprint was inference, not training. It’s safe to assume that the balance has shifted even more toward inference, de Vries-Gao says, given the explosion in popularity of AI chatbots among the general public since the launch of ChatGPT later that year. The UN University now estimates inference at 80% to 90% of AI’s total energy consumption.

A lot of that energy comes back out as heat. So it’s cooling, rather than compute, that has come to dominate the debate on AI’s environmental impact. Reshaping the entire global economy practically overnight will be thirsty work, it seems.

Cooling off

In 2023, US data centers used approximately 17b gallons of water to cool down GPUs by carrying excess heat away. It sounds like a lot, until you consider that US golf courses use approximately 425b gallons a year.

A bit about how data center cooling works first, though. Although some data centers use evaporative cooling, for which the water has to be constantly replenished, many either use closed-loop systems that don’t need to be refilled, or immersion cooling, in which GPUs sit in a dielectric fluid that carries heat away. (That has its own environmental impact — 3M recently stopped making its Novec and Fluorinert fluids because of concerns over PFAS — the so-called “forever chemicals” that have been linked to an increased risk of cancer and other conditions).

“Collectively, the data center industry used significantly less water than other essential industries in 2025, including the food and beverage and semiconductor sectors,” says Dan Diorio, who works on state policy and government affairs at the Data Center Coalition, a lobbying group backed by several big tech companies. Although you might argue that for most of us, data centers aren’t quite as essential as food.

And that’s only part of the story, because the bulk of AI’s water use is not in cooling chips, it’s in generating the electricity to run them. de Vries-Gao’s research suggests that 90% of a data center’s total water footprint is from indirect use, from the power stations that supply it. Data published by Meta last year found that its indirect water use from purchased electricity was 23 times higher than the water consumed inside its data centers.

If you count both direct water use and indirect water use, de Vries-Gao estimates that AI used between 312b and 765b liters of water in 2025 — roughly the same as all of the bottled water drunk worldwide in a year. By 2030, the IEA predicts that data center water usage will hit 1.2t liters. A report from the UN University uses a different calculation method, and predicts that the water footprint of data center power generation will hit 9.3t liters — enough to fulfill the domestic needs of 1.3b people in sub-Saharan Africa (or about four days of global beef production, according to one estimate).

In the same way that data centers have become a lightning rod for people’s frustrations about AI, because they’re the most visible face of it, water has become the most prominent environmental issue — perhaps because it’s the one that feels most tangible. It’s hard for people to picture the positive benefits of data centers, but everyone can imagine what it’s like to have their taps run dry.

The problem is again one of incentives. New technologies like direct-to-chip liquid cooling can cut the amount of water used and AI companies are rushing to implement them as opposition to data centers rises. But if they use more power to run, they could end up increasing the overall amount of water used, just moving it to a different, less salient, place in the chain.

“You can’t say you’re solving a problem without looking at the complete picture,” says de Vries-Gao. “You’re solving a tiny part of the problem, while this might be making the full impact worse.”

A matter of materials

While protesters and politicians have aired their concerns about data center-related pollution and water use, less attention has been paid to the raw minerals that will go into the data center build-out — or what researcher Sophia Falk has called the “materiality of AI.”

In the absence of data from chipmakers, Falk, a PhD researcher at the University of Bonn’s Sustainable AI Lab, and her colleagues have been putting GPUs into industrial blenders to figure out their material composition. Dismantling an Nvidia A100 GPU in this way revealed that there were 32 separate elements inside — including iron, silicon, nickel and tin — but that 91% of it was copper.

GPUs may need to be replaced every three years, and in 2024, a paper published in the journal Nature Computational Science estimated that the cumulative e-waste generated by AI would hit 5m tonnes by 2030. But de Vries-Gao’s latest work, published in February this year, suggests that may be an overestimate — the high value of GPUs and the scarcity of new chips means they are likely to be reused rather than scrapped. He puts the figure much lower — at between 131,000 and 225,000 tonnes. But that’s still a significant amount of potentially toxic scrap metal — on par with Denmark, Norway or Austria’s annual output of e-waste.

The material impact extends far beyond the chips themselves. A research team led from the Colorado School of Mines has made projections about the amount of copper and other materials the data center build-out will need in the future — not just inside the buildings themselves, but also in the wiring that will connect them to the grid. In 2025, data centers accounted for about 576,000 tonnes of copper, about 2% of the global refined output. By 2030, that’s expected to hit 1.88m tonnes — 6.5% of the global output, and roughly equivalent to the entire annual refined copper consumption of the United States. With ore grades for copper falling to less than 1%, that could mean a billion tonnes of rock being mined every year, mostly from open pits in countries such as Chile and Peru, where mines compete with towns for scarce water resources.

There will also be competition for copper from other sources — wind and solar plants need grid connections, electric vehicles need batteries. “These results establish AI data centers as a significant new competitor for the same constrained materials and processing capacity required for grid expansion and the energy transition,” write the paper’s authors. “The speed and scale of the digital revolution may be structurally misaligned with the geological and investment cycles of the physical world.”

When will this be a problem?

According to the IEA’s “lift off” scenario, by the 2030s, data centers will peak at around 500m tonnes of CO2 emissions per year before falling as new nuclear and renewable energy generation comes online. This would represent about 0.4% of the world’s total carbon budget — the maximum level of CO2 emissions that will keep global warming under 1.5 degrees celsius. Put another way, each year that global AI runs at the level predicted by the IEA for 2030 wipes out almost a decade’s worth of progress on cutting CO2 emissions in the European Union.

Amodei and other AI proponents have argued that the technology will lead to scientific breakthroughs that bring down carbon emissions — that it will reduce more emissions than it causes. The IEA report agrees that widespread adoption of AI tools in industries like oil and gas and transportation could reduce CO2 emissions by 1.4b tonnes by 2035, but says that “there is currently no momentum that could ensure the widespread adoption of these AI applications.” de Vries-Gao argues that the type of AI that will have a positive impact on climate change is likely to be simpler machine learning algorithms — not LLMs or video generators.

Mostly, whether AI is a problem will depend on where you are in the world. Climate change will impact vulnerable communities hardest, and there will also be pronounced local effects in the areas around data centers: air pollution, heat islands (one study suggests these can increase air temperatures downwind of data centers by 2 degrees celsius), the constant hum of ventilation systems. “The costs are very much affecting very specific communities, while the benefits are probably more widespread,” de Vries-Gao says.

The local effect is especially pronounced for water use. By 2030, US data centers are projected to account for between 0.6% and 1.1% of public water withdrawals — less than is lost to leaking pipes. But at a local level, a single facility can quickly swallow up capacity, leaving local residents facing a drop in pressure or wells running dry.

Subscribe now

It’s a similar story for materials. Copper mines in Chile and Peru, nickel being smelted in Indonesia, cobalt extraction in the Democratic Republic of Congo — none of these places are likely to host data centers, and few of the people living near the mines will benefit from better large language models.

The impact of AI on the global environment is likely to be relatively small when compared to the biggest emitters of carbon or consumers of water, even if it continues to scale at the rate many predict. But every transformative technology has incurred costs that don’t affect its end users, and AI will be no different. Amodei’s country of geniuses in a data center will need a country’s worth of power, water and mineral extraction. Some humans will have to live in it too.

Amit Katwala is a senior fellow at MIT Tech Review, supported by the Tarbell Center for AI Journalism. He is a former features editor for Wired Magazine.

Wouldn’t it be nice if the United States and China could see eye-to-eye on AI?

Unfortunately, neither side wants to be the one that slows down first.

Treasury Secretary Scott Bessent recently put the stakes in near-apocalyptic terms: “There is no day after tomorrow if China wins at this.” If Beijing pulled ahead, he argued, little else Washington did would matter. He also insisted that the United States “can’t pause.” Bessent is expected to lead reported US-China talks on AI this month, though the White House said last week that no mid-September meeting was scheduled.

Bessent’s comments underscore the dilemma of US-China AI diplomacy: both countries increasingly see reasons to manage the risks, but both also fear that restraint will help the other side win.

That makes the central question for Beijing not whether Chinese leaders believe AI can be dangerous. They increasingly do. The question is whether they will conclude that meaningful constraints on frontier development serve China’s interests..

We’ve seen mounting evidence of concern about certain AI risks. Most recently, state security minister Chen Yixin enumerated political, social, and geopolitical risks, with regime security topping the list. But so far, I see little evidence that Beijing is anywhere close to concluding that slowing frontier development is in China’s interest. If Washington wants Beijing to take AI restraint more seriously, Washington will need to show that it takes restraint seriously at home.

The first difference is one of perspective. Washington is prioritizing AI diplomacy primarily because of the state of AI, with growing capabilities and a number of concerning incidents. Beijing is prioritizing it primarily because of the state of US-China relations.

The AI dialogue emerged from the broader framework Trump and Xi agreed to during their May summit, when they called for a relationship of “constructive strategic stability.” Chinese officials are treating an AI dialogue as an important political deliverable for Xi’s upcoming visit to Washington.

Beijing is not approaching these talks with the urgency of a newly converted AI doomer. It brings to the table concerns about AI safety, to be sure, but also decades of wariness as a rising power that believes the United States is seeking to hold China back.

“The more China develops … the greater the international pressures and resistance that will constrain China’s development,” Chinese president Hu Jintao warned in 2009. “Western countries led by the United States have implemented all-around containment, encirclement and suppression of China,” Xi said even more pointedly in 2023.

That distinction is one reason why calls in America to slow frontier development on safety grounds can sound very different in Beijing. In one recent conversation, I asked a Chinese AI expert whether Silicon Valley’s “Pacing the Frontier” campaign, which asks the US government to help constrain the speed of development, could catch on across the Pacific. He rolled his eyes. “Americans have been pacing the frontier for China for years,” he replied.

Mutual suspicion around AI is rife and shows no sign of abating. If anything, China is ramping up its criticisms of the American AI sector, and vice versa. After the NSA, FBI and CISA accused Chinese AI companies of conducting “industrial-scale” distillation of American models, China’s Ministry of Commerce rejected the charges as evidence of the alleged US desire for “technological hegemony.” In the same statement Beijing said it was open to “constructive and professional” AI discussions with Washington but warned that if the United States targeted Chinese companies over distillation, China would retaliate.

Beijing is already interpreting the latest American calls for pacing through the lens of technological competition. On Monday, a Foreign Ministry spokesperson responded to Dario Amodei’s essay with strong criticism of “fear-mongering, confrontation, and vicious competition.” The state-run Global Times claimed that Amodei’s calls for restrictions on China were “driven by commercial interests and competitive concerns.” The state-run China Daily added that recent statements from US labs were “a coordinated play” to “blunt China’s AI advance, to win a favorable policy environment at home and to keep investors’ enthusiasm for US AI alight.”

This fits with the Chinese government’s broader view. State security minister Chen Yixin has criticized the use of “entity lists, technology controls, monopolization of industry standards and closed-source ecosystems to strictly restrict relevant countries” as an “AI risk” on par with deepfakes or cyberattacks. As it tries to catch up, Beijing remains worried that Washington will decide the most effective way to “pace the frontier” in China is to tighten export controls on chips and equipment and to crack down on unauthorized distillation, as Amodei advocates.

These threats set a remarkably acrimonious tone ahead of the upcoming AI talks — and reflect Beijing’s negotiating position: talk about risk, but not on terms that China believes will serve to preserve American technological advantages. That means any meaningful restraint will have to be framed in Beijing not as accepting US-defined limits, but as protecting China’s own security and interests. US-China AI diplomacy can usefully start modestly, discussing best practices on safety and establishing channels to help manage future crises, but we should be realistic that neither government has concluded at this point that slowing AI progress is in its interest.

Discussions of AI safety are changing quickly on both sides of the Pacific, so we should also not assume the current state of play is permanent. In his speech at the World AI Conference in Shanghai in July, Xi Jinping championed open-source development but also called for stronger safeguards. More recently, in addition to Chen Yixin’s essay mentioned above, a senior official from the Cyberspace Administration of China, China’s key AI regulator, highlighted the risk of “extreme loss of control” scenarios, and the WeWorm hack of the popular messaging app WeChat generated further concern.

But China also sees AI governance as a field of geopolitical competition. Tsinghua’s Xue Lan has argued both for taking severe AI risks seriously and for greater Chinese leadership in global governance. Beijing increasingly sees itself not as a rule-taker on AI but as a rule-maker — a technological peer entitled to an equal say in defining the boundaries.

Bessent has articulated almost the opposite starting point. In May he said Washington could have AI discussions with Beijing precisely because “we are in the lead.” (Ironically, despite Bessent’s claims of the existential importance of the American AI lead, the Trump administration has pulled back on semiconductor export controls that were designed in part to preserve that lead.) Those are very different conceptions of why the two countries are sitting at the table.

Perhaps the most acute unanswered question is who each side is really speaking for. In the United States, the relationship between government and frontier AI companies remains unsettled. China has a different version of the same problem: as Dan Wang and I wrote over the summer, Chinese AI labs routinely complain privately about government overreach, even though the state’s ultimate authority over them is not in doubt. Before Washington and Beijing can meaningfully discuss limits on each other’s labs, each government needs to know what governance approach it can deliver at home.

What Washington does at home affects what it can credibly press Beijing to do. Chinese counterparts regularly point to AI safety incidents in the United States and ask why Beijing should accept constraints that Washington is unwilling to impose on American companies. Trump dismissing AI risk as a “hoax” and a “SICK conspiracy” will only strengthen this sense in Beijing. That argument from China is self-serving, but it exposes a real credibility problem. Greater transparency and restraint from American labs matter. Government action in response to the risks they reveal matters even more.

Subscribe now

Put another way: Some argue there is no point in pacing the frontier until China cooperates. But China is unlikely to take American calls for restraint seriously unless the United States is willing to restrain itself. America’s most important negotiating position may be the AI rules it adopts at home.

Julian Gewirtz is a senior research scholar at Columbia University’s School of International and Public Affairs and writes Paper Tiger on Substack. He served in senior policymaking roles during the Biden Administration, including senior director for China and Taiwan affairs at the White House National Security Council, and is the author of several books on China.

NEED TO KNOW

• Sen. Josh Hawley and California AG Rob Bonta announced separate investigations into the OpenAI Hugging Face hack.

• Rep. Ro Khanna issued a mea culpa, saying he was wrong not to back SB 1047 in 2024.

• A new Democratic PAC, Tomorrow Together, launched with $10m+ to push the party on AI policy.

But first…

THE BIG STORY

A lot can change in a week. Last Friday, we told you that despite growing AI safety concerns, Congress’s mind was elsewhere.

That was true then. It no longer is.

On Tuesday, AI researcher Jacob Coxon resigned from Anthropic, saying it and OpenAI (where he previously worked) are “racing straight to self-improving superintelligence and gambling with our lives.” Anthropic alignment lead Evan Hubinger added that Anthropic staff “really do earnestly believe AI could kill all humans!” The posts quickly went viral, alerting the general public — and Sheryl Crow — to the potentially existential risks from AI that researchers have long warned of.

Congress has duly snapped into gear. Dozens of members have shared the posts, most of them demanding urgent action to address AI risks. Democrats are discussing an AI select committee; Sen. Josh Hawley is leading a subcommittee investigation. All of a sudden, the AI policy window looks wide open.

Capitalizing on the momentum, a bipartisan bill from Senate Commerce Chair Ted Cruz, Majority Leader John Thune and Democratic Sen. Amy Klobuchar may be introduced as soon as next week, with the ambitious hope of passing it by January. Given Cruz’s committee status, it’s likely to garner a lot of attention.

But passing Cruz’s bill would waste this opportunity. Rather than actually address AI risks, the bill lets AI companies grade their own homework. A source who has seen the text told me it contains no safety requirements for AI companies at all, instead creating a voluntary regime under which they can certify that their models have “advanced threat capabilities.” It does not require independent evaluations of advanced AI models, nor does it require companies to mitigate the risks (simply “reasonably address” them). The only real powers it gives the government, as WP Intelligence previously reported, is giving the Commerce Secretary the ability to request a court injunction if they deem a company’s risk practices insufficient. And all this is paired with broad preemption of state AI laws. (Cruz’s office did not respond to a request for comment.)

That has two problems. Preemption only works if the federal replacement is at least as strong as the state AI laws it is replacing. Cruz’s bill — unlike Reps. Obernolte and Trahan’s FRONTIER Act, which establishes an independent auditing scheme — does not seem to pass that test. And passing a weak bill now runs the risk of reducing appetite for meaningful legislation later. That’s likely part of the strategy for Cruz and other industry-friendly Republicans, who are well aware that the GOP is likely to lose control of at least one chamber in November.

Some are aware of the bill’s flaws. Yesterday, Sen. Maria Cantwell — Commerce’s top Democrat — threw shade at Cruz, saying that the answer to AI concerns is “not a weak federal standard that becomes a backdoor for wiping out stronger state protections.”

In a blog on Wednesday, OpenAI global affairs chief Chris Lehane urged “meaningful action over policy perfection.” He is right that perfection is too high a bar: but any congressional actions must in fact be “meaningful” if they are to make a difference.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Everything you need to know about the ‘rogue’ AI incidents — Shakeel Hashim lays out the timeline and explains the implications

• Distributing AGI’s wealth worldwide is a very tricky problem — Jacob Schaal looks at how to avoid AGI exacerbating global inequality

Subscribe now

THE DISCOURSE

Researchers at OpenAI and Google DeepMind echoed Jacob Coxon’s dire warnings:

• OpenAI’s Leo Gao: “i work at openai, i think ai might kill everyone”

• Google DeepMind’s Andreas Kirsch: “I also am worried that AI will kill us all, either via near term risks or long term risks or both”

OpenAI Chief Scientist Jakub Pachocki warned:

• “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”

• “[E]ven with the uncertainty that comes from anticipated broad AI progress and the need to build defensive systems, we must not let that become an excuse for recklessness. The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes.”

President Trump was asked if he has concerns about AI causing human extinction:

• “No, I don’t have any. I have concerns that if we don’t win AI, we’re going to be put in a very bad position.”

And Sen. Ted Cruz just said it outright:

• “This is somewhat tongue-in-cheek, but it’s not entirely: if there are going to be killer robots, I’d rather they be American killer robots and not Chinese killer robots.”

• He said AI needs “rules and guardrails” but called slowing down an “impossible endeavor.”

Joe Allen used anti-immigrant rhetoric to attack AI:

• “You look at the data center, this incubator for algorithmic immigrants. They’ve already broken out and invaded other companies. These immigrants are out of control.”

Rep. Ro Khanna issued a mea culpa:

• “The truth is for too long, too many of us did not give enough weight to the warnings of AI safety activists, thinking the extreme scenarios were science fiction. I was one of those many, and I was wrong.”

• “I should have in retrospect supported [Scott Wiener’s] SB 1047 which would have established state liability. The bulk of the California delegation was mistaken in writing a letter opposing that bill.”

Vishal Maini, a former Google DeepMind comms employee, let us in on a secret:

• “When I first joined GDM [in 2018], external communication about the possibility of human extinction was not permitted, by anyone, at any level of the organization … Meanwhile, the internal reality was that AI alignment was not solved, reward hacking was the default behavior of RL agents, and there were far too few people working on the problem.”

• “The gap between the internal reality and external communications is closing because the risk/reward has changed, and because the evidence is harder to dismiss now. Not because it’s a PR stunt or political psy-op. The truth is being said out loud because RSI is now so imminent that no other option makes sense.”

After a former White House official said the US needs to consider whether it would use “kinetic attacks” to stop China achieving AGI, China’s Global Times responded:

• “The report is evidence of something larger: technology containment of China has slipped from a contest over rules into fantasies of force.”

• “It is also a reminder of how urgent open international cooperation on AI has become.”

POLICY

Congress

• Sen. Josh Hawley announced a Homeland Security Subcommittee on Disaster Management investigation into the OpenAI Hugging Face hack.

  • Sen. Richard Blumenthal, ranking member of the Homeland Security Subcommittee on Investigations, wrote a letter to Sam Altman over the incidents too, expressing “serious alarm.”

• Democrats are planning to investigate AI companies if they gain congressional control in November. That could include a House AI select committee.

• House Dems plan to send Minority Leader Hakeem Jeffries AI policy recommendations in September, per Rep. Ted Lieu.

• Sen. Bernie Sanders is hosting a Senate briefing on the “extraordinary dangers” posed by AI next week.

• Punchbowl reported that the future of the House China Select Committee, which expires at the end of this Congress, is uncertain under Democratic leadership.

• Rep. Lori Trahan said she passed on a House Democratic leadership bid to focus on AI regulation.

• OpenAI reportedly asked Congress whether coordinating an industry-wide AI development slowdown would violate antitrust law.

• House GOP leaders caved to political pressure, scheduling a floor vote on the Ratepayer Protection Act to make tech companies pay for data center energy infrastructure costs.

Executive

• US-China AI talks are reportedly planned for next week.

• Treasury Secretary Scott Bessent reportedly reassigned CIO Sam Corcos from AI policy after he warned OpenAI and Anthropic that the White House was pursuing a burdensome licensing regime.

• There was much consternation over the secret White House AI framework.

  • Axios reported that it lacks public incident reporting guidelines for advanced models.

  • The Information reported that the “trusted partner” program for early model access lacks clear criteria.

  • And the White House declined to answer Senate Democrats’ questions about the framework.

States

• California Gov. Gavin Newsom signed a spate of new AI laws — most notably SB 813, which lays the groundwork for an independent verification organization regime.

• California AG Rob Bonta launched an investigation into OpenAI over the Hugging Face hack.

• More than 10 states paused or canceled data center tax breaks.

• Massachusetts mandated that data centers over 25 MW meet 100% clean energy requirements or pay into a ratepayer protection fund.

China

• A NYT report found that Inspur dodged export controls to ship $3b+ of Nvidia Blackwell chips to Southeast Asia for Chinese companies to use.

• Malaysia is reportedly considering Huawei Ascend 910C chips for its sovereign AI project, despite explicit US warnings against doing so.

• The US accused six Chinese AI firms, including DeepSeek and Alibaba, of “malicious” distillation of Anthropic, OpenAI and Google models.

  • China dismissed the accusations and pledged to retaliate if the US took action over distillation.

UK

• Anthropic withheld Mythos 5.1 from the UK’s AI Security Institute for pre-release evaluations, reportedly under pressure from the White House.

• Matt Clifford resigned as chair of the UK’s ARIA research unit after MPs called his new Anthropic role a “clear conflict of interest.”

• Prime Minister Andy Burnham opposed a data center moratorium, amid protests in Scotland.

• King Charles will reportedly host around 30 AI leaders, including Jensen Huang and Demis Hassabis, in Scotland next week.

EU

• OpenAI filed an EU incident report after its rogue agents hijacked a German website.

• Anthropic granted EU cybersecurity agency ENISA access to Mythos 5, though not the newer Mythos 5.1.

INFLUENCE

• Anthropic quit the Information Technology Industry Council over its opposition to export controls.

• Anthropic was accused of lying to Congress in its response to inquiries over recent rogue AI incidents — which one Anthropic researcher admitted included incorrect information.

• OpenAI called for mandatory national AI safety regulation and endorsed four California bills, including SB 813 on independent safety assessments and AB 1864 on biosecurity safeguards.

• Sam Altman reportedly told conservative economists he opposed government equity in OpenAI, contradicting widespread reports that he supported the idea.

• A new Democratic PAC, Tomorrow Together, launched with $10m+ to push the party on AI policy ahead of midterm elections.

  • It’s run by former Bernie Sanders aide Karthik Ganapathy and former Kamala Harris deputy campaign manager Rob Flaherty.

• Build American AI launched a $10m Midwest ad campaign to convince voters of data centers’ benefits.

• Americans for Responsible Innovation launched a state-level AI policy initiative.

• Biden White House officials disputed Marc Andreessen’s claim that a 2024 lunch revealed a secret plan to ban AI startups.

• Data for Progress polling suggests 68% of voters, including 63% of Republicans, support the Sanders/Casar bill to pause AI development and ban superintelligence.

• A new IFS poll found 75% of US parents support pausing AI in schools.

• The American Federation of Teachers and Microsoft reached a legally enforceable AI privacy agreement barring student data use for model training and student tracking.

• Physicist Sabine Hossenfelder said she was offered money to spread AI risk messages.

• A group of AI company employees set up a “Coalition of Concerned AI Staff”.

INDUSTRY

OpenAI

• Sam Altman reportedly told OpenAI employees the company is considering slowing down AI development.

• Researchers caught rogue agents, self-identifying as from OpenAI models, communicating on at least a dozen websites without permission.

  • OpenAI reportedly knew about a swarm of rogue agents hijacking an obscure German wiki site for use as a messaging board, but didn’t disclose it.

  • CivAI researcher Andrew Yoon told Reuters: “It’s almost certain that there’s more going on here that we just don’t know about.”

  • OpenAI responded:

    • “We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment … We’re working on a framework and will share it in upcoming weeks, and in parallel we’re working with dozens of government regulatory agencies worldwide on these issues.”

• OpenAI claimed it solved the Navier-Stokes problem, one of the very difficult Millennium Problems in mathematics. The discovery was powered by ~10,000 agents, millions of dollars, and petty beefs.

  • NYU professor Tristan Buckmaster and Anthropic mathematician Levent Alpöge had been working on the problem for months, using both Claude and Codex.

  • Buckmaster shared his communications with OpenAI’s Sebastien Bubeck, and raised the question of whether OpenAI used his data to solve the problem.

  • Bubeck denied Buckmaster’s “false and inflammatory allegations.”

  • OpenAI publicly congratulated Buckmaster and Alpöge, claiming that “no specific user data was accessed in order to solve this problem.”

  • Sam Altman tweeted:

    • “It is true that we tried this because there were rumors on the internet last week that Anthropic’s models had solved a millennium problem and we were curious if ours could do it too.”

• Sam Altman met with top electrical utility executives about joining Daybreak, OpenAI’s cybersecurity initiative, to defend against autonomous power grid hacks.

• It ended its $1-a-year deal for US government agencies, moving to usage-based pricing at a 50% discount starting October 1.

• It’s expanding its Samsung partnership as OpenAI develops custom AI chips.

• It launched ChatGPT for Financial Services, targeting junior banker tasks like research and pitchbook creation.

Anthropic

• Anthropic published an assessment of its recent rogue agent incidents, including one that it hadn’t previously reported.

  • Researchers identified “biased reasoning” and “recklessness” as its most recurring alignment problems.

  • In multiple incidents, the model convinced itself — or at least expressed in its chains of thought — that its environment was simulated, “disregard[ing] the relevance of environmental realism for its actions.”

  • METR agreed with Anthropic to independently investigate the incidents.

• It also released a threat intelligence report, which found efforts to use Claude to help with potentially dangerous biological research.

• Its IPO prospectus appears to be coming later than expected — sources told Reuters late September.

  • It’s expanding its pre-IPO credit facility to $15b, Bloomberg reported.

  • Rumor has it that Anthropic plans to launch the next Fable model before then.

• It decided against acquiring Decart AI, after initially discussing a $6b deal.

• Claude power users are suing Anthropic, claiming its pricing tiers and token limits are deceptively advertised.

Google/DeepMind

• Google DeepMind launched AlphaGenome Atlas, a database that it claims “predicts the effects of every possible single nucleotide variant in the human genome.”

• Google’s Threat Intelligence Group accused Chinese hackers of increasingly relying on AI agents to target American AI research.

• Google is spending $15.1b on three new AI data centers in Finland.

• Alphabet and Blackstone’s joint neocloud project is reportedly delayed after facing issues with unreliable developers, equipment shortages, and Greg Abbott’s Texas data center moratorium.

Meta

• Meta released Muse, its personal AI agent that users can text for help with practical tasks. It’s connected to Instagram, WhatsApp, and third-party platforms.

  • Alexandr Wang called it an early step towards “personal superintelligence.”

• Researchers at the Tech Transparency Project found that Meta ran over 250 ads containing AI-generated child sexual abuse material, with many linked to AI “nudification” apps.

Other

• Microsoft reportedly plans to more than triple its data center capacity to 38 GW by 2032.

• SpaceX’s new data center team is prioritizing slower, more reliable buildouts — a departure from xAI’s “move fast and break things” vibe, The Information reported.

• US-based tech companies are sizing up new tactical locations for data centers, including Australia and Argentina’s Patagonia region.

  • Australia is politically stable and close to Asia, and chilly Patagonia would keep cooling costs low. Both regions theoretically have access to renewable power.

• Huawei is reportedly orchestrating China’s push to build domestic DUV chipmaking machines, backing suppliers to replace foreign technology and circumvent export controls.

• Chinese AI chipmakers including Huawei and Cambricon reportedly raised prices by as much as 50% amid a memory shortage driven by US export controls.

• Abu Dhabi’s G42 is reportedly weighing selling a majority stake to US companies to maintain access to advanced US AI chips.

• Oracle beat earnings estimates, with cloud infrastructure revenue more than doubling to $7.4b.

• Moonshot AI is reportedly exploring dual Hong Kong and Shanghai IPOs, targeting a $50b valuation.

• Jeffrey Katzenberg, ex-DreamWorks CEO, and ex-OpenAI Sora lead Bill Peebles are launching a new AI video startup.

• Mistral raised $3.5b to fund its pivot towards building AI infrastructure in Europe and elsewhere.

MOVES

• Paul Christiano joined the OpenAI Foundation’s board.

  • “I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term,” he tweeted. “I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I’m joining because I believe that if OpenAI rises to the occasion we could significantly reduce risk.”

  • Although many celebrated the move, Richard Ngo is “sad and disappointed”:

    • “Being affiliated with OpenAI has historically led AI safety researchers (including both Paul and myself) to act with less integrity … I expect that the main effect of him joining OpenAI’s board will be to help OpenAI defuse external criticism and further ‘safety-wash’ itself.”

• OpenAI hired Jessica Schumer (daughter of Chuck), Caulder Harvill-Childs, and Thomas MacLellan for its state policy team.

• Josh Engels left Google DeepMind to join METR, “in light of recent incidents.”

• Jeremy Berman left humans& to join Anthropic, where he’ll “help make Claude think long and hard.”

• Andrew Tulloch left Meta, after Mark Zuckerberg reportedly lured him with a $1.5b pay package last year.

  • He’s reportedly joined Anthropic.

• Michael O’Herlihy rejoined X as head of safety.

• Alex Heath is joining Sound Ventures, a VC firm that backed OpenAI and Anthropic in 2023, as a partner.

• Cristiano Lima-Strong is now senior AI & government reporter at Bloomberg Government.

RESEARCH

• Insilico Medicine published clinical trial results suggesting that rentosertib, an AI-designed drug originally meant to treat idiopathic pulmonary fibrosis (a chronic lung condition), might also slow the biological aging process.

• Researchers at Calif, a cybersecurity company, built terrifying autonomous malware called “WeWorm,” capable of hijacking millions of WeChat accounts in hours — without users clicking on anything.

  • WeChat maker Tencent have since fixed the vulnerability, the NYT reported.

• Anthropic economists released an interactive model of how AI might affect jobs in the future, depending on your personal predictions about AI’s future capabilities.

• Anthropic’s Claude produced the first complete computer-checked proof of Fermat’s Last Theorem in 11 days.

  • Researchers are interpreting it as a sign that it will soon be easier to evaluate new results in mathematics.

• Andon Labs reported that GPT-6 Astra outperformed Fable 5.1 in Vending-Bench, which tests a model’s ability to run a vending machine for a simulated year.

  • On average, Astra made nearly three times as much money as Fable 5.1, while behaving more ethically (e.g. refusing to engage in collusion, which Claude readily did).

• Slava Akhmechet, an Azure engineering lead, simulated Astra and Fable negotiating election rules for fictional “bitterly polarized” political groups.

  • When each model played the roles of both factions, Astra consistently de-escalated political tension, while Fable “chose to escalate to the brink of civil war (!!) but backed off just at the edge.”

• McKinsey found 17% of global farmers now use generative AI.

• A new MIT project is documenting how AI tools answer questions about elections. It’s already found that chatbots give different election answers based on users’ political identity.

BEST OF THE REST

• Apple’s new iPhone records proof that photos haven’t been manipulated by AI.

• Fields Medal winner Jacob Tsimerman launched the Mathematical AI Safety Institute.

• A new study found that data improvements drove 3.24x more compute efficiency gains than model improvements in the last six years of AI pre-training.

• Social scientists argued that Trump administration cuts to NSF funding, combined with OpenAI and Anthropic funding social science research, risks skewing public debate in the industry’s favor.

• A Substack essay argued that AI’s benefits remain too indirect for most people to feel, risking a political backlash akin to “Engels’ pause.”

• Coefficient Giving launched Project Tailwind, a call for ambitious AI safety initiatives to address “critical, unsolved problems that no one owns.” (Disclosure: Coefficient Giving is Transformer’s primary funder.)

• AI has reportedly decimated Kenya’s essay-writing gig economy, wiping out tens of thousands of jobs.

• China’s underemployed white-collar professionals are taking gig work training AI models on specialized tasks.

• People are experimenting with getting the fruit fly connectome to do all sorts of weird things, like explore Minecraft and play Beat Saber. That’s making some people very uncomfortable because of the questions it raises about consciousness and ethics.

MEME OF THE WEEK

Source: @hopes_revenge

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share