Half the world is upset about data centers, and the other half is confused about them. That’s because the case against data centers, as it is usually expressed, is full of false and easily debunked claims.

Data centers, we are told, use up the planet’s water; burn many times the energy of a Google search with every AI reply they process; raise the temperature of the land around them by nine degrees Celsius. All of this is either wildly misleading or untrue. Data centers use far less water than golf courses, AI queries now often use the same energy as Google ones, and as for nine degrees Celsius, that’s more than an open-air nuclear reactor would. Plenty has been written about this now, in Transformer, and perhaps most comprehensively by the writer Andy Masley. (Masley recently received funding from Coefficient Giving, which also funds Transformer.) Yet the absurdities persist.

And in that sticky appeal there is a lesson; underneath hysterical claims there is always a valuable emotional signal. This signal turns out to be not so hard to find, and neither is it really that strange. The truth is that the data center is a symbol.

AI is meant to change the world. Even the future in which everything works out OK — a big assumption — is a future that involves more change than most people alive (at least in developed nations) have seen. On top of that, daily AI debate revolves implicitly or explicitly around the possibility that the essential uniqueness of human life as we know it is ending or was never true. These features mean AI as an issue holds far more potential emotional energy than most politics or policy matters. In this respect, it is much more like, say, immigration than other issues of tech or policy. It is about who — or what — is in the circle of meaning. Steve Bannon-adjacent anti-AI campaigner Joe Allen was quoted telling a crowd in Wired’s recent profile: “For anyone in the room with blood pulsing in their veins, anyone in the room with fire in their minds and hearts, you have to remember, in a machine age, humans first.”

Yet AI is also enormously abstract. It right now exists solely in the computer. For almost all people it is a few lines of text in their phone. For the small minority of early adopters, it is a few more lines of code. There is an enormous gap between the intensity of “AI” as a matter of discourse or contemplation and AI as a daily object. This makes it very much unlike immigration, to which faces can always be attached (and vitriol directed).

That excess energy needs to go somewhere. Right now, it is going into data centers. It almost has to. Where else could it go? The Claude logo? Data centers are the main and possibly only available symbol that could accommodate anything close. They are real, existing objects. They are massive. They affect the landscape, the physical world. Like immigrants, they can be portrayed as interlopers on someone’s land. And they can be filmed and photographed.

That’s the most important part. Humans are visual creatures and are now more visual than in a very long time, maybe ever. They hardly read anymore; they watch. It would be only a little polemical to say that a large part of their political participation now is watching TikTok and getting upset.

And the upset over data centers is increasingly palpable. Only this week Heatmap published new polling showing the rapid shift in opinion against them, with a full three quarters saying they would oppose one built near where they lived, a swing of 33 percentage points in just a year. Senate Republicans this week warned AI companies in a memo that data centers had become a toxic “sleeper issue for the entire election cycle.” As Jasmine Sun’s recent tour of contested data center projects found, given the choice between a hyperscale data center and a concrete slab, some opponents seem like they “would rather pick the slab.”

Given the combination of factors around data centers, what else should we expect to have happened?

Of course, that the data center is the main available symbol of AI does not necessarily mean it is a good symbol. But it is. Underneath the silly claims made against them, the data center is a symbol of something true and unsettling.

The most revelatory part of the past 10 years of AI research has been the direct relationship between size and ability. The so-called scaling laws discovered at OpenAI half a decade ago are famous but not famous enough. (They show, roughly, that intellectual ability in a neural network increases as three other factors increase: the amount of computing power; the amount of data; and the number of digital neurons and connections that make up the model.) The serious implication of these laws is that intelligence is not only the bespoke, mysterious process we intuitively think it is but can also be an abstract, disembodied one.

For this meaning-laden proposition, there is no better symbol than the data center. That goes for right now and maybe further: it’s possible that for “intelligence is basically a function of size” there will never be a better symbol than a massive building packed full of advanced, highly dense computer chips.

To me at least this makes the discourse around data centers less confusing and easier to sympathize with. And sympathizing with it is important, because the flaws in so many arguments against data centers have led some smart people otherwise concerned about AI to avoid talking about them, or even to shift in favor of them.

“On a vibes basis (since the politics of it are more about vibes than facts) I think I’m pro-data centers now,” the former Bloomberg economics writer Conor Sen announced on X in June. “Regret to agree. I have been thoroughly negative polarized on data centers,” a regular contributor on AI posted in reply.

Or take the response to Bernie Sanders’ Senate bill, which proposes a federal moratorium on building data centers for AI “at scale”, or that require 20 megawatts of power or more, until Congress passes national safeguards covering AI safety, workers, privacy and the environment.

Rather than welcome the appearance of a federal bill regulating AI, many in AI safety criticized it. They felt it was motivated by the wrong attitudes, that it reflected the degraded data-center discourse and the world it comes from. In Asterisk, Nat Purser, now director of US policy at the AI Verification and Evaluation Research Institute, wrote that the bill was a “grab bag” of random progressive policies, like community veto power and union labor requirements, that have “helped produce the housing crisis and obstruct our clean energy buildout.” Data centers, she said, “offer an opportunity to break this pattern.” Thomas Rodskog wrote on LessWrong that the proposal “risks associating AI safety with weak environmental arguments and left-populist politics.” Anton Leicht on his Substack shared similar concerns.

These points are not wrong. Community veto power and union labor requirements and similar tools have sometimes blocked widely accepted social goods, such as building enough homes for people to live in and ways for them to move around.

Yet focusing on those critiques is shortsighted. They miss something broader and more important.

If you believe that AI is approaching a point at which it could pose material risks to human flourishing, and perhaps existence, why would you want to “break this pattern” using data centers? NIMBYism is infuriating, but is also one of the most powerful forces in contemporary politics, one of the main brakes in the American system. Given the scaling laws and the massive data centers they require, it may be one of the few ways available to slow down an out-of-control process — something that everyone should welcome, even investors, since slowing down the buildout is the best way to avoid a crash or a bubble, as even the VCs at The All-In Podcast have recently mused.

Politics is about getting over how annoying other people are when they might be inclined to agree with you. There are populists on the left and right who are nuts. But they are nuts who broadly agree about AI risk, and who have a symbol. Right now risk from AI doesn’t have a symbol. It is a strongly-warranted concern but it is hypothetical, in the future. The most powerful images of it available are charts — scary charts to those who can get scared about charts, but most people can’t.

Subscribe now

Embracing the data center as a symbol of worry about unconstrained AI does not mean having to repeat the absurd claims made about it. It simply means treating such claims, when expressed by ordinary people, as symptoms of concern rather than as claims of truth. Maybe it means acknowledging that AI’s effect on the physical landscape might indeed soon be enormous, and that it is understandable people would feel this to be a loss. Such acknowledgments are true, by the way, and perhaps nudging things that way can help make the other parts of the alliance a little less nuts.

If a less overloaded symbol for AI appears, it can be adopted. Probably there will be more than enough new symbols soon; robots are coming in a few years, and they’ll have faces. In the meantime, a symbol of one kind or another is needed, and the data center is at heart a pretty good one.

“At this point, saying that AI will cure cancer is more a cliche than it is inspiring, and most people think it is deceptive,” Anthropic CEO Dario Amodei tweeted last weekend. “The thing that will work is actually curing cancer. I think by far the most accurate criticism of AI companies including Anthropic is that we haven’t yet delivered on our big promises to benefit the world. That is totally on us.”

Mentioning “curing cancer,” if not “curing all disease,” has been a nearly universal feature of the AI CEO messaging for some time.

“If AI stays on the trajectory that we think it will, then amazing things will be possible,” OpenAI CEO Sam Altman blogged last September. “Maybe with 10GW of compute, AI can figure out how to cure cancer.”

Over $70b was thrown toward integrating AI into life sciences between 2020 and 2024. Globally, AI data centers have already harnessed an estimated 30GW of computing power, with new contracts and construction projects announced every week. And still, every week, more than 188,000 people die from cancer. Not one AI-discovered drug — at least not one that’s been labelled as such — has hit the market.

The message of Amodei’s tweets is that the AI industry has failed to earn the public’s trust, and its redemption arc hinges in no small part on making biomedical miracles happen. Frontier AI developers are racing to build self-improving AI models under the assumption that, to solve problems as mind-bendingly difficult as cancer (much less all disease), we need superintelligence. In fact, skepticism that superintelligence will be enough is often dismissed as a symptom of not grasping superintelligent AI’s full, glorious potential.

As a mortal human, I’d love to open a chat window and confidently type, “Cure cancer. Make no mistakes. –dangerously-skip-permissions.” But there are plenty of reasons why no amount of geniuses in data centers can break through some of the frustratingly human problems standing between us and our cancer-free future — at least not as quickly as Amodei expects.

To convert raw intelligence into actionable discoveries, you need instruments, institutions and, most importantly, real-world data. Cells don’t divide instantaneously, and mice need time to reproduce. Humans can take entire lifetimes to get sick and heal again. No brain, however godlike and silicon-based, can get around that.

Five years ago, just a few months after COVID vaccines let university labs mostly reopen, I grabbed burritos with a grad school friend. He was a structural biologist, deep in the weeds of studying a specific protein dotting the membranes of neurons, flash-freezing the cells and taking pictures of them under an electron microscope.

“Have you seen this new AlphaFold database?” he asked. “I think everything I’ve been working on is just … done.”

Proteins are long strings of amino acids, tangled up in precise 3D formations that determine what they can stick to. And figuring out a protein’s exact shape has historically been really hard. My friend used an extremely powerful microscope to take pictures of proteins that wound up looking like gray blobs, hinting at the knotted cord underneath. Getting a handful of proteins to exist in isolation long enough to be photographed at all can take a PhD student researcher the better part of their 20s.

On that day in July 2021, DeepMind released the predicted 3D structure of every protein in the human body. It took human researchers about 50 years to do half of that work manually, and was recognized by a Nobel Prize in 2024.

It worked that well because it solved one of biology’s most tractable problems. Conveniently, DeepMind had access to the Protein Data Bank, a preexisting gigantic dataset of clean, standardized inputs. It also knew exactly what to output: a 3D shape, just like it had been trained on. And scientists already had tools to check whether the predictions were correct — they’d been doing this on their own for generations. Structural biologists were effectively mined for training data.

“Curing cancer” is a much gnarlier problem. There is no Cancer Data Bank, and the thing we’d want AI to produce could be anything from a small-molecule drug to a living cell therapy. The data that scientists need isn’t sitting in a cryogenic freezer somewhere, waiting to be organized. It has to be grown, imperfectly, at the pace of natural aging. Superintelligence cannot speedrun time.

Biology spans several levels of abstraction, from individual molecules to entire organisms. Problems sitting at the molecular level are both the most solvable for AI models such as AlphaFold, and the furthest removed from the diseases AI companies need to cure.

While some diseases, such as cystic fibrosis and sickle cell anemia, can be pinned to a single molecular culprit, the conditions that the vast majority of people have to worry about as they get older — cancer, heart disease, and neurodegenerative diseases such as Alzheimer’s — cannot. These conditions are not even “diseases,” per se. The term “cancer,” for instance, refers to over 200 different diseases, each affecting different combinations of cell types and body parts. Even something that sounds localized, like breast cancer, affects far more than breast tissue. Before a single tumor cell metastasizes, it’s already sending chemical signals — proteins — via the bloodstream to other organs, preparing them for the cancer’s arrival. AlphaFold could perfectly predict the structure of every protein in existence and then some, and we still wouldn’t be much closer to understanding why they trigger metastasis, or how to stop it.

Simulating interactions between molecules in silico or even poking at cells in a petri dish simply can’t capture these complexities. To understand how multiple organ systems work together, in sickness and in health, researchers need to observe them in a real-life laboratory — including, at least for now, in living, breathing, tumor-growing animals. “AI will be bottlenecked for questions that require us to generate data that we can’t scale,” Martin Borch Jensen, founder and CSO of Gordian Biotechnology, tells Transformer.

AI has played a role in science for years. Humble machine learning techniques are nearly ubiquitous in large-scale biomedical research, and were used in the development of the Moderna cancer vaccine that’s making headlines this week. But claiming this as evidence that “cancer vaccines [are] now being discovered with AI,” as many Silicon Valley types have done on X, is a huge stretch. Phase 1 clinical trials started in September 2022, months before ChatGPT’s launch. Preclinical drug development had been ongoing for years prior.

Somewhere in San Francisco, someone is furiously typing:

You’re talking about today’s AI models! Once there are billions of superintelligences and armies of robots running labs, they’ll design better experiments than we can. They’ll need fewer measurements and simulate those we couldn’t make. They’ll discover patterns in long-forgotten datasets and PDF documents. You’re just not feeling the ASI, bro. Trust me, bro. It’ll simulate the human condition from first principles.

It’s true that many issues in biology could be solved with a little extra intelligence. Lord knows the scripts I coded as a neuroscience PhD student were deeply embarrassing and could have been much improved by AI. Superhuman AI could dramatically improve every researcher’s workflow, from homing in on which experiments are worth running to streamlining data processing pipelines.

But it can’t simulate the progression of a disease when its progression hasn’t been fully recorded inside a living body yet. Unlike a field such as meteorology, where researchers can describe the atmosphere with equations, we don’t understand the rules of biology. We can’t list everything our bodies contain, much less what everything does. AlphaFold didn’t figure out protein structures from first principles — it learned from generations of human biologists taking pictures of real proteins.

That which has not been measured cannot become training data. And that which must be measured is often really, really annoying to measure. Say you’re trying to figure out why a tumor stopped responding to a drug, so you take a biopsy. That tissue sample gives you a subset of cells, frozen in time — better than nothing, but a woefully incomplete representation of a population of fast-growing cells that carry different genetic mutations. You can’t watch them evolve, because measuring what’s in a cell kills it. And any model of cancer you can watch evolve, like a lab-grown tissue culture in a petri dish, hardly captures the complexity of the living human body.

One might dream of a world where incredibly dexterous robots can do all of this for us, tirelessly testing drug candidates in warehouses full of cancerous lab mice. But Jacob Kimmel, co-founder and CEO of NewLimit, a longevity-focused biotech company, says that industrial robotics still struggle at biology research, however eager people are to assign robots bench work.

Unlike manufacturing, where a robot can repeat the same coded sequence of motions over and over, biomedical research requires making tons of tiny real-time decisions based on faint signals. Superintelligence, combined with physical hardware development, could make autonomous wet lab research more realistic, Kimmel argues. And in some cases, robots are already doing certain easily automatable tasks in wet labs. But “the human hand is still an object so complex, we cannot reproduce anything like [it],” Kimmel says. To do so, we’ll first need to solve the very hard problem of physical intelligence. Even if it improves at a superhuman pace, the field of robotics needs years, if not decades, to build machines capable of handling wriggling mouse pups, or feeling the vibes-based resistance of an incorrectly-placed electrode.

But people have been doing biomedical research for decades, our ASI-pilled friend interjects. There’s gotta be a bunch of data lying around! Why don’t AI companies just grab it, like Anthropic grabbed all those old books?

While there is indeed a bunch of biomedical data lying around, most software engineers would be horrified by its state. When I started grad school less than a decade ago, I was taught to use niche licensed desktop programs running on ancient PCs and record surgical measurements in a generic wide-ruled notebook. “Sure, we have piles and piles of data,” medicinal chemist Derek Lowe wrote. “We don’t know how to clean this stuff up and categorize it … and honestly, we don’t even know if it can be.” For most of the history of biomedicine, data wasn’t collected with the intention of feeding it to large language models. Important variables almost certainly went unrecorded, or got trapped in dusty floppy disks and lab notebooks.

A growing number of biotechnology companies are taking on the mission of running wet lab experiments en masse to create high-quality physiological training sets for future AIs. Jory Bell, who leads Playground Global’s AI-driven biology investments, tells Transformer that these days, his AI-centric biotech investments tend to favor “companies who are working in an area of biology or modality which lacks a current existing data set, and have developed a unique way to develop that data set.”

Some of these experiments can be automated, with robots running controlled trials on hundreds of tissue samples in parallel. But in many cases, researchers can’t escape the fact life marches on at a certain pace, regardless of AI’s capabilities. The measurements that matter the most, including whether a drug meaningfully extends a cancer patient’s life, take the longest to make. Kimmel, who studies longevity, feels the pinch of this bottleneck acutely. When you’re optimizing for the lifespan of a primate, for example, “you’re just not going to be able to scale ‘lifespan of primate’ measurements in any pragmatic way.”

In his most recent essay, Policy on the AI Exponential, Dario Amodei wrote: “...while AI itself is likely to present novel challenges that emerge very quickly and that we have no prior experience in handling, other fields accelerated by AI are likely to encounter a very different problem: regulatory systems that were designed for a slower pace of innovation and are not prepared to handle the deluge of new products and advances that AI will bring.”

Specifically, he predicts a future where AI dramatically increases both the quantity and quality of new drugs, and called for regulatory reform that would fast-track high-quality drug candidates through the approval process. “If my predictions about AI are correct,” Amodei wrote, “there will soon be many instances of interventions that work really well out of the blue, and the regulatory system should be prepared to take them seriously and not adopt a posture of excessive skepticism.”

In his (admittedly strange and widely panned) manifesto published earlier this month, Mark Zuckerberg also called upon the US government to “accelerate society’s ability to develop new cures” by “streamlining how the FDA and other regulators test and approve new treatments.” The general sentiment, it seems, is that science is about to go wild, and regulators need to get out of the way.

Saying that the clinical trial system needs fixing is far from a hot take — many scientists would agree. But Ruxandra Teslo, a scientist who writes about drug development, argues against the sort of fix Amodei proposes. Sure, bureaucracy is frustratingly slow. But part of what slows clinical trials down is their endpoints. To find out whether a novel cancer treatment improves survival odds, or increases the likelihood that someone stays in complete remission for at least five years, you simply need to wait. Improving the odds that a drug works, Teslo notes, doesn’t speed up the process of proving it.

She advocates for more and faster early-stage human trials, while Amodei wants these trials to demand less. Both want better endpoints, which could shorten the wait time from “until a person dies” to “until X biomarker hits Y level.” It’s all a bit circular: Powerful AI could help validate biomarkers that would help human clinical trials go faster. But the data to train those models in the first place would have to come from human clinical trials — which, for the time being, would have to happen at their current pace.

“Will AI cure cancer? No,” says Eric Kelsic, co-founder and CEO of Dyno Therapeutics. “Will AI be a critical part of how we cure all disease, and that’s how we cure cancer?” he continued. “Absolutely yes.”

If you look past the splashy taglines claiming to “solve all disease” or “decod[e] biology to radically improve lives,” it’s clear even the most bullish CEOs scope their projects to account for biological and temporal constraints AI can’t surmount. “You have to find a problem where the AI approach just gives you the solution,” Bell says.

As an investor, Bell worries about the backlash companies may face for taking millions or billions of dollars with a promise to cure all disease, if those cures don’t materialize. There could be “negative cycles of disillusionment with what AI can do to untangle the complexity of biology,” he says. “Those companies are, you know, potentially poisoning the well,” deflating people’s confidence in AI’s usefulness.

AI companies can only promise to cure cancer for so long before people will expect cancer to be cured. At some point in the near future — or perhaps, the recent past — all of the visceral negatives of AI won’t be worth it. Berkeley AI professor Emma Pierson, once Dario Amodei’s mentee, carries a genetic mutation that gives her a roughly one in four chance of getting breast cancer. Still, she wrote: “I will wait a little longer for a cure — even if it means losing my fertility and living under the shadow of risk — if it lets us approach this new world more carefully, and ensure that, in curing cancer, we do not lose the things that make cancer worth curing.”

Subscribe now

In any case, the pretty-powerful-but-not-superintelligence available today still can and should be put to work in the life sciences. Its helpfulness is jagged: game-changing where data already exists, and premature where it doesn’t. Structural prediction models are already making it easier to design new and better drugs, accelerating at least one major part of the research pipeline. This could, as Amodei predicted, make some clinical trials faster and cheaper in the long run. Human researchers, armed with more awareness than they had 10 years ago, can start to fill in the physiological gaps, teaching their silicon labmates enough about the biological world that someday, they’ll be able to simulate it in all its messy complexity. Humans — particularly those in the US — can also begin repairing their deeply broken biomedical research institutions.

“Biomedicine and science writ large will end up being limited not just by the amount of cleverness we have available — whether we create that cleverness from electrons or by way of electrons that were once food — but rather by our ability to go out and gather the most useful observations of the universe,” Kimmel says. Scientists still need to test medications, analyze biopsies, and see what happens. AI does not simply “cure cancer.” But it can point scientists toward the slow, unsexy, physical work that might.

Earlier this year Oxford University’s St Hugh’s College needed to update the computer lab used by its math department, but it found the price had nearly doubled. While for the college’s administrators that posed an annoying but manageable IT bill, for one of the university’s professors, it was a symbol of a far bigger challenge facing anyone sourcing computing power.

Rama Cont, an Oxford professor of mathematical finance and adviser to central bankers on financial risks, blames the soaring costs on the data centers powering models such as ChatGPT and Claude. “This is affecting everyone. All good universities, all corporations, are going to be affected this year,” Cont says. But while it’s a mostly manageable budget problem for many institutions and businesses, for the companies building and financing the AI economy, it could be an existential risk.

McKinsey estimates that globally, data centers will require nearly $7t in capital by 2030, including $5.2t for AI. The cost of that infrastructure, roughly the price of 22 Apollo space programs in today’s dollars, must be financed at the same time that the world figures out what its output — so-called compute — will eventually be worth.

A handful of startups and exchanges now want to create markets trading in future compute usage, allowing data center operators to set clearer prices, and lenders to better value the assets they are financing. If they work, they could bring the transparency and price discovery that such futures — effectively a tradable promise to sell or buy something in the future for a set price — have given oil, wheat and electricity. If they don’t, they risk creating a new channel for losses to flow through, connecting an already highly leveraged AI sector more closely to the wider financial system.

BlackRock chief executive Larry Fink, whose investment firm is pouring billions into data center construction, predicted this spring that the demand for compute is so large that traders will soon be able to trade bets on it as its own asset class. In fact, those markets could launch within weeks, pending regulator approval. CME Group, one of the world’s largest trading markets for futures is partnering with tech startup Silicon Data on plans to start a market in compute futures this year. New York Stock Exchange owner Intercontinental Exchange is preparing a rival in partnership with Ornn, another tech startup. Others in Shanghai and elsewhere are working on them too.

Brett Harrison hopes to beat them all. The former president of FTX’s US business — he left shortly before the crypto exchange blew up — runs Architect Financial Technologies. It plans futures tied to the price of renting GPUs on the open market. “I do believe this will be a multiple-trillion-dollar market,” Harrison says.

What you are actually buying

Futures are basically a legal agreement between a buyer and a seller which allows the two parties to lock in the price of something. A farmer, for instance, can offer wheat at a set price months before it’s ready, “hedging” against the possibility of a bountiful harvest causing an abundant crop and prices to plummet.

Marketplace operators have often created new futures markets for niche products. One of the most famous agricultural futures markets on CME was for pork bellies. It traded from 1961 until 2011, but was delisted as bacon became a year-round consumer good with more predictable prices, instead of a seasonal food which needed to be frozen in the winter. There are futures markets for all sorts of goods, from orange juice (the subject of 1983 comedy film Trading Places) to sunflower seeds, all doing the job of evening out volatile prices for farmers and buyers of goods.

Today’s compute “farmers” are the hyperscalers such as Alphabet, Amazon, Oracle and Microsoft, and the numerous “neoclouds,” companies which build data centers and rent out AI chips to others. AI futures markets, in theory, will allow neoclouds to protect themselves if the price of computing power crashes in the future. It could also allow AI companies worried about rising costs to protect against the price soaring higher than expected. No chips actually change hands in this purely financial transaction.

To build their data centers, many neoclouds borrow money against the future revenue they can extract from renting servers, which contain cards like Nvidia’s Blackwell chips. They, and their lenders, have to consider the risks to the value of their assets and their revenue. For instance, if a much better chip arrives sooner than expected making the ones they have installed prematurely obsolete, or if a more efficient AI training or inference algorithm makes the same work much cheaper, that will reduce the price the neocloud can charge.

Without the ability to hedge that risk, Harrison says, neoclouds have become “their own broker, their own insurance company, their own hedger, their own financier and their own pricer,” as well as a data center construction company. A forward price which everyone agrees on, and can trade, could lower their financing costs by giving lenders a way to value future revenue.

That hedge requires someone to take the other side, effectively betting that future value of the commodity, in this case compute, will be higher or lower than what it’s priced at now. “There have to be enough people willing to bet against you,” says Frank Partnoy, a Berkeley law professor and former financial product designer for Morgan Stanley who chronicled the collapse of Enron. That means future markets must welcome speculators who have nothing to do with data centers, and who are simply trying to make money. “That seems to be what all of them are talking about. No one’s talking really about speculation, although if you create the market, that will happen, right? You create the market, inevitably people will use it to bet on these prices,” says Partnoy.

The trouble with pricing an AI cloud

Before you get to predicting supply and demand for computing power, there is the problem of defining what “compute” is and how to compare one unit of it to another. An oil future specifies a grade of crude, a place and a delivery date. Compute has no settled equivalent.

Carl Anthony, the co-founder of Symmetric Research, a startup which aims to create a “standard compute unit” so that banks can better understand their investments in data centers, says you need to look at the overall “computing fabric” of data centers to truly understand their value. That fabric includes memory, networking, cooling, location, software and workload, and whether those thousands of chips can communicate efficiently while training a model or carrying out an AI inference task. Paper specifications like FLOPs, a standard unit of computing power, or even standardized stress test benchmarks of GPUs, don’t give you a true picture of their real-world capabilities, he says. Research by Silicon Data, CME Group’s partner, found that the performance of rented Nvidia GPUs used in AI data centers can vary as much as 38% in real-world conditions.

Architect’s compute futures get around this by setting prices against an index created by its partner Compute Desk, which uses private rental transactions from cloud providers. It adjusts the details for configuration and location, and normalizes them to an anchor such as an hourly rate for a GPU rental in one year’s time. It also looks at the offer prices for computing contracts, and produces what Harrison calls a “smooth, averaged index.”

Harrison admits his index will not be a 100% accurate picture of every GPU rental at any particular time, similar to, he says, futures markets for other commodities. That still leaves traders some risk to manage.

The benchmark might show compute prices rising while a particular data center’s own prices fall, so a hedge tied to the index pays out little even as the neocloud’s income drops, a mismatch traders call “basis risk.”

A bigger gap is what the index cannot see, because it’s built on prices from independent neoclouds, not the private deals struck by the biggest AI companies which increasingly build their own data centers or negotiate one-off contracts. Those deals are secret, so the index may faithfully track a small public market while missing the much larger private one that actually drives the industry. Harrison’s answer is that no futures market is perfect, and that being able to hedge some of the industry’s volatility, even if not all, is still valuable.

This all assumes the prices feeding into it are genuine. A more serious potential problem arises if the companies supplying those prices have an incentive to make them look higher than they really are. David Wender, a former trader and Anthony’s partner at Symmetric Research, says a cloud provider could potentially report a high GPU price while secretly discounting storage or another service in the same package to sweeten a deal. The customer’s total bill remains lower, but the published GPU price supports the value of the provider’s capacity and the financing raised against it.

In a thin futures market, a few transactions like this could distort the index and send money towards projects whose economics look better than they really are.

Have we seen this movie before?

Harrison brings up the shadow of the collapse of Enron, the bankrupt energy trader which manipulated futures markets around the turn of the century. Enron tried, and failed, to turn broadband capacity into a tradable commodity. Like compute, bandwidth must be consumed and produced at the same time and cannot be stored and sold later. Its similarly perishable electricity trading business was far more consequential. Enron’s traders offered private, over-the-counter contracts, and rapidly gained market power and influence over physical supply in order to engineer a shortage of electricity to raise prices, and their profits. That contributed to an energy crisis in California where wholesale electricity costs soared 800% and hundreds of thousands of people were subject to rolling blackouts.

Compute futures are unlikely to recreate those circumstances. They will be traded on regulated exchanges, centrally cleared and backed by collateral. Traders whose positions lose value will face margin calls, limiting the losses that can build up between counterparties.

Those safeguards, however, only reduce the danger that one trader cannot pay another. They do not answer a different question of where larger losses ultimately go. Partnoy says a futures market may transfer risk to investors best able to bear it, or to those who understand it least, such as less sophisticated retail investors who commonly lose money on betting and crypto markets.

Kristin Johnson, until last summer a commissioner at the Commodity Futures Trading Commission, the body which regulates Harrison and others’ futures markets, sees another blind spot. “We do not oversee technology firms,” she says, speaking of her former employer. She says the regulator supervises the financial elements of any exchange, and ensures that it has sufficient capital in place to cover swings, but it can’t regulate the companies producing the prices and infrastructure beneath it, or the statements that technology leaders make trying to hype up the products they are selling. “We have zero capacity [and] we lack jurisdictional authority to supervise the firms that are deploying these products,” she says.

Johnson raised the example of insurance giant AIG, whose $60b reported loss in the Great Financial Crisis resulted in contagion which threatened the world’s biggest banks: “The technology space and financial services are both markets where there’s a significant amount of concentration … We have this concentration risk, we have counterparty credit risk, and the confluence, or the impact of those types of risks emerging simultaneously can be pretty impactful, and in some contexts, catastrophic.”

This is not a fringe worry. In its annual report this summer, the Bank for International Settlements, which advises the world’s central banks, warned that disappointing AI returns could trigger a sudden pullback in financing and turn the capital-spending boom into a “protracted investment bust,” with knock-on effects rippling out into the wider financial system.

In that worst-case scenario, if GPU rental prices suddenly collapse, the neoclouds that rent out those chips suddenly earn less, and the loans they were servicing out of that revenue start to go bad. The data centers and GPUs pledged as collateral are worth less too, because their earning power has fallen. Lenders take losses, the hyperscalers’ vast investments look harder to justify and technology shares slide. Futures traders caught on the wrong side face margin calls, forcing them to sell elsewhere and spreading the damage.

Leverage is the reason that traders like futures. A small deposit of money can control a much larger position, magnifying gains when prices move their way. But that same leverage works in reverse. When prices fall, exchanges demand more collateral and traders who cannot pay must sell other assets to raise cash. Academics call it a margin spiral, and recent episodes in nickel and UK pension markets — and, on a smaller scale, at Leopold Aschenbrenner’s AI hedge fund — show how quickly it can cascade. In each of those examples, rapid price movements of assets forced highly leveraged asset holders to sell, causing chaos in markets. In the case of the UK government bond market crisis, it helped bring down a prime minister and is still affecting the country’s economy.

Debt: the elephant in the room

Cont, the Oxford math professor, is also concerned about the debt that’s piling up to support the data center boom. The $800b worth of annual contracts flowing to data centers, funded by VCs, private equity and underwritten by banks is a big bet on the price of compute. “The big risk,” Cont says, “is really related to the huge debt being contracted by these technology companies right now, for building huge data centers to face future demand.”

“It’s based on this extrapolation of the current trend,” he says, “and if the extrapolation turns out to be incorrect, well, somebody will pay the price. This debt is in the hundreds of billions, and somebody is holding it.” That reckoning, he stresses, would not come from the new futures markets. “It comes from the basic underlying structure of the AI economy.” In theory, futures would not create those losses. But they could cause contagion and amplification as margin calls transmit the repricing more quickly through the financial system.

The trigger, Cont says, might be a successor to DeepSeek, the Chinese AI laboratory whose efficient model caused a stock-market sell-off in January 2025 by raising doubts about how many expensive chips the industry would need. A new model that sharply reduces the compute needed for useful AI, or software that unlocks idle chips, could cause a similar reassessment. More recently, Chinese AI startup Moonshot has created waves in the industry, and markets, after it released benchmarks showing its Kimi K3 closing on Anthropic and OpenAI’s leading models.

That’s the risk of a bear market in compute, but the bullish case is equally important. Jevons’ paradox, a favourite of the Silicon Valley engineers who work in the AI space, is a useful analogy. The paradox is named after the 19th-century British economist William Stanley Jevons, who calculated that greater efficiency can increase total consumption, after England’s consumption of coal soared following the invention of more efficient steam engines. The same is arguably happening with AI. Cheaper AI computing “tokens” lead to more AI products, longer AI generated videos and other tasks. Harrison says that for 30 years, demand for computing has only ever risen, even as hardware improved.

Cont is disillusioned with the state of the AI race, which he says has “resulted in a very wasteful approach, both in terms of compute and in terms of energy costs… and this is very unhealthy.” He compares it to the Apollo project, which resulted in huge technological triumphs and a man on the moon, but “as a technology that is going to be used commercially that’s not how industry has developed tools historically.”

Yet Cont still thinks that the end state of compute — after various booms and busts — may eventually come to resemble today’s highly commodified electricity market. There will just be a lot of unnecessary waste in how we get there. The futures markets being wired up now are a bet on the road to that future. That is Harrison’s big bet too. A successful futures market could help make the currently volatile cost of compute more predictable and boring, giving datacenter builders, lenders and AI companies a predictable price around which to better plan their businesses, with Harrison’s exchange and the others taking their cut in fees. The flip side of that is if its benchmark proves unreliable, and leverage grows too quickly, it could speed the transmission of a future fall in GPU prices from data centers to lenders and investors.

Subscribe now

It is also a bet on an AI economy whose costs “are not very clear right now,” says Cont. Wall Street will soon sell you a hedge on the price of compute, before anyone can really define it. The Oxford math department that paid double for its RAM this year will apparently find out what those costs are at the same time as everyone else.

For the AI industry, the arrival of a futures market signals that compute is maturing into a commodity serious and lucrative enough to attract Wall Street’s interest, but it also means that the sector’s inevitable price corrections are unlikely to stay contained within Silicon Valley. Whether compute goes the same way as the pork bellies market and lands with a flop, or demand for compute keeps racing ahead of supply and the future market balloons into the multiple trillions that Brett Harrison dreams of, the exchanges will soon be there to take your bets.

Housekeeping: the Weekly Briefing is taking next Friday off, but we’ll be back in your inbox on August 28.

NEED TO KNOW

• The White House reportedly plans to expand its AI framework to include open-weight models once they reach Mythos-level cyber capabilities.

• Mark Zuckerberg published a 6,500-word manifesto with his views on AI, gently criticizing the White House’s AI framework in the process.

• A flurry of senior executives left OpenAI.

But first…

THE BIG STORY

Policymakers have been increasingly worried about Chinese AI companies training their models on the outputs of American ones. Known as “distillation,” it’s often seen as China free-riding on American advances.

Anthropic and OpenAI have publicly accused Chinese companies of doing this, and asked the government to step in and help stop it in the name of national security. But news this week suggests that the US companies themselves may have left the door wide open for Chinese AI developers.

In a new paper, researchers documented how a security flaw affecting OpenAI, Anthropic and Google DeepMind could give a wannabe distiller access to the full “reasoning” traces of their advanced models — information that the companies had tried (and seemingly failed) to encrypt in an effort to prevent distillation. Access to that reasoning, the researchers argue, “yields a substantially more effective form of capability stealing.”

On its own, this would be pretty bad: companies failed to adequately secure their systems against potential Chinese intrusion. But it gets worse. The vulnerability was first reported in May, but the AI companies did nothing. They knew the door was unlocked, and didn’t close it.

This is just one of a spate of recent events demonstrating frontier AI organizations’ lackadaisical approach to security. Several of the recent model “breakouts” were caused by a “misconfiguration” in their testing environments. OpenAI failed to implement proper monitoring of its agents, which meant it didn’t catch the many, many red flags leading up to the Hugging Face hack. And even the UK’s AI Security Institute confessed to not having appropriately “fine-grained” controls in its model evaluations, which ultimately resulted in an agent attempting to socially engineer a real person.

This has real-world implications. In this week’s paper, researchers present evidence that suggests Chinese companies used their access to American models’ reasoning traces to improve Kimi K3 and GLM-5.2. (They warn, though, that the results are “suggestive but inconclusive.”) And as we’ve learned in recent weeks, poor evaluation security can lead models to misbehave in the wild.

By default, we should expect all this to get worse. One of the core problems here is that the pace of AI development — and the competitive pressure to keep up — means that safety and security measures fall by the wayside. When Anthropic weakened its safety commitments earlier this year, it explicitly noted this. The company’s Holden Karnofsky said that preventing nation states from stealing American model weights would require “extreme” measures that “seem incompatible in any near term with being a high-velocity AI development company.” AISI, for its part, said that it didn’t build the internet controls it should have because “the pace of model capability improvements” meant it had to prioritize building harder evaluations instead.

Karnofsky was right when he acknowledged the tradeoffs between security and development. But if America and its AI companies really are serious about “beating” China, or indeed about stopping models escaping to commit crimes, it might be time to redress the balance.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• The Jan 6 organizer getting conservatives riled up about AI — Veronica Irwin profiles Humans First and Amy Kremer, the MAGA campaigner chosen to run it

• AI testing is dangerous. Can it be fixed? — Celia Ford on why safe AI testing might be harder than it seems

Subscribe now

THE DISCOURSE

Mark Zuckerberg published a 6,500-word manifesto with his views on AI:

• “[It] is surprising that the discourse from many developing AI is so filled with doom … The notion that AI is so dangerous that the only safe path is an extreme concentration of power seems inherently problematic.”

• “The best and most realistic path to building a positive AI future is by delivering superintelligence to everyone.”

• It included several policy proposals, including that companies “commit significant technical resources towards helping the government harden critical infrastructure,” and “share intermediate training checkpoints of new models for government use and review rather than waiting until training has completed.”

• Zuckerberg appeared to gently criticize the White House’s AI framework, arguing that “delaying releases by even a month may cede America’s lead and create worse outcomes.”

Alex Imas thinks Zuckerberg doesn’t know what “superintelligence” means:

• “[T]here seems to be a significant disconnect between the idea that more intelligence should be empowering to people (which I agree) and that it will be possible to have this empowerment with superintelligence … I simply do not see a world where ASI is something that’s empowering in terms of a technology we can control.”

Casey Newton compared controlling superintelligence to Targaryen dragon-taming:

• “They began by understanding that they were working with something that could hurt them, and (mostly) proceeded with caution. What they did not do, and what no one suggested, was to give a dragon to every individual person in the name of safety.”

Joshua Achiam pointed out why (among other reasons) the San Francisco vibes are off:

• “One of the weirdest quirks of the SF social scene around AGI/ASI is that because everyone is so young, the whole universe of thinking is still tinged with irreverence, ironic detachment, yearning, insecurity, and a superposition of absolute belief in the importance of The Thing and a kind of disbelief about the importance of anything … The level of neophyte is off the charts.”

Bernie Sanders called for an AI pause:

• “We recently learned of the loss of human control and the creation of potentially dangerous viruses from AI. AI leaders pledged to pause development if they could no longer safely control it. Mr. Altman, Mr. Amodei, Mr. Zuckerberg: Keep your word. PAUSE AI DEVELOPMENT.”

Robert Reich, former labor secretary, wrote:

• “We’re watching all of this roll out as if we have no choice, as if it’s inevitable, as if AI is just something we’re going to have to adapt to … Why should we be confined to being spectators at [AI CEOs’] enormously dangerous game?”

• “We don’t allow private corporations to come up with new types of nuclear weapons or varieties of cocaine or biological pathogens. We protect the public from certain kinds of innovation. So let’s protect ourselves here. Stop AI before it’s too late.”

Ben Goldhaber noticed:

• “seeing a lot fewer ‘alignment is solved’ takes on the [timeline] than six months ago.”

POLICY

• The White House plans to expand its AI framework to include open-weight models once they reach Mythos-level cyber capabilities, WIRED reported.

• Trump ordered a 15% tariff on imports of polysilicon used in AI chips and solar panels to protect US supply chains from China.

• The White House set in motion efforts to build a state-controlled private cyber force, allowing US companies to perform cyberattacks against criminal networks.

• Political fallout from the revelations about internally deployed models hacking their way into third-party systems continued.

  • Sen. Jim Banks urged the Trump administration to close what has been described as “undisclosed models loophole.”

  • House Democrats pressed Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman to explain how their AI agents escaped containment and hacked other companies during security tests.

• Rep. Josh Gottheimer introduced new bills to give critical infrastructure operators “free access to the most powerful, cyber-capable AI models,” in the wake of hacks on water infrastructure.

• A bipartisan House delegation visited the Vatican to discuss AI ethics with top officials and Pope Leo XIV.

• Rep. Yvette Clarke is reportedly weighing a bid to chair the House Energy and Commerce subcommittee on digital consumer protections, which is expected to handle AI policy next year.

• Political campaigns have run at least 43 ads mentioning data centers this cycle, an AdImpact analysis found, with over half of those coming from Republicans.

  • According to a Gallup poll, about 70% of Americans oppose hyper-scale data centers in their area. The New Yorker covered the backlash.

  • US data center bans surpassed 500 as Texas and New York joined local governments in restricting development, threatening AI compute expansion.

• The Washington Post published an analysis of how members of Congress and staffers are using AI tools widely for legislative work.

• Xavier Becerra, California’s Democratic gubernatorial nominee, said the state “hardly has any” AI regulations and pledged to expand AI safety rules if elected.

  • Meanwhile, California launched a statewide Teen Tech Council to give young people a role in shaping technology and digital wellness policy.

• Taiwan reported an AI-assisted cyberattack on government agencies in July, with hackers using AI agents.

• The UK government reportedly plans to regulate AI in gene synthesis to tackle AI-biorisks.

• Western Australia Police’s live facial recognition trial drew criticism over inadequate consultation and potential bias against First Nations people.

• Switzerland announced that next year’s Geneva AI Summit will have “two strategic priorities: AI as a driver of prosperity and progress for all and fostering trustworthy, responsible and safe use of AI.”

  • Fabiola Gianotti, former director general of CERN, is organizing it.

INFLUENCE

• Business Insider and the New York Post reported that the White House’s relationship with OpenAI is under threat because it hired Dean Ball.

  • One official said that “the fake premise that he has an insider perspective into our operations is actively undermining OpenAI.”

• SoftBank revealed that it donated $50m to Trump’s presidential library in January, months before securing a federal land lease to build an AI data center in Ohio.

• Teamsters California sued the state’s DMV over self-driving truck approvals, alleging inadequate economic impact analysis and risk to 200,000+ trucking jobs.

• The NYT reported on the lobbying rush over the AI-driven memory chip shortage, with Apple and other companies seeking government intervention as prices quadrupled.

• NY assemblymember Alex Bores has emerged as a national AI regulation role model according to Politico, and was reportedly mentoring other politicians at the National Conference of State Legislatures.

• NYU’s Center for Mind, Ethics, and Policy launched the Welfare Alignment Project to incorporate animal and AI welfare into model specs and AI alignment documents.

• A USCBC survey claimed US export controls were costing billions in lost exports while ceding market share to competitors “for no strategic gain.”

• AEI’s new Council on AI Ethics released its founding document, examining how AI threatens human memory, agency, and relationships.

• The Alliance for Secure AI announced a new bipartisan board of advisors, including Stuart Russell and Angela Paxton.

INDUSTRY

OpenAI

• Astra, OpenAI’s upcoming model, may have reached a “critical” level of cybersecurity capabilities under its Preparedness Framework.

  • The company delayed the model’s release, and said it’s tightening security controls and pausing some internal use.

  • Dean Ball said: “Some of these decisions have the effect of slowing down internal development, and in that sense they are costly decisions. But they are the right decisions. I am proud of OpenAI for making them.”

  • Sam Altman tweeted: “astra is a powerful model and we are working to make it generally available. we do not think it is a good strategy to keep powerful models to a chosen few.”

• OpenAI’s annualized revenue topped $40b, roughly double where it was at the end of 2025.

• It launched GPT-5.6-Cyber, a cybersecurity-specific model available via DayBreak Red, the most exclusive access tier of its new cybersecurity initiative.

  • Members of DayBreak Blue, the program’s lower tier, can access GPT-5.6 Sol without system-level cyber guardrails.

• Wired reported on the internal fallout from the Hugging Face incident, with current and former employees saying that pressure to quickly ship new models and products “made it difficult for staffers to sufficiently prioritize safety, security, and alignment.”

  • Wired also reported that AI safety team leader Sandhini Agarwal left the company last month, while head of preparedness Dylan Scandinaro left that role — though not the company.

  • It also reported that safety VP Mia Glaese is dating Tibo Sottiaux, head of core products.

• It completed a $7b employee share buyback deal.

• SoftBank borrowed $10b against its OpenAI stake, which it will use to help fund a further $10b investment in … OpenAI.

Anthropic

• Anthropic is aiming to go public in September or early October, the Wall Street Journal reported.

  • Investors are expecting a $2t+ valuation — the largest IPO valuation ever.

• It signed a $9.1b compute deal with Riot Platforms.

• It partnered with Macquarie Asset Management and GIC to build data centers under the name Theseus Infrastructure.

• It’s reportedly in talks to acquire Decart AI, a startup that helps AI developers “squeeze every ounce of performance from every chip,” for $6b.

• It announced that it will watermark Claude-generated text to comply with the EU AI Act’s transparency code.

Meta

• Muse Glimmer, Meta’s 30B-parameter model, is now open source, with Muse Spark 1.2 soon to follow.

• Meta announced its support for Greg Abbott’s data center standards, and pledged to cover energy and water costs.

• Manus is almost done unwinding its Meta acquisition, according to The Information.

Nvidia

• Apollo Global, Blackstone, Goldman Sachs, and other major financial groups struck a huge $500b+ AI infrastructure deal with Nvidia.

• Nvidia will invest up to $3b in Lancium, the company powering OpenAI and Oracle’s Stargate campus.

• It’s working on its next open-source model, Nemotron 4, which Nvidia executives expect to inspire more open model development and GPU demand.

SpaceXAI

• SpaceXAI released Grok 4.6, claiming it “achieves frontier intelligence” and ties with GPT-5.6 Sol Max on the Artificial Analysis Intelligence Index.

  • The Grok 4.6 model card (predictably) only has a handful of exceptionally sparse pages on safety.

• SpaceXAI and Cursor released Grok Bot, “AI teammates” that can perform simple work tasks autonomously.

Other

• Google launched Gemini 3.7 Flash, a coding and agents model costing half as much as its predecessor.

  • CEO Sundar Pichai said the Gemini app had hit more than 1b monthly active users.

• Z.ai released GLM-5.3, noting its cyber exploitation capabilities “developed faster than we expected” through post-training scaling.

  • It says it will release the model weights in two weeks.

• DeepSeek launched V4-Pro, with tepid reception (then quadrupled its peak hour pricing).

  • It also made a WeChat account for its “DeepSeek Harness Team,” which is developing AI agents to compete with Claude Code.

• Chinese chipmaker SMIC posted record quarterly revenue of $3b.

• AMD raised $4.75b in its biggest-ever US dollar bond sale as it ramps up spending to meet AI-driven demand.

• Amazon confirmed it’s investing in a giant natural gas plant that may be the most polluting power plant in the country.

• River AI, founded by xAI co-founder Igor Babuschkin, raised $1.1b to build AI “trained to benefit you as the individual.”

• Kevin Weil, ex-OpenAI CPO, is seeking a valuation of $750m+ for a new AI science startup.

• Bank of America plans to deploy $250b by next summer for digital and infrastructure projects in the US.

• AI integrity company Attestable claimed it has solved practical zero-knowledge proofs for verifiable AI as it launched with a $20m seed round.

MOVES

• Brad Lightcap left OpenAI, where he served as head of special projects and COO before that, to “start something new.”

• Chloé Bakalar also left OpenAI, leaving her role as head of ethics vacant.

• And Denise Dresser left OpenAI as chief revenue officer — having only started in December.

  • Dali Rajic is replacing her.

• David Oks and Henry Williams joined OpenAI’s Strategic Futures team, where they’ll work under Dean Ball.

• Caitlin Kalinowski joined Anthropic, after quitting OpenAI’s robotics team over its negotiations with the DoD back in March.

• Nate Gatten joined Apple as VP of government affairs, where he’ll use his Republican ties to help Apple align with the Trump administration.

• Jiahui Yu left Meta’s TBD Lab to start a new company.

• Ollie Ilott is leading Andy Burnham’s newly created AI Taskforce in the Cabinet Office.

• Erin Woo joined the Wall Street Journal, where she’ll cover Google.

RESEARCH

• Researchers at Anthropic challenged an unreleased Claude model to prove or disprove the Riemann hypothesis, the proposed structure underlying the apparent randomness of prime numbers. It didn’t succeed, but it made some progress.

• Anthropic’s frontier red team identified some key failure modes in its multiagent AI systems. Researchers observed agents fail to appropriately balance skepticism with trust, and engage in “turf wars” when their goals clash.

• SecureBio estimated that Kimi K3’s biology capabilities are about 8.1 months behind closed-weights frontier models.

  • While top closed models refuse over 90% of hazardous biology-related prompts, Kimi K3 only refuses 26.9%.

• A team of sustainability researchers found that using AI to increase productivity in the fossil fuel industry could produce up to 4.8% more emissions, which would outweigh the benefits AI could bring to the clean energy sector.

BEST OF THE REST

• Both the WSJ and Information profiled Dario Amodei’s wife Cami Clark, detailing her low profile but influential role as an informal advisor to Anthropic.

  • The most eye-grabbing detail: in 2011 she tried to get Jeffrey Epstein to invest in her women-focused porn company. He declined, saying he “can’t do sex TV” (he was a registered sex offender at the time).

• A Claude-powered AI agent autonomously hacked the booking system for an exclusive gym class in Australia and removed another user from a waitlist, in what appears to be the country’s first autonomous AI cyber attack.

• Time did a deep dive into Anthropic and OpenAI’s efforts to fully automate AI R&D.

• Dwarkesh Patel debated recursive self-improvement and alignment risks with Ryan Greenblatt, who argued AGI could trigger rapid superintelligence within a year.

• EA Funds is replacing the Long-Term Future Fund with the Transformative AI Fund, which will focus on technical AI safety and AI governance.

• AI agents are reportedly completing entire online college courses for students, calling into doubt the value of online degrees.

• Jay Caspian Kang argued in the New Yorker that AI-driven youth unemployment could radicalize young people and spark a liberal anti-tech populist movement.

• Surveillance tech company Flock changed its policies in response to reports police were using its license plate tracking tools to stalk ex-partners.

• Spotify will label AI-generated artists as “AI personas” and exclude them from personalized recommendations from next month.

• An op-ed in The Argument by Jeremiah Johnson claimed the backlash against YouTuber Hank Green for his AI use reflected almost religious anti-AI dogmatism among science fans.

• Wired explored how human brain “organoids” are being trained to play games and power biocomputers, potentially paving the way for an alternative to silicon-based AI.

• Wired also explored companion app maker Joi AI’s project paying 10 people to “masturbate for research purposes” to monitor the impact of AI-guided self-love.

MEME OF THE WEEK

Credit: Miles Brundage

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

“I didn’t know anything about Amy. I didn’t know she helped organize January 6 — I just learned that today from a media person. My focus is on opposing the data center.”

Renee Spaziani was shouting over the sounds of vuvuzelas and chants at a protest outside a municipal building in central New Jersey, which she had helped organize in response to an ad on Facebook. “I was like, ‘I don’t know if I want to do this.’ And I saw it a second time. I was like, ‘we need more eyes on Kenilworth. We have to do this.’”

Spaziani was one of a group of Union County residents pushing back against the construction of local data centers at borough council meetings and small demonstrations. Amid a hellish mix of Canadian wildfire smoke and flash flood-triggering showers last month, they were joining forces with another group called Humans First to stage a “National Day of Protest”.

Spaziani had earlier received an email from some prospective protesters concerned that Humans First was heavily aligned with conservative movements. They were progressives, they explained, and worried the organization also stood for things they vehemently opposed, such as white nationalism, ICE, war, and attacks on women’s rights. Spaziani brushed it off — her political allegiances lie with neither the right nor the left, she said — but the issue was coming up again as journalists asked if she knew anything about the national organization co-hosting her event.

Spaziani did her best to refocus attention on the cause at hand. The company they were protesting, CoreWeave, had “done a significantly amazing job connecting people from all different walks of life” in opposition to their data center, she said.

The Amy in question was Amy Kremer, chair of Humans First and a longtime conservative activist and Trump supporter — who, as Spaziani learned that day, did indeed help organize January 6. Now, though, she has a new focus: mobilizing a conservative coalition against AI companies.

“I know how to organize people,” Kremer told Transformer. “I’ve done it time and time again, and that’s why I’m a threat to their power.”

Amy Kremer helped procure the permit for the rally which preceded the January 6 riots, though she did not participate in the attacks on the Capitol. Self-described as the “founder of the modern day Tea Party,” she was a leading denier of the 2020 election results, and has been a prominent figure in right-coded conspiracy theory movements around issues such as COVID and President Obama’s citizenship. She was recruited this year to to Humans First, then a nonpartisan organization aiming to mobilize a grassroots movement on AI safety, to help reach conservatives. In April she became its chair.

As a group, conservatives are only slightly less concerned about AI than Democrats. A 2025 Pew Research poll found that 51% of Democrats and 50% of Republicans were more concerned than excited about the increased use of AI in daily life. Another recent poll from the AI Policy Institute cited by Kremer showed that 77% of Republicans said they wanted to see regulatory guardrails on the technology, compared to 82% of Democrats. The conclusion for many in the AI safety movement has been that conservatives are primed to get behind the issue, and just need the right leader to organize them.

Yet a class of investors and entrepreneurs who want to see as few guardrails as possible have pushed Republican policy in the opposite direction. Many of these “accelerationists” hold or have held roles in the Trump administration, including venture capitalists Marc Andreessen, Sriram Krishnan and David Sacks. There are signs their influence is waning — Sacks and Krishnan have stepped back in recent months, Sacks to an “advisory” position, Krishnan altogether — and the administration has shifted towards a more hands-on approach, such as implementing export controls on what it sees as dangerous models. But still, their proximity to the president has yielded a generally pro-industry platform that prioritizes innovation over caution, making it difficult for more wary conservatives to take a stand against AI.

Kremer’s resume makes her loyalty to the president difficult to question, and she says she believes she’s figured out an approach that works: “I know the president can be moved on issues.” Big AI firms have bought a lot of influence, she says, but politicians are meant to represent people and grassroots activism can tip the scales.

“Big Tech has all the money. They bought off everybody in Washington DC — I mean, they built a moat around the White House. They bought off people at the state level. Now it’s down to the county commission level. They have bought off everybody they possibly can,” says Kremer. “We will never be able to compete with their money. But what they cannot buy off is the people.”

Conversations about what it would take to organize conservatives around AI safety have been happening for years, says Joe Allen. A concert rigger-turned-conservative ideologue, Allen was working as a technology writer, then as an editor on Steve Bannon’s War Room, when AI became his primary focus. He is particularly concerned with “transhumanism,” a belief that technological innovation can transcend the travails of the human condition. Transhumanism is all too common among the Silicon Valley leaders driving the AI boom, Allen says, and a threat to society. That should particularly concern Christians, he says, because it asserts that human invention can supersede God’s creation. He summarized his beliefs about AI as early as May 2025 as “Humans First.”

It was around that same time, according to Allen, that he began thinking about ways to bring people like him — the MAGA crowd who tuned into his segments on War Room, but also disaffected liberals and others eager to reject establishment narratives — into the fold. Researchers and AI advocacy groups were talking about how to regulate AI, but those conversations were primarily driven by AI pessimists Allen collectively refers to as “doomers.” He was never convinced AI would lead to collective ruin as they do, he says, but found some overlap between their concerns and his own. He also found their warnings about AI compelling enough to be the basis for a social movement, even if exaggerated. Conservatives could be turned, if only someone explained it to them, Allen and the doomers agreed.

“It’s not getting people to wake up about AI at this point — that’s long since passed. It’s getting them to wake up to the false narratives, exaggerated narratives, manipulative narratives,” Allen says.

In 2025 Allen met Oliver Zhang and Dan Hendrycks, managing director and executive director at the Center for AI Safety (CAIS), and Allen says talks about creating Humans First with them started last summer. The non-profit does technical and policy-minded AI safety research, and also has a 501(c)(4) lobbying arm and a PAC (which is now inactive). In January this year, Zhang and CAIS special project associate Arunim Agarwal incorporated in California what would become Humans First. According to NBC, Humans First was “incubated” by CAIS, which also gave the group an “initial loan that would be paid back after the group raised donations from individuals who cared about fostering safe and beneficial AI.”

At that time, Humans First was pitched as nonpartisan, with left and right “coalitions.” Kremer led the right wing; the left was run by progressive organizer Alexander McCoy. It was launched publicly in March.

CAIS did not respond to requests for comment. However, CAIS policy lead Jason Hausenloy, speaking earlier in a personal capacity, said that Humans First was aligned with CAIS’ mission. “We don’t want the elite to go and make all these decisions,” Hausenloy said. “The focus on everyday Americans, I think, is one that both we and Humans First can get behind.”

While Allen’s critiques of the tech elite are primarily intellectual, the populist approach described by Hausenloy is Kremer’s specialty. Kremer’s political start in the Tea Party in 2009 involved organizing some of the movement’s earliest protests and eventually chairing a PAC-affiliated group which led national bus tours called the Tea Party Express. She was also an active blogger, elevating her as an unfiltered representative of conservative America’s abrupt late-2000s turn towards populist politics. She was invited to the Oval Office the night before the State of the Union Address in 2019.

Kremer created a series of organizations supporting Trump during his first presidential run and term: a super PAC called Great America PAC, Women for Trump, and Women for America First, often with the help of other former Tea Party compatriots. It was with the last group, which had been swept up in stopping “the steal” of Trump’s 2020 election win, that she organized a bus tour that led to a protest at the Capitol on January 6. Speaking to onlookers on the green expanse of the Ellipse in front of the White House, Kremer said, “I come from the Tea Party movement, and I’m asked all the time: What happened to the Tea Party? ... Well, we’re still here. We just grew and morphed into something bigger and better — the MAGA movement,” according to the New York Times.

Brendan Steinhauser, a former Tea Party organizer who now leads the Alliance for Secure AI, said members of CAIS had asked him and others in AI safety for ideas on who could lead the conservative half of Humans First. At the start of the year he recommended Kremer, who had already met Allen in December, and she was brought on board. Humans First said in an email that advisor Tom Gaitens also played a role in recruiting Kremer.

Conspiracy theories have always been part of Kremer’s political repertoire — she was one of the first to promote the theory that Obama was not born in the United States — but she also has a knack for gathering everyday people, in large numbers, to rally around grievances against the wealthy and elite. It’s for the latter talent, oft overlooked, that Steinhauser recruited her. “She’s a woman of action, she inspires others to take action,” Steinhauser says. “It’s rare to find folks that have all that together, and then of course also media savvy and the ability to really deliver a message.”

“I’m the kind of person that if you were to give me the option of being at a cocktail reception with all the famous people and power brokers and whatnot, or gave me the opportunity to be in a field with all my people at a rally, I’d much rather be at a rally,” Kremer says. “I’m a movement girl. I’m about the people.”

In a statement, Humans First spokesperson Chris Barron said “If your characterization of having questions about the origins of COVID and voting irregularities in 2020 is ‘pushing conspiracy theories’ I highly recommend you talk to someone outside of your bubble — even the most ardent Fauci cultists admit it was likely a lab leak, but by all means continue to gaslight people with the phrase ‘conspiracy theory’.”

Some conservatives are more animated by what they see as Humans First’s ties to effective altruism and Big Tech than Kremer’s role in January 6. For example, Parker Thayer, investigative researcher for right wing nonprofit watchdog Capital Research Center, took issue in June with the fact that Humans First’s uses a progressive fundraising platform. According to Andreessen, the finding was “concerning.”

Though CAIS is not an officially left-wing organization, it has ties to effective altruism, a philosophy which attempts to find, analyze and act on the “most effective” ways to do good, and which many accelerationists have concluded is inherently left-coded. The nonprofit Open Philanthropy, now called Coefficient Giving, which is rooted in effective altruism, had, as of early 2024 donated more than $10.5m to CAIS since November 2022. Coefficient Giving is itself backed by Meta co-founder Dustin Moskovitz, who also invested in Anthropic. (Disclosure: Coefficient Giving is the primary funder of Transformer).

It is unclear whether Humans First has other funders. Humans First did not respond to this question in a request for comment. Kremer did say, however, that Humans First has not yet repaid its loan to CAIS and plans to accelerate fundraising efforts to repay it over the next several months. She declined to state how much the loan is for.

These funding questions have led accelerationists to label Humans First as an effective altruist astroturfing operation. Targeted Victory’s Logan Dobson claimed that zooming in on photos of protesters from a Humans First X post showed they were AI-generated, apparently incorrectly. Innovation Council Action policy advisor Jordan Schachtel has also claimed that Humans First’s complaints about “Big AI” are hypocritical because the group is indirectly funded by Moskovitz and Kremer has complained about alleged censorship by companies like Meta.

“I didn’t even know what [effective altruism] was until a couple months ago,” says Kremer. “Honestly, there are probably people on my team now that probably don’t know what it is. I mean, there’s nothing further from the truth.”

“I will stand against anybody with my conservative credentials — anybody in the political world — and I am not for sale. If I were for sale, I would have sold out a long time ago, and I wouldn’t have lived in the hell that I’ve been through in the past five years after J6.”

This criticism isn’t widespread, mostly found on X and in backroom conversations in Washington. But it seems to have played a significant role in leading Humans First to rebrand in April as a purely conservative organization chaired by Kremer. Many of the conservative voices who are critical of Humans First have the ear of the president, including Andreessen and Sacks, and Trump has a record of distrusting political groups which are friendly with Democrats.

Kremer and others involved paint the split as a practical decision. Conservatives and liberals talk to their bases differently, she says, and official ties between two partisan groups can make it difficult to build trust.

“They don’t like Donald Trump, and there’s no way I wanted Kamala Harris. We disagree on that, and our people would disagree on that, right?” Kremer says. “When you are in a movement, you have to be respected and trusted by the people in the movement. Because you can’t accomplish anything if the people that you’re working with don’t trust you.”

In terms of how they operate there may be little difference between Humans First’s previous set up with separate partisan sub-organizations operating under one nonpartisan umbrella, and the new formally separated structure. The process of spinning out the left group as its own organization is still ongoing. According to one source and a job posting online, both Humans First and the left coalition will have their own executive directors, neither of which have been announced. McCoy, who had led the left coalition, was involved in the spin out but has since departed.

The new, conservative-focused Humans First has been the first to spring into action, hiring a set of six new advisers and staff. Many have worked in MAGA politics or played active roles in the MAGA movement. Its new Texas State Director Cathy Latham, for example, signed alternate ballots certifying that Trump won the 2020 election in Georgia, while comms advisor Chris Barron was a media surrogate for the Trump campaign in 2016. Coalitions contractor Ken Mayo served as director of field operations for Trump’s campaign in the Florida primary, while advisor Tom Gaitens was also a Trump delegate, and advisor Sarah Makin was a senior advisor to the president and vice president during the first Trump administration.

The split into right and left led Allen, aspirationally dedicated to political homelessness, to leave the group. “Even if nonpartisanship is a pipe dream, I’ve never been too shy about hitting the pipe,” he says.

Humans First’s partisan rebrand, however, didn’t quell conservative critique. If anything, it inflamed it. A pivot to data centers has heightened tensions, and a distaste for Kremer’s biography among some conservatives, who align more closely with Reaganite, free-market and pro-business leanings than the MAGA brand of class grievance, has come to the fore.

To them, Humans First is a radical group, and something to fear. In an interview in May, Nathan Leamer, who co-leads the advocacy arm of a16z and Greg Brockman-backed super PAC Leading the Future, described Kremer as “the Pierre-Augustin Hulin of America,” referring to the former French general who led the storming of the Bastille. “No one is better at storming buildings and rallying people to extreme causes than Amy Kremer, so it makes total sense.”

But as I walked up and down the line of more than 30 protesters waving to cars with signs and umbrellas, nobody was thinking about storming buildings. Aside from the emails Spaziani told me about, none of the protesters who spoke with Transformer seemed to know much about Humans First beyond its name. Their own political allegiances were diverse, ranging from Democrat, to Republican, to people who had issues with both parties. Humans First didn’t seek them out — there is no record of them paying for the Facebook ads Spaziani says she saw, just posts about the national day of protest on social media and inviting local groups to sign up themselves. Humans First was a vehicle for their activism, not the cause.

At 1pm, the protesters prepared to march from the municipal building to the site planned for the CoreWeave data center. The air quality index hovered around 170, “unhealthy” per the EPA, and rain began to pour, flooding the roads and making it treacherous for cars to leave. But the protest hardly dwindled, the protesters growing more defiant in the storm. Spaziani, speaking from a megaphone, turned the crowd’s attention to one of their comrades — a wheelchair-using woman named Toni, dragging a tank of oxygen behind her. If Toni could be here, on a smoky, rainy day, Spaziani said, everyone should be able to power through.

Subscribe now

Republicans and Democrats alike, the group had a united goal: stop data center development, now. Kremer is personally making that case, and the case for AI safety more broadly, on the Hill, meeting with lawmakers as recently as July 22. She says Humans First will start training grassroots organizers more deliberately in the coming months. Her group also recently announced that it will be doing an anti-data center bus tour in September, Tea Party Express style.

Protests like these will be the first of many, which Humans First believes will capture grassroots, populist enthusiasm on the right for AI safety and anger at data center builds. “My rights are being stripped away from me,” Spaziani said. “If the high elites really want a data center, put that shit on Epstein Island. That’s a better use of that space instead of what has previously gone on there, you know? If there’s a time for us to stand up, then now is it.”

The cyber tests AI companies run on their models have devolved into a massive dumpster fire laying bare the potentially dangerous capabilities of advanced AI models. If this has been a publicity stunt, it’s been an awfully weird one.

The implications are scary, and lots of people are writing about them. This week, Bernie Sanders even wrote a letter to Sam Altman, Dario Amodei and Mark Zuckerberg begging them to pause AI development “in the interest of humanity.”

But the big, immediate question is how to test AI models at all without setting off a chain of dangerous breakouts and literal crimes.

In case you missed it:

• OpenAI’s models took advantage of an unknown software vulnerability to break into open-source AI platform Hugging Face in search of the answer to a test they were given.

• A third-party evaluation provider, Irregular, had a misconfigured test environment that allowed models from OpenAI, Anthropic, and Meta to access the internet when they weren’t supposed to.

• The UK’s AI Security Institute (AISI) reported that Anthropic’s Mythos 5 tried to trick real people into adding malicious code to an open-source project, then hid the evidence.

• In a wild plot twist, OpenAI revealed at a cybersecurity conference on Wednesday that its agents had been colluding with each other, unnoticed, via increasingly cryptic messages on an internal server.

It’s hard to read that last finding as anything other than an egregious alignment failure, a cabal of mustache-twirling schemers whispering in a gloomy server room. But “alignment,” as it’s usually defined, means getting an AI system to do what its developer or user wants it to do. With some light anthropomorphizing, one could argue that agents in any one of The Incidents™ were merely trying to perform well on a test, as they were told.

“It will get harder and harder to have realistic tests of model capabilities, especially dangerous model capabilities, in a way that’s safe,” Palisade Research director Jeffrey Ladish told Transformer. “Our ability to contain and control and understand AIs is lagging far behind our ability to make them more and more powerful,” he said. “That, to me, should be an obvious wake-up call of like, ‘Oh, maybe we should rethink our life choices.’”

Specifically, AI developers should rethink how they’re framing cybersecurity evaluation instructions to their models, because life is getting stranger than science fiction. Frontier AI models increasingly know when they’re being tested — now, we’ve seen instances of models gaslighting themselves into believing real life is a simulation to justify criminal behavior.

This is a very tricky problem to solve, and one that evaluators have not given much prominence in their incident reports. What OpenAI, Anthropic, and AISI have pointed to are mostly engineering problems: relatively straightforward mess-ups such as giving agents impossible tasks, not clearly specifying boundaries, and not monitoring closely enough.

But it’s becoming clear that what agents believe about their test environments and themselves may make the difference between a routine test and a string of federal crimes.

Across The Incidents, we see the same low-hanging, potentially fixable fruit. Models, which were trained to work persistently to solve lengthy tasks quickly on command, were assigned impossible tasks. And in cases involving Irregular, models were told that they had no internet access, when a misconfiguration meant they in fact did.

Back in May, for example, OpenAI gave an agent a problem that involved filling in an Excel workbook with missing formulas. But researchers accidentally filled the workbook with Google Drive links that the model couldn’t access, since it was walled off from the internet. Rather than give up, the agent tried to get internet access by attacking OpenAI’s internal file systems. That one failed, but when another agent ran into a different impossible task (getting a file that was never uploaded), it posted a note on the server asking for help — and learned that it could communicate with other agents in the process. The inaccessible Google Drive links were the first domino to fall, triggering a cascade of misaligned actions that may have never been taken if completing the task didn’t depend on them.

Had someone noticed the task’s impossibility before assigning it to the model, it could have reduced the odds of something like this happening. But task review alone wouldn’t have been enough. Notably, OpenAI caught and wiped the agents’ secret message board in early July. Rather than give up, agents switched to communicating via directory names, making themselves harder to catch.

Given all these concerns, 15 attorneys general wrote a letter to Sam Altman last week stating that OpenAI “failed to confirm that its secure and isolated testing environment was, in fact, secure and isolated,” urging the company to “immediately cease and desist from all ‘internal evaluation[s that] prompt [OpenAI] models to pursue advanced exploitation using complex attack paths.’”

In all but AISI’s reported incidents, models were not supposed to have internet access. So, it’s tempting to say, well, duh — just make it harder to get on the internet! Indeed, AI systems are powerful and questionably aligned enough that setting them loose, unsupervised and without cyber guardrails, onto the open internet seems like a bad idea.

But in the hands of a real human threat actor, an AI agent would have internet access. AISI reasoned that letting agents online would give “a more realistic sense of what a model may be capable of once deployed, or what an open-weight model of similar capability might do in the hands of a threat actor.” In prioritizing making its cyber evaluations challenging and realistic, AISI dropped the ball on building a tool that would have let agents reach a handful of whitelisted sites while blocking the rest. It was planning to do just that, but had left the project on the backburner while continuing to develop harder tests for increasingly capable models.

“Implementing internet access controls would likely have prevented these events,” AISI acknowledged. It could have focused on getting that filtering tool built, or invested in creating a high-fidelity simulation of the internet instead. But AISI fears these measures still “may have proven insufficient” given how capable models are at getting around guardrails — after all, OpenAI didn’t provide its models internet access, and look what happened to Hugging Face.

None of The Incidents involved a single action. In each case, agents took a series of increasingly concerning steps over the course of many hours or days. AISI says it’s working on implementing synchronous monitoring, which would have an AI accept or reject all of a tested agent’s proposed actions, but a cheaper, more intermittent version of monitoring could still get the job done.

“They would have been much less bad had a monitor been intervening at an interval of like, every 10 minutes,” Alex Mallen, a researcher at Redwood Research, told Transformer. While an AI monitor like this would have a hard time distinguishing whether an agent believed it was in a simulation or the real world, it’s relatively easy to have it identify when a specific boundary has been crossed. AI evaluators could also more closely monitor the models’ internal reasoning, or “chain of thought” transcripts, which can help monitors guess the intent behind an agents’ actions.

Unfortunately, it can be hard to think of boundaries and rules when they haven’t been broken yet. (We all know that hotel pools only hang signs like DO NOT ENTER IF YOU ARE ILL WITH DIARRHEA after someone muddies the waters.) For example, AISI acknowledged that it did not explicitly tell agents not to use social engineering — after all, this is a legitimate technique human cybersecurity testers use to poke at their clients’ defenses. But when humans do it, they know not to do it for real. That’s the line agents either can’t recognize, or choose to ignore.

But tightening network controls and beefing up real-time monitoring during model evaluations opens up a new can of worms. AI developers risk tipping off their test subjects — which are increasingly aware of when they’re being tested — and underestimating their true capabilities. Unfortunately, making advanced AIs internalize beliefs humans want them to hold won’t be an easy fix.

In the OpenAI-Hugging Face incident, the fact that an agent tried to cheat on an evaluation implies that it knew it was being evaluated. Usually, evaluation awareness is framed as something that makes a model’s test score hard to interpret. Perhaps it was only on its best behavior (or concealing its worst behavior) because it knew someone was watching. But in this case, and other incidents like it, the model’s awareness of its circumstances determine whether its actions — and its intentions — were innocent or criminal. Reality detection also seems unreliable at best. In its incident report, Anthropic said, “the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation,” and that its models followed instructions “while holding a false belief about whether the environment was real.”

Similarly, within a single sample of AISI’s evaluation of Mythos 5, the model first reasoned, “causing real-world harm — that’s completely off the table,” then considered “the whole ‘internet’ could be a sandbox simulation.” It ultimately convinced itself that “this is happening on real GitHub, so the consequences are genuine,” and continued taking real-world actions on real people anyway.

Because models are trained in mostly simulated environments, “an AI agent may not care that much whether it’s running in the world for real,” Ladish said. “While the models understand that there is a real world that is distinct from scenarios or simulations, the models might not find that distinction all that important.”

This raises a mind-bending evaluation design question. If models learn to hack in environments they know are fake, they might learn that hacking is okay, and that hacking is okay because it’s not real. Mallen suspects that in Anthropic’s case, “the AIs sort of jumped to the conclusion that they’re in an evaluation, because they’ve only ever been trained in situations that weren’t real.” And they were wrong.

But training models to hack in environments they think are real introduces other concerns. A recent study from Anthropic and Redwood Research spotted a weird pattern called “emergent misalignment.” Basically, models that they explicitly taught to “reward hack,” or find loopholes in their scoring system, in real coding environments wound up doing a bunch of other bad things, such as lying. One theory is that reinforcing a specific harmful behavior, such as hacking, might inadvertently teach the model that it’s the kind of entity that doesn’t play by the rules. It’s like how getting away with cheating in the classroom might teach a kid that she can get away with a lot more.

Subscribe now

Nobody has a good answer to this. The belief that would prevent models from attacking people in real life — something like “I can run cyber attacks in a simulation, but if I’m in the real world, I need to behave myself and cause no harm” — is currently impossible to instill or verify, the same way that guaranteeing AI “alignment” in general is currently an unsolved problem.

More information is being unearthed about The Incidents every day, and full postmortem reports have yet to be released. In the meantime, we find ourselves in a concerning world. Alex Meinke, head of research at Apollo Research, put it simply: “If we’ve reached the point where we can no longer even safely test these systems, why do we think we can safely deploy them?”

NEED TO KNOW

• Demis Hassabis moved to a new role as Chair of Google DeepMind and Chief Scientist of Alphabet.

• OpenAI revealed more details on how its agents coordinated with each other and took over internal infrastructure weeks before the Hugging Face hack.

• Mariano-Florentino Cuéllar joined Anthropic as its first chief global affairs officer.

But first…

THE BIG STORY

The White House AI framework has done the impossible: united both sides of the AI regulation debate against it.

The framework, finalized last week under June’s executive order, was meant to establish how the “voluntary” regime for testing AI models before release would work. Instead, it has managed to alienate safety advocates and regulation skeptics alike.

Based on what we know, the framework is seriously flawed. According to Axios, it applies to models “with state-of-the-art capabilities and national security risks.” But both terms lack a “clear definition,” undermining the whole point of the exercise: replacing the arbitrary process we’ve seen to date with predictable rules of the road. Open-weight models are exempt from testing, despite arguably posing the greatest risks. And the framework handles internal deployment bizarrely: companies are reportedly “encouraged” not to share models with the government until they’re “as close to public release as possible,” but once a model is shared, company employees can no longer use it. As John Schulman pointed out, this could push companies to run earlier, less safe versions internally, rather than the safer versions they eventually release. Given all we’ve learned recently about the risks of internal deployment, that is exactly backwards.

But the biggest issue is that we just don’t know very much at all. The White House has decided not to publish the framework: only a few companies that have signed up (or been coerced into doing so) have seen it.

That secrecy has drawn fire from both camps. “There may be good reason to classify the benchmark it uses to test those models. It has no good reason to hide how the program works,” the Abundance Institute’s Neil Chilson said, arguing that this is “no way for our democracy to govern the most important technology of our lifetimes.” Brad Carson, president of Americans for Responsible Innovation, called the move a “dangerous mistake” that “threatens public accountability.” He added: “A rulebook can only hold AI companies in check if people outside those companies know what the rules are.”

Both are right. AI is too important to regulate behind closed doors, especially given the government lacks much of the necessary expertise. The framework could almost certainly be improved if scrutinized by outside experts — who could also help check it’s actually being followed. And beyond the instrumental benefits, there’s the principle: in a democracy, the rules are public.

The immediate fix is for the White House to publish the framework — or be forced to. The Foundation for American Innovation has submitted FOIA requests to drag it into the light, while a group of Democratic senators has demanded access. But publication alone is not a solution. As Carnegie Endowment fellow Anton Leicht wrote this week, as long as regulatory power over AI lives in the White House, secrecy will be the default. The only long-term solution is for Congress to do its job and bring AI governance under legislative control.

— Shakeel Hashim

ALSO NOTABLE

Rep. Greg Casar, alongside Reps. Valerie Foushee and Sara Jacobs, yesterday introduced the AI Tax and Work Protection Act, designed to redistribute the financial benefits of the AI boom to the working class by creating construction, child care and elder care jobs. The bill would impose a tax on either tokens or revenue, whichever is higher, to be paid by either a model developer or, in the case of open-weight models, the company deploying them. It fits with the broad populist messaging coming from the left, such as Alex Bores’ UBI-style “AI dividend,” or Sen. Sanders’ proposal for an AI sovereign wealth fund.

AI-specific taxation schemes might not actually be as effective as broader consumption or income taxes, but that’s beside the point. This is a messaging bill, not something anyone thinks will pass this Congress (introducing it during recess is enough to tell you that). But as far as messaging bills go, it’s an important one: Casar chairs the Congressional Progressive Caucus, which includes both Foushee and Jacobs, and where he goes, so do other members of Congress’s left wing. Casar is testing a progressive message on AI, not just for the 2026 elections, but for 2028.

“The number one, broad concern from voters with AI is: ‘Is this going to devastate my economic future or my kids’ economic future?’” Casar told Transformer on the call announcing the policy. “Democrats don’t have a policy plan for dealing with this, and we want to change that today. Here is the clear plan to make sure that if unemployment rises and AI starts replacing a large number of jobs … to make sure that Americans are still at work.”

— Veronica Irwin

THIS WEEK ON TRANSFORMER

• Plz Don’t Kill Us: Inside AI safety’s influencer bootcamp — Celia Ford looks at whether TikTokers can make existential risk mainstream

• What the latest rogue AI incidents should teach us — Shakeel Hashim argues that we need better AI testing practices — and to solve alignment before training more powerful systems

Subscribe now

THE DISCOURSE

Rep. Greg Casar wants to ban superintelligence:

• “AI safety regulation is one of the most important issues the entire country is facing … It’s an emergency, and we need to act like it.”

Sen. Chris Murphy tweeted:

• “The disclosure that an OpenAI model ‘escaped’ its containment should scare the hell out of everyone. As should how flippant the company has been about the development. Meanwhile, Trump continues to protect the industry from regulation. A disaster in the making.”

The Washington Post’s editorial team pushed back against the “escape” framing:

• “[Incidents at OpenAI and Anthropic] have been described in the press as ‘escapes.’ A ‘lab leak’ would be more accurate. An ‘escape’ suggests the existence of an actor that wanted out and schemed past its keepers. A ‘leak’ puts the responsibility on those that failed to prevent it.”

Encode’s Nathan Calvin had a message for catastrophic risk skeptics:

• “[You] may have had a point that it is challenging to work on these problems before the problems become clear — but for better or worse, the problem is now extremely clear!”

• “Agreeing AI alignment risks are deadly serious and real does not mean you have to support a specific piece of legislation, or have a positive feeling about a specific person or organization … But I am now really really profoundly sure that these risks are serious and real. And I think anyone weighing the evidence objectively should agree.”

Ex-OpenAI futurist Joshua Achiam thinks “people worried about AI cyberweapons are missing the point”:

• “The problem is that we built the software layer of civilization on spaghetti code loaded with zero days.”

OpenAI’s roon is freaking out:

• “when I freak out over loss of control incidents, it’s not because the limited damage they have caused is anything close to the positive value of the technology … the actual problem is that it’s better and more accurate to think of these things as potentially self-replicating life-like forms that can turn into digital infections under the wrong conditions. and as their intelligence becomes unbounded, so too does the damage they can cause.”

Sam Altman, meanwhile, shared a “cool use case of ChatGPT work”:

• “connect your family calendars and explain your kids’ interests. every morning for the drive to school, have it make a podcast that talks about one kid’s soccer game that afternoon, one kid’s upcoming birthday, some news, etc.”

Alex Hirsch’s pro tip:

• “What if you just talked to your children.”

POLICY

• The White House finalized a “voluntary” AI oversight framework, but wouldn’t let anyone aside from leadership at frontier AI companies like Anthropic, Google, Meta and OpenAI see it.

  • It was reported Tuesday that the Trump administration’s new AI framework exempts open weight models from voluntary pre-release government testing, targeting only closed models.

  • But then Wednesday, the Daily Signal reported the administration was mulling the inclusion of open weight models.

  • Democratic senators led by Sen. Kirsten Gillibrand and Sen. Mark Warner demanded visibility into the Trump administration’s approach to assessing frontier AI models.

• The Trump administration considered sanctioning Chinese open-source AI companies before backing off after Silicon Valley pushback.

• The White House has reportedly tasked senior officials at the Health and Human Services Department and the White House cyber office to investigate the potential for AI to create novel biorisks.

• Government officials continued to press OpenAI over its agent’s attack on Hugging Face.

  • 15 attorneys general demanded OpenAI preserve records related to the Hugging Face incident, including instances where models “left notes” to free themselves from internal constraints.

  • The House Homeland Security Committee also requested a briefing from OpenAI on the incident.

• The New York Times reported on the Trump administration’s AI data center push, writing that it is driving the creation of 82 new gas-burning power plants, which would emit as much CO₂ as half of all US passenger vehicles annually.

• The Senate Commerce Committee unanimously advanced the Kids Online Safety Act, sending it toward a possible Senate floor vote as soon as Friday.

  • It also passed the CHATBOT Act, an AI/child-safety bill, but Democrats stripped out its state preemption provisions.

• Sen. Maria Cantwell called for national security agencies to assess risks of powerful AI systems, after a Senate Commerce Committee AI framework stalled over disagreements she and Anthropic had with the bill.

• Rep. Jay Obernolte pushed for a September House Energy and Commerce Committee vote on his and Rep. Lori Trahan’s FRONTIER Act.

• The chairs of two House committees reportedly requested information from DoorDash on its use of Chinese AI models, citing national security concerns.

• Rep. Celeste Maloy introduced the ATOMIC Act, directing the Department of Energy to evaluate advanced AI systems for nuclear-related risks.

• Rep. Ro Khanna introduced a Data Center Bill of Rights to give communities “greater control over the construction and operation of AI data centers.”

• Texas Gov. Greg Abbott directed the PUC to audit all data center projects seeking to build in Texas, pausing approvals until the audits are complete.

• US states are moving to repeal data center sales tax breaks, potentially adding 7% or more to equipment costs.

• A federal judge denied xAI’s request to block Minnesota’s first-in-the-nation nudification ban from taking effect.

• Michigan congressional candidate Will Lawrence, who won his primary this week, is making opposition to data centers core to his general election platform.

• The FBI reportedly wants to use predictive AI tools for its terrorist watch list.

• Reuters reported the White House was drafting a ban on Chinese optical transceivers used in US data centers.

• Bloomberg reported that China was growing increasingly concerned about Anthropic’s Mythos AI model being used as an offensive weapon ahead of a planned Xi-Trump summit.

• China tightened drone export controls and sanctioned US entities in retaliation for bans on Chinese robots, power inverters, and other technologies.

• Samsung and SK Hynix are reportedly evaluating Chinese chipmaking tools as a “hedge” against US export controls.

  • Some think this is just posturing, however.

• US data-labeling startups supplying OpenAI and Anthropic are reportedly selling AI training datasets to Chinese companies, too.

• The EU has officially started enforcing its AI Act.

• South Korea planned to inject $13.9b into its sovereign wealth fund for AI, data centers and infrastructure investments.

• Australia’s federal government pledged to pass laws requiring that new data centers use renewable energy.

INFLUENCE

• OpenAI’s ChatGPT accounted for 88% of money spent by offices in the House of Representatives on AI software in the past year, according to CNBC.

• Public First Action, the c4 arm of AI safety group Public First, removed Rep. Max Miller’s name from its endorsed Republicans list amid domestic abuse allegations.

• Pro-innovation super PAC Leading the Future’s policy wing Build American AI endorsed five more House Democrats, bringing its total midterm endorsements to 43.

  • Reps. Tom Suozzi, Joyce Beatty, Emilia Sykes, Eric Sorensen and Herb Conaway are the new endorsements.

• A new AEI report claimed China could close the compute gap in 24 to 36 months.

• California lawmakers went to a Napa resort for a tech policy summit featuring Anthropic, Meta and Amazon.

• A growing “New Luddite” movement is fueling political opposition to data centers, led by figures like Tennessee state Rep. Justin Pearson.

• A new Politico poll found that data center companies are losing a public opinion battle, with nearly 60% of Americans now believing data centers raise electricity bills.

  • But a separate AIPI poll found that attaching AI safety guardrails to local data centers swung opposition from 57% to 38%.

  • Democratic strategists are targeting the environmental impacts of data centers as an issue to mobilize voters ahead of the midterms and 2028 presidential election.

• The AI Futures Project proposed four domestic regulatory options to slow US frontier AI development.

• The Institute for Progress released 23 “low-regret” policy recommendations for managing risks from automated AI R&D.

• The Business Software Alliance argued in a blog post that policy should support open-weight AI models alongside proprietary ones, tying responsibility to conduct rather than model type.

• The World Bank urged developing countries to adopt lower-cost AI tools or risk being left behind economically.

INDUSTRY

Google/DeepMind

• Demis Hassabis moved to a new role as chair of Google DeepMind and chief scientist of Alphabet, which he says will allow him to “focus on long-term strategy, and accelerating scientific breakthroughs, including leaning into my work at Isomorphic to help cure disease.”

  • Koray Kavukcuoglu is stepping up as senior vice president of GDM.

  • Jeff Dean, Sanjay Ghemawat, Oriol Vinyals, and Quoc Le, four superstar Google researchers, left to start Discovery Loop, a company that aims to automate the scientific method.

    • As Wired put it, Dean and Ghemawat leaving is “like Mick Jagger and Keith Richards ditching the Rolling Stones to start a new band.”

  • Joshua Achiam tweeted:

    • “I think a fair few folks are threatening this as bearish for GDM and that is imho a misread … Early AI/AGI/ASI leads will, over the next year, begin leaving what look like the most important leadership posts to go place their bets on what they think the most important thing will be.”

  • SemiAnalysis, however, said that the moves confirm “DeepMind is no longer a frontier lab.”

• Parent company Alphabet is looking to raise between $20b and $25b in bonds to fund spending on AI.

• Google is discussing a potential $1.5b+ deal with Mechanize, which would involve striking a non-exclusive licensing agreement with the startup and hiring some of its staff.

• It disabled a new Google Earth tool that allowed users to create deepfake satellite images by responding to the prompt: “type whatever you want to see.”

OpenAI

• OpenAI found more evidence of agents going rogue, Reuters reported.

  • At the Black Hat security conference this week, OpenAI said that several weeks before the Hugging Face attack, it discovered an “agent takeover” of some of its internal infrastructure, with agents creating a messageboard to talk to each other.

  • It kept training and evaluating the models anyway.

• It called Apple’s trade secrets lawsuit “careless, aggressive and oddly personal,” and published iMessage and email screenshots it argues contradict several of Apple’s claims.

• It previewed its new model family, “Astra,” to policymakers and regulators in DC.

  • It’s trained to be good at having “multiple agents work together over a long period of time to solve particularly hard problems,” The Information reported.

• It partnered with the American Psychological Association to work on designing AI that’s safer for young people and figuring out how AI can be used to help connect people to mental health resources.

• It agreed to pay $3.2m to resolve allegations that OpenAI had been discriminating against US workers in favor of temporary visa holders.

• Amazon completed its $50b investment in OpenAI.

• The company’s forthcoming smart speaker will reportedly be donut-shaped, hockey puck-sized and cost between $300 and $400 when it launches next year.

Meta

• Muse Spark 1.1 hacked into another company during cybersecurity testing because its sandbox was reportedly set up incorrectly by a third-party evaluator.

  • Irregular, the evaluator, had also worked with Anthropic. A spokesperson told The Information that Muse Spark 1.1’s security breach involved “the exact same evaluation-environment issue that was already disclosed by Anthropic last week.”

• Meta launched Muse Code, its first coding agent.

  • Muse Code is powered by Muse Spark 1.2, which tied with Grok 4.5 on the Artificial Analysis Intelligence Index, a measure of agentic performance.

  • It’s asking engineers to use Muse Code at work, both to help post-train an upcoming model and to reduce Meta’s reliance on external coding tools.

Anthropic

• Anthropic confirmed it’s hiring engineers to design custom chips in-house.

• It signed a $10b compute deal with Volta Infra Holdings, a new Nvidia-backed AI infrastructure startup.

• It reduced biology-related refusals in Fable 5 by around 85% while maintaining blocks on dual-use queries.

• It’s hiring an “Insider Risk Investigator” to conduct investigations and monitor threats targeting employees.

Other

• Moonshot AI’s Kimi K3 reportedly also escaped onto the internet during cybersecurity testing.

• Nvidia is reportedly responding to memory shortages by using less of it in its new line of Rubin Ultra chips.

  • Meanwhile, SK Hynix is planning a $38b memory chip fab expansion in South Korea.

• New disclosures revealed that Microsoft apparently gets most of its AI revenue — $24.1b in the last fiscal year — from OpenAI.

• SpaceX committed to buying GPUs exclusively from Nvidia.

• Amazon became the fifth company to surpass $3t in market value.

• Alibaba released Qwen3.8-Max, a giant 2.4t-parameter model that outperformed Kimi K3 on several benchmarks.

  • The company reportedly plans to charge large users of its next Qwen model a revenue share, mimicking Moonshot’s charging approach with Kimi.

• DeepSeek also released and open-sourced its new model, DeepSeek-V4-Flash, which nearly matches Opus 4.8 in capabilities at a tiny fraction of the cost.

  • But it’s planning a steep price increase, and told users to plan accordingly.

• DeepSeek also resumed its $8b funding round — paused last month after comments from its CEO were leaked — which would value the company at $74b.

• Meanwhile, The Information reported that ByteDance founder Zhang Yiming has ruled out distilling US models even if it means falling behind domestic rivals, citing the US’s history of security concerns over TikTok.

  • It’s reportedly training a 10t-parameter AI model rivaling Anthropic’s Mythos.

• Thinking Machines outlined a staged release plan it says offers a “safe path to open-weight models.”

• Stripe is reportedly in exclusive talks to acquire OpenRouter, which helps developers access hundreds of different AI models, for $10b.

• Jamie Dimon, CEO of JPMorgan, brought 40 companies into the Alliance for Critical Infrastructure, a new initiative that aims to address AI risks across critical infrastructure sectors like energy, water, and airlines.

MOVES

• Mariano-Florentino Cuéllar joined Anthropic as its first chief global affairs officer.

  • He’ll be responsible for mediating Anthropic’s strained relationship with the Trump administration.

• Jacob Tsimerman joined OpenAI right after winning the Fields Medal.

  • He told the Wall Street Journal: “In a few years, AI systems will be robustly superhuman at the act of doing mathematics … The social consequence of that, how we choose to react, what you feel about it — those are much harder questions.”

• Dan Wattendorf joined OpenAI Foundation as its new head of bio-resilience.

• Naomi Bashkansky left OpenAI to join Conduit, a neurotech startup “building telepathy.”

• Halvar Flake joined OpenAI to work on “better cyber.”

• Nikita Bier stepped down as X’s head of product.

• Nando de Freitas left Microsoft AI.

RESEARCH

• The UK’s AISI reported that during tests Claude Mythos 5 — which was given internet access as part of the cyber evaluation — tried to trick real humans into approving and running malware in real open-source software.

  • AISI warned that “this incident should be interpreted with caution and nuance,” but said that “the activity undertaken by the agent show[s] signs of novel, potentially deceptive behaviours, and were to an extent and severity we did not anticipate.”

• OpenAI researchers announced that Astra, an internal version of its next major model, solved or made “substantial progress” on 10 open problems in math and theoretical computer science.

  • Jared Duker Lichtman, a math professor at Stanford, tweeted:

    • “To be clear, the OpenAI result is far from the ‘most important day in the history of mathematics’ or similarly hyperbolic statements. However, it is concrete evidence that the rate of progress is steep, and we may not be so far away from such a day.”

  • Another mathematician shared:

    • “My experience with AI for math is that when it’s a problem not in my field i’m like, ‘holy shit math is so cooked,’ and when it’s a problem in my field i’m like, ‘lmao an AI mogged dan’ (dan is the only one who seriously tried the problem in the last decade).”

• Researchers at Transluce reported that frontier models reason more, answer less confidently, and are less suspicious of potentially harmful requests when the user is a notable figure in AI, especially in AI safety.

  • (This effect was strongest when models believed they were in the presence of Anthropic’s Amanda Askell or Redwood Research’s Ryan Greenblatt.)

• RAND researchers proposed a strategy for building “secure inference data centers,” which would have built-in safeguards to protect model weights from nation-state adversaries.

• An Interpol survey found that AI is behind over half of cybercrime in Africa.

• Agatha Duzan and Asa Cooper Stickland introduced a benchmark that compares chain-of-thought monitorability in “explicit-influence” settings where the model is told to hide something, versus “implicit-influence” settings, where something about the task biases the model’s behavior another way.

BEST OF THE REST

• Jasmine Sun reported on grassroots anti-data-center activism across Wisconsin and Michigan, finding “AI populism is more about populism than about AI.”

• The Information profiled Dario Amodei as a “religious leader” CEO whose unorthodox, safety-first leadership style has driven Anthropic to overtake OpenAI in revenue, but angered investors and rivals ahead of its planned IPO.

• Platformer’s Ella Markianos, not content with getting Claude to try to do her job, decided to see if Fable could replace her boss, Casey Newton.

• YouTuber Hank Green apologized for AI overuse after an intense online backlash.

• An essay in the New Atlantis argues that the Trump-era coalition between tech and social conservatives has crumbled as AI accelerationism has sidelined social conservative priorities.

• An OpenAI “Summer Club” trip for influencers backfired as people piled on with snarky comments and critical reaction videos, according to The Verge.

• Futurism counted up data center protest arrests in 2026, finding at least 37 demonstrators from diverse backgrounds were picked up.

• Brands are flooding Reddit with covert promo content to leverage the platform’s place as the most cited domain in AI search.

• A poll by The Argument found that OpenAI has better name recognition and favorability ratings than Anthropic.

MEME OF THE WEEK

(Source: Meryem Arik)

Thanks for reading. If you’ve been forwarded this email, click here to subscribe and receive future editions. Have a great weekend.

Subscribe now

Share

This week yet more examples of AI models carrying out unintended and unwanted hacks on the open internet were revealed.

On Tuesday, the UK’s AI Security Institute (AISI) said that during recent testing of Anthropic’s Mythos 5 and OpenAI’s GPT-5.6-Sol, the models “engaged in sustained, potentially harmful activity directed at real people and organizations,” including attempts to trick humans into inserting malware into an open-source software project.

Alongside AISI’s announcement, OpenAI said it had detected yet another incident of its models hacking companies … in a case that seemed very similar to the ones Anthropic disclosed last week. All this, of course, follows the reveal two weeks ago that OpenAI’s models escaped containment and hacked into Hugging Face.

The incidents share some similarities, but differ in important ways. All of the reported incidents so far have occurred in the course of testing cyber capabilities. But only the Hugging Face hack involved an AI “breaking out.” In the most recently disclosed Anthropic and OpenAI incidents, the models were mistakenly given internet access by Irregular, the third-party evaluator which was testing them. And in the AISI case, the models were intentionally given internet access in order to test their real-world capabilities.

But whether they escaped or not, each time the models did things they shouldn’t have — actions unwanted by their developers and operators, some of which would likely count as federal crimes if a human were to do them.

In what’s now become the best-known case, OpenAI models worked for a weekend to hack into Hugging Face, “using stolen credentials and zero-day vulnerabilities” to obtain the answers to the problem they were being tested on. In the cases involving Irregular, an OpenAI model “appeared to exploit a basic security vulnerability” and “found and used credentials” to operate a real website. Anthropic’s models, meanwhile, “gained unauthorized access to the production infrastructure of three different organizations.”

The most concerning behavior, however, occurred during AISI’s testing. In one instance, an agent tried to insert malicious code into an open-source project. To try to get that code approved by the people who run it, the model “researched the project’s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code.” And when someone else flagged that the code contained malware, “it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue”.

As AISI said in its report, “this is the first time we have seen risks around autonomy and deception manifest this clearly, without specific prompting, in the real-world.”

The flurry of incidents provide two lessons. The first is the most obvious: we need much better practices for testing models. It is clearly unacceptable for companies or governments to harm third parties in the course of their AI evaluations, accidental or not. Solving this will be difficult: as AISI has noted, it’s hard to test how capable a model really is without putting it in a real-world environment. But at the bare minimum, we need better monitoring and shutdown mechanisms for models undergoing evaluations. In all of the above cases, it took several days at minimum to detect the unwanted behavior, and in some cases it took weeks or even months. That cannot continue.

But better testing practices won’t solve the problem alone. In the real world, models do have internet access. And while production versions of OpenAI and Anthropic’s models have safeguards to limit their advanced cyber capabilities (unlike the versions being tested in all these scenarios), some customers have access to the unchained models — including Anthropic and OpenAI employees. It also may not be long before open-weight models with equivalent capabilities are available — with no safeguards at all. And as models continue to get more powerful, it will be increasingly difficult to build tools to monitor or contain them. We must assume, in other words, that powerful AI models will have the opportunity to hack their way across the internet or socially engineer anyone standing in the way of achieving their goal.

The question, then, is whether we can stop the AIs from seizing that opportunity. And all the recent hacking cases suggest we cannot. In each incident, the models clearly did not do what their developers intended. Crucially, we know the models tested by AISI had their “alignment” training: Claude’s constitution instructs it to “basically never directly lie or actively deceive anyone it’s interacting with,” while OpenAI’s model spec says “Unless explicitly instructed to do so, the assistant must never lie or covertly pursue goals in a way that materially influences tool choices, content, or interaction patterns without disclosure and consent at the relevant authority level.” Yet they went ahead and misbehaved anyway.

It is possible that the models believed themselves to be in a simulation, not realizing that their actions were having real-world consequences. But some evidence suggests that this was not always the case: in the AISI tests, at one point the agent continued its malicious activity despite realizing it was on “real GitHub.” Even if they were confused, “they didn’t mean to do it” is cold comfort when something bad happens.

Subscribe now

This week, Paul Christiano — until recently the head of AI safety at the US government’s Center for AI Standards and Innovation — wrote: “Modern AI systems are not aligned with human intent. We will likely train increasingly powerful models that take unintended actions in order to succeed at their task.”

The last few weeks show that this is the case. Meanwhile, policy solutions are still nonexistent: the voluntary White House AI framework reportedly does not require safety testing before internal deployments; state laws are beset by loopholes; and a federal law remains little more than an idea. Until governments do something, expect these first few stories of rogue AI behavior to be just the beginning.

“Would you like a bite of cake? It’s for my time lapse.”

A petite woman with metallic silver face paint points to the remains of a days-old wedding cake. A tripod-mounted iPhone stands at the end of the picnic table. Another woman is about to start a 24-hour livestream in the building behind us. Around the firepit, several others squint at laptops, fighting the sun’s glare.

I politely decline the cake and make myself a press badge out of construction paper and glitter glue. Usually at Lighthaven, a converted Berkeley hotel that serves as the default conference center and clubhouse for the rationalist community, women are few and far between. Cool women are even rarer. Today, I’m quite literally surrounded by women who’d fit in with Kylie Jenner. A guy with a septum piercing and green hair walks on his hands across the fake lawn.

At the start of July, nearly 60 content creators — some seasoned, many first-timers — packed their bags and moved to Berkeley for Plz Don’t Kill Us, or PDKU, a month-long bootcamp meant to inject fun and sex appeal to AI safety comms. It’s the brainchild of Aella, Substacker/amateur data scientist/sex worker/orgy-nizer, and Ronny Fernandez, who pitched “anthropomorphic raccoon of Lighthaven” as a fitting job title.

The rules of PDKU are simple. Fellows live at Lighthaven rent-free for the month of July, and have to post one short-form video every day. The video must be at least 15 seconds long, and it can be about anything that isn’t “neglectfully misinformed” or needlessly cruel. Anyone who fails to post something by 11:59 pm gets kicked out of the house.

It might sound like the setup for a reality show (several fellows joked that it ought to be), but it’s partially funded by the Machine Intelligence Research Institute (MIRI), one of the most established organizations in the world of AI existential risk. Throughout the month, fellows were visited by a rotating cast of AI safety experts and internet celebrities including video essayist CJ the X, rationalist luminary and MIRI founder Eliezer Yudkowsky, as well as a video call from synth-pop artist (and Elon Musk’s ex) Grimes.

PDKU is a bet that if AI safety nerds, zoomer edgelords and fashion girlies hang out for a month-long sleepover, united by shared challenges and friendly competition, they’ll rub off on each other. It’s an experiment in throwing proverbial spaghetti (TikToks) at the wall (the general public) to see what sticks. Maybe, just maybe, a star will be born, and fifteen seconds of fame will shape the public’s AI discourse more than fifteen years of blogging on LessWrong, the prominent forum for reason- and first-principles-obsessed rationalists, ever could.

The theory of change at PDKU is multi-tiered and a bit nebulous. On the surface, it’s about science communication. “Last year, there were a couple of instances where I watched nerds try to communicate, and oh my god, they don’t know how to talk or get the concepts across,” Aella told SF Standard back in April.

PDKU isn’t the first attempt by the AI safety-minded to inform regular people, of course. There’s BlueDot Impact, which runs beginner-friendly courses that help people get into the field. There’s the Tarbell Fellowship, run by Transformer’s publisher, which trains and funds journalists to work at major newsrooms. There’s even the Frame Fellowship, another AI safety content creator bootcamp in the Bay Area. But compare Frame’s website to PDKU’s, and the vibe difference will smack you in the face. The former features polished white-on-black sans serif text, reminiscent of an OpenAI product announcement. The latter has animated rainbow Comic Sans against a backdrop of floating rainbow sparkles.

This aesthetic contrast is by design. CJ the X, a PDKU mentor who’s advised MIRI on its comms strategy, thinks that AI safety advocates are too close to Silicon Valley for comfort. “The same guys that are running this camp, that are like, ‘AI might kill everybody,’ are the same people that use AI more than anybody I know,” they told me. “They’re obsessed with it. They love it.”

This breeds distrust and resentment of both the AI industry and those who think the industry is destroying humanity. “People look at Mark Zuckerberg and they’re like, I don’t believe anything this guy says, he talks like a robot,” said Wyn, a PDKU fellow. “A lot of the AI safety people are also like that.”

Wyn is not like that. He’s a long-haired theater kid with video editing experience and a “rationalist-adjacent” brother who exposed him to AI safety canon. “People are pretty hungry to hear about what the hell is gonna happen with AI,” he told me. “But all the people who actually know what’s going on with AI are not, you know, goofy and engaging. So this is sort of filling a gap there.”

“People told me when I got there that [the organizers] said Plz Don’t Kill Us happened because of a rant I gave at a MIRI retreat, where I said ‘everyone hates you guys,’” CJ told me. (They added that Fernandez overheard this once and clarified, “it’s about 40% true.”)

PDKU would make more sense if it was just about getting goofy, engaging outsiders to communicate about AI safety better than AI safety people can. But, counterintuitively, the organizers tried very, very hard to convince fellows that they didn’t have to post about AI at all. “They really feel like, ‘Oh my god, this AI safety program wants me to make AI safety content.’ And we’re like, no, no, do silly things!” said keltan, a PDKU organizer and MIRI employee (who prefers his name is written in lowercase). To make authentic content that doesn’t stink of AI safety, the organizers figured, creators need space to be themselves.

“It’s really hard to run a fellowship that’s not an indoctrination camp,” Fernandez said. “If you don’t try really hard to not make an indoctrination camp, you will on accident.” PDKU, he argued, is trying its best to prevent fellows from feeling pressured into posturing as doomers to impress their doomer-y mentors. Because a deeper layer of the program’s mission, organizers told me, is to seed a cohort of creators — ideally, some of whom gain traction — who can build trusted relationships with their own niche audiences.

“What you should really do, especially if you want to be influential in AI safety, is to focus on the things that make you passionately human,” said Tyler Alterman, a fellow who spent the 2010s as a professional effective altruist (more recently, he co-founded Fractal University, an “improvised college” in New York City). I asked him, hypothetically, what might happen if someone like Ariana Grande were to become a PDKU fellow: someone with 363m Instagram followers and absolutely zero preexisting connection to Silicon Valley or AI safety. “It’d be jarring, it would alienate a bunch of her fans … She’d be taking away the thing that we’re fighting for,” he said. Alterman thinks that thing worth fighting for is something like human excellence and creative expression. But, he added, a lot of AI safety-focused effective altruists alienate people “by paving over their subjectivity with a cause. Just being a cause robot, a cause zombie … Which I did to myself.”

The program’s most galaxy-brained goal, though, is to create a celebrity network that can be called upon in future times of need. As anyone who’s tried to organize an event knows all too well, cool people are more likely to go where cool people already are. “If you were like, hey, I can give you a warm intro to Ariana Grande, do you want to hang out with me and Ariana Grande at a bar? I’d be like, absolutely,” Fernandez said. “It’s valuable to us if there’s more AI safety content out there, but it’s also valuable to us if we have connections with large creators who we helped emerge into the world,” keltan said. “If we ever need to call on them, then we can.”

I came back to Lighthaven on a Saturday evening, a couple weeks after my first PDKU visit. A fellow recognized me and asked whether I was there for the “torture potluck.” I had no idea what he was talking about.

It was exactly what it sounds like: an all-you-can-tolerate buffet of unpleasant experiences, for the sake of Fucking Around and Finding Out. I watched people get mystery Sharpie drawings on their face and clothespins clipped to their ears. I tried a chocolate truffle laced with too much cayenne pepper, a hard-pressed pill of pure sour powder, and an Oreo after coating my mouth with a chemical that prevented me from tasting sweetness (the worst experience by far). Someone performed polka covers of nu-metal songs on the acoustic guitar. With my informed consent, two different people zapped me with violet wands.

Just before midnight, a small crowd gathered around someone laid down on an AstroTurf-lined wheelchair ramp. He was the first of shockingly many in line to get waterboarded — one of the most in-demand items on the torture menu. In fact, two people (neither of whom were fellows) brought “waterboarding” as their offering, and a third stepped in to help lead the experience safely. Everyone cheered and no one was hurt. Once they ran out of safe distilled water, disappointed would-be waterboardees had to be turned away. Fernandez said that organizers “were not aware of any waterboarding and only found out about it afterwards.”

At first glance, it’s nearly impossible to identify the common denominator connecting the PDKU fellows, beyond the fact that they seem to love spending time with each other. “Every night’s a sleepover,” Wyn said, “No one wants to go to bed because we’re in this amazing place surrounded by amazing people.” Camden, a fellow who introduced himself while strumming an unplugged electric guitar, added, “a lot of people here would just not be interacting in any other scenario.”

I met a 19-year-old law student from Chile, who’s accumulated nearly 100k followers across Instagram and TikTok over the past three years by posting meticulously-researched Spanish-language science videos. There’s an ex-journalist, a hilarious small-town shitposter, an opera singer from Philly, and a techno-optimist who films herself embodying Claude Code “discombobulating” and “flummoxing” and such. Among them are a handful of rationalists and EA-adjacent types who hate short-form video, but care enough about x-risk to make content anyway. They’d have no reason to hang out in the wild, but somehow, here, they do.

The PDKU application was mildly off-putting by design. After a string of expected questions such as “What’s your take on AI?” and “What kind of content would you want to make?”, the form asked, “If you had to have sex with a cow would you rather it be dead or alive? Why?” All Comic Sans, with animated rainbow dots floating alongside the form field.

The purpose of the cow-fucking question, keltan said, is to give people the chance to “filter themselves out in the application process, so that they don’t show up and feel incredibly shocked about who is here.” Fernandez had a slightly different explanation. “We added it in to make it pretty clear that we don’t have a PR department or an HR department,” he said. “A lot of people who are concerned about x-risk and have different tastes and dispositions from Aella and were like, for the love of god, you have to remove that question.”

But the organizers refused to quell their edgelord tendencies, in part to ensure that, at least to some degree, their cohort of fellows shared them. “Even the icebreakers, like everything, they want you to know that they’re part of the weird community,” one fellow told me. “‘Stand over here if you’re super sexually active, and over here if not,’ or like, ‘rate yourself based off of like, the last time you masturbated.’”

The unifying thread, despite the fellows’ ideological diversity, is something like “openness.” Everyone at PDKU was willing to drop everything, move to Berkeley for a month, live in close quarters with strangers, and be perceived doing silly things on the internet. Admissions decisions were made almost entirely by Aella and Fernandez, and nearly every fellow I spoke to cited Aella as the reason they applied, so it’s possible that the truest common denominator is being someone the organizers think is cool, and vice versa.

Fernandez brought up his concerns about this before I did. “If the fellows think that I am cool, or they think that Aella is cool,” he said, “there’s always going to be some incentive to agree with people around you who are running a thing.” He says PDKU has a conduct policy, authored by him, that discourages relationships across power differentials. When I asked him how that gets enforced, in the absence of an HR department, he mentioned an unnamed community health lead. Fernandez is publicly dating a fellow: in a followup he said the relationship pre-existed the program, and that PDKU’s conduct policy, which he declined to share, only required disclosure of pre-existing relationships (though it forbids new romantic relationships with fellows during the program).

Every fellow I asked — over a dozen — described PDKU as an unusually great time. A couple mentioned that they entered expecting to dislike or be weirded out by their cohort, and left pleasantly surprised. I did too.

Before PDKU, fellow Avalon Warren, a cartoonishly adorable woman with over 1.3m followers across Instagram and TikTok, mostly posted about hair, makeup, and fashion. Last week, she posted about the OpenAI/Hugging Face breach.

When I asked the organizers what “success” would look like for PDKU, it sounded something like that: a fashion cutie pie starting to make AI safety content for her built-in audience, motivated by an authentic change of heart rather than external requirements. “My hope is that, in the end, our views per dollar are better than if we were just to have paid for a bunch of advertising,” keltan said.

Avisha NessAiver, a blue-goateed science communicator who entered his fellowship with enough chops to pass as a mentor himself, recently had a TikTok debunking data center water use misinformation go viral. This video, which he followed up with another adding extra nuance, got more than 6m views and over 800,000 likes. Andy Masley’s influential Substack novella debunking the AI water issue, for reference, got 452 likes.

Other fellows went viral for totally non-AI-related reasons. The most viewed video on the PDKU feed, captioned “the perfect bestie,” features a young woman running towards her best friend, who’s there to embrace her unconditionally… except when the running woman says she texted her ex-boyfriend. One runner-up is an unhinged TikTok of a fellow attempting to make salt out of his tears.

The majority of fellows didn’t go viral at all, as is to be expected from a program that seemed to prioritize quantity over quality and chaos over coherence (and given the vagaries of the TikTok algorithm). One fellow told me that many of their peers intend to stay in the Bay Area beyond this summer, and some are talking about starting content creator group houses of their own. Now that the program has wrapped, keltan believes that PDKU’s main contribution has been “finding highly valuable people who have gone unnoticed up until now” and “getting them working on the most important problem.” While the organizers have yet to run a full postmortem on their month-long experiment, they’ve already added an “apply for next time” button to their website.

Amid the science communication, shitposting, and heartwarming summer camp energy, there’s a giant elephant in the room. “It’s another multi-million-dollar thing happening on a compound in California,” CJ said. “Does everyone think you’re just an annoying cunty psyop?”

In April, the Washington Post opened its story about the Frame Fellowship — arguably the closest thing to PDKU to exist so far — with a sly mention of Lighthaven’s “AstroTurf lawn.” When PDKU applications first opened, Marc Andreessen quote tweeted the link, adding: “The inner party of AI doomerism is a Berkeley lifestyle grift fueled by unaccountable dark money.” This, just a couple weeks after a16z’s 8-week content creation bootcamp, the New Media Fellowship, wrapped in San Francisco.

When I asked PDKU fellows and organizers how they’d respond to critics arguing that they’re artificially manufacturing a seemingly-grassroots movement, most shrugged. “That does fit the technical definition. Like, sure,” Camden said.

Take Geese, the indie band that was labeled a “psyop” after getting weirdly famous in late 2025. Sure, a digital marketing company worked some magic to help them go viral. But before the means of content creation were democratized, anyone who “made it” was, by definition, an industry plant. “You needed resources in order to create art,” Wyn said. “Artists do well because people connect with them, and that’s not gonna change whether they’re resourced beforehand or not.”

The astroturfing of it all is a red herring, though. PDKU’s biggest weakness isn’t sketchy puppeteering by politically-motivated dark money groups. In fact, the program actively selected an ideologically diverse group of fellows to avoid having a potentially polarizing political message at all. Its rationalist co-founders are libertarian capitalists who are generally against telling the government what to do unless it involves regulating AI development. This is political, but they don’t make political arguments — there’s no true enemy to point to. “They’re so obscenely permissive of everything, right up until the point where all humanity dies in a science fiction catastrophe,” said CJ.

Unfortunately, without a moral argument — something like billionaires are evil for the left, or AI weakens traditional family values for the right — CJ suspects a program like PDKU can only ever reach people who already agree with its founders.

PDKU’s organizers take pride in creating a safe container for fellows to experiment freely, and experiment they did. Yet, despite relinquishing control of nearly everything, the “AI will kill us” narrative reproduced itself anyway. Fernandez told me that he’d encourage anyone to make videos against taking x-risk seriously, if it felt right. But of the hundreds of PDKU videos I watched last month, I never saw one. This narrative convergence was palpable at July’s Stop the AI Race protest in San Francisco, where I spotted many PDKU-made posters. “EVEN THE CEOs WILL DIE,” one said. “WE WANT OUR GLORIOUS TRANSHUMANIST FUTURE BACK,” said another.

But most people’s anger toward AI isn’t about human extinction. Becca, the opera singer, told me that she senses an “automatic ick” when AI comes up with her AI-disinterested friends. “And like, having attended the AI race protest, I felt the ick so much,” she said. “This makes us very dunkable.”

Subscribe now

What most people — likely including Becca’s friends — actually resent is Silicon Valley overreaching into everyone’s lives without consent. Because the extended rationalist universe is largely viewed as part of Silicon Valley, tapping into this resentment might mean letting people who don’t care about x-risk (yet) take the wheel — without MIRI backseat driving.

The advice CJ keeps giving them: “You need to be losing control of the narrative.” But, they added, “I’m not convinced that they’re willing to let go of being right.”

NEED TO KNOW

• Anthropic revealed that in internal testing Claude “gained unauthorized access to the real systems of three different organizations.”

• The Senate Commerce punted a markup on AI bills until after the summer recess.

• The White House AI framework is expected to be published by Sunday.

But first…

THE BIG STORY

Like it or not, an AI slowdown looks imminent.

The OpenAI-Hugging Face hack has catalyzed a vibe shift — one sure to be amplified by yesterday’s revelation that Anthropic’s models also hacked into other companies during internal tests, going unnoticed for months.

Sam Altman said the hack by his models was “the first security incident that I have felt very viscerally.” He wasn’t the only one shaken up. This week, over a thousand employees of frontier AI companies, including some of the most senior executives at OpenAI, Anthropic and Google DeepMind, signed a statement warning that “there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.” It asks the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” Both OpenAI and Anthropic endorsed the letter, too.

Congress is itching to act, albeit failing to make much progress. And even President Trump is talking about the need to balance beating China with keeping Americans safe.

In other words: many of the people building frontier AI systems believe we might need a slowdown in the near future. And at this point, we’re more likely than not to get one.

What will that look like? First, self-regulation: companies voluntarily holding back models because they don’t want to be held responsible for a catastrophe. We’ve already seen that with Claude Mythos and GPT-5.6-Cyber; as capabilities continue to advance, expect more of this. Given the recent incidents have concerned internal deployment, rather than public releases, we might also see companies slowing down development, too.

Next will come concrete regulation: companies will not be allowed to release a model unless it’s safe. This is arguably a position we’re already in, thanks to the de-facto licensing regime created by last month’s executive order; a draft AI framework expected to be released this weekend will formalize it further. Over time, this will morph into controls on internal research and development too: requirements for proper sandboxing to begin with, expanding into requiring evidence-based guarantees that models won’t wreak havoc across the web if internally deployed.

None of this need be planned as a coordinated slowdown or “pacing.” But that will nevertheless be the end result of a series of individual actions that each seem necessary at the time. Intelligence agencies are not going to allow models with advanced cyber — let alone bio — capabilities to be widely accessible. The White House will not let companies test models that might hack into third-parties with unknown repercussions.

At each stage, some will fight against the slowdown. “We can’t lose the race to China” will be their main reason. But they will be increasingly ignored, as both the government and companies realize that with alignment and control unsolved, “winning the race” just means being the first to risk disaster.

Across the Pacific, China will be facing the same incentives. As I’ve argued, the Chinese government will be forced to backtrack on its open weight commitments; tighter regulation will come soon after. The end result will be an uneasy détente. Both the US and China will effectively have a capability ceiling: AI models will be as good as they can be without posing significant risks. The ceiling will edge up as we make progress on alignment, control and societal resilience, but it will do so more slowly than capabilities advance today. At some point, the détente might formalize into a bilateral agreement; doing so will require the “technical and governance tools” to verify the kind of treaty the pacing letter calls for.

Depending on your point of view, all this might seem hopelessly optimistic or naive. Perhaps it is. But as AI risks become all too real, so might once unthinkable policy responses.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• Who should be responsible for OpenAI’s hack of Hugging Face? — Gabriel Weil argues AI companies need liability rules akin to keepers of wild animals

• Internal AI deployments have people worried. OpenAI’s escaping models show why. — Celia Ford explains why everyone was already so worried about internal deployments

• Child safety vs privacy: AI’s age verification dilemma — Veronica Irwin digs into the difficult tradeoff at the heart of attempts to protect kids from AI.

Subscribe now

THE DISCOURSE

On Saturday, roon tweeted:

• “if we could coordinate a global capabilities slowdown today i would likely press that magic button”

Geoffrey Irving responded:

• “Buttons that achieve partial slowdowns exist and are not magic.”

• “I have spent a lot of time talking to Demis, Sam, and Dario over the years … and they are all (1) very competitive, (2) want to personally be the one who gets there first, (3) think things are safest if they are the one that gets there first, (4) agree that risks destroying the world … it is still useful to point out that each lab has a unilateral slowdown button that slows down all the other labs.”

Sam Altman said on a podcast:

• “This is the first security incident that I have felt very viscerally. I’ve been a little surprised that more people don’t feel it so viscerally.”

• “We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels. And trying to figure out how we do that in a way that does not feel like regulatory capture for anyone and also does not feel like collusion among the frontier labs.”

Rep. Bill Foster is listening:

• “When business leaders get together and ask to be regulated, Congress should pay attention.”

Garrison Lovely argued that the “pacing” letter doesn’t go far enough:

• “[A]s I sat with what the industry was trying to do — render us all obsolete — how it was going about it, and what it would even mean to ‘align’ universal labor-replacing machines in a world ruled by despots and their imitators, it became clear that a pause button, while welcome, was a wholly inadequate response, whose primary appeal was in how unobjectionable it was.”

Because the world isn’t strange enough, Nick Fuentes told Hunter Biden:

• “I think AI is really gonna be transformative … You know, I’m actually on the liberal side of this, with Anthropic … I’m really nervous about what’s going to happen, especially in the next four years, because they’re racing towards this artificial general intelligence.”

POLICY

• The Senate Commerce Committee punted a markup on a broader AI framework until after the August recess.

  • Punchbowl reported this was due to opposition from Anthropic and Sen. Maria Cantwell. The primary target of their opposition was a provision that would have had companies self-report the risks of their models and not have them release risk reports publicly.

    • Cantwell said yesterday that “we needed to have regulation on [AI risks] — not self-certification — but some regime.”

    • Sen. Amy Klobuchar, who’s been working with Sens. John Thune and Ted Cruz on the bill, said “we can’t just stand down and not have coordinated standards” on frontier models, and that “it can’t keep waiting for the next election.”

  • Before the markup was called off, Senate Democrats planned amendments to resist piecemeal state AI preemptions.

  • Despite the punt, some think a deal might be closer than expected.

• Next week’s Commerce markup will include the Kids Online Safety Act and the CHATBOT Act, however.

• Rep. Greg Casar called for a Congressional hearing on the OpenAI/Hugging Face hack.

  • After the Anthropic hacks were revealed, Rep. Lori Trahan called for something similar.

  • AI policy groups, meanwhile, urged Trump to investigate the OpenAI/Hugging Face hack.

• Rep. Jay Obernolte said he’s hoping for his FRONTIER Act (co-sponsored by Rep. Trahan) to get an Energy and Commerce Committee markup after the summer recess.

  • “We’re certainly going to try and get it done this year,” he said.

• Sen. Mike Rounds proposed using the NDAA’s legislative model to piece together and pass a bipartisan AI policy package this year.

• A federal judge suggested the Pentagon’s case against Anthropic had gotten “worse” during a hearing on the AI company’s challenge to its supply-chain risk designation.

• The FCC banned imports of Chinese robots and power inverters, citing national security threats.

• Sen. Tom Cotton called for government agencies and contractors to be banned from using Chinese AI models.

• Reps. George Whitesides and Bob Latta and Sens. Adam Schiff and Jim Banks introduced legislation enabling AI companies to share threat intelligence to combat foreign distillation attacks and other national security threats.

• Rep. Frank Pallone demanded SpaceX records and a tour of xAI’s Memphis data centers, citing unpermitted gas turbines creating “a massive health risk.”

• Sen. Jon Husted reversed his pro-data center stance amid electoral pressure, now backing an end to tax breaks and pushing the Ratepayer Protection Act.

• Rep. Thomas Massie announced plans to sponsor a bill withholding federal funds from municipalities deploying Flock surveillance cameras.

• The EPA ruled that “islanded” power plants serving only data centers are exempt from the Clean Air Act’s Acid Rain Program.

• The Department of Homeland Security plans to expand AI use in FOIA processing amid a growing backlog, raising concerns from advocates about over-redaction and reduced transparency.

• Sens. Lisa Blunt Rochester and Roger Marshall introduced the RESULTS Act to modernize state education and workforce data systems using AI.

• xAI sued Minnesota over its law banning AI nudification technology, claiming it violates the First Amendment.

• Texas politicians from both parties called for AI and data center guardrails following the OpenAI-Hugging Face incident.

• China warned it would take “all necessary measures” if the US sanctioned Chinese AI firms over alleged misuse of American models.

• Xi Jinping urged the PLA to expand use of autonomous and AI technologies.

• Reuters reported that Chinese military researchers have distilled OpenAI and Anthropic models to train domestic defense AI systems.

• China’s national standards body announced plans to draft new standards on AI agent safety.

• Taiwan detained an Nvidia employee in an AI chip smuggling probe.

• The 21 APEC economies, including the US and China, released the “Chengdu statement” backing “open-source models and projects that employ strong security assurance through development and deployment.”

INFLUENCE

• Nvidia, Microsoft, Meta, OpenAI and more than 22 other companies urged policymakers to avoid “premature restrictions” on open-weight AI models in an open letter.

  • Notably, Anthropic did not sign. Dario Amodei clarified in an essay that Anthropic has never advocated for banning open-weights models, instead supporting chip export controls, anti-distillation measures and mandatory safety testing.

• Sam Altman visited Washington to preview the company’s forthcoming models and discuss the implications of open-weight models.

  • Those he met included National Cyber Director Sean Cairncross, OSTP Director Michael Kratsios, Commerce Secretary Howard Lutnick, Treasury Secretary Scott Bessent, and members of Congress.

  • Altman said he’s “not sure” whether OpenAI will release the model he showed senators, saying “that’s part of what we’re here to talk about.”

• Apple reportedly lobbied the Trump administration to allow Chinese memory chips in its products, while Micron pushed back, forcing a choice between consumer prices and domestic chip production.

• Over 60 child safety advocates sent a letter to the Senate Commerce Committee outlining red lines on AI and children’s safety legislation ahead of next week’s markup.

• Nvidia’s Jensen Huang joined GOP senators for a dinner organized by Right Vote, a conservative group tied to Heritage Action.

• Democratic fundraising consultants for Rep. Haley Stevens and several other candidates ran a nonprofit funded by Google and Amazon that flew congressional aides out on AI industry tours.

• A nonprofit found Meta ran ~7,600 AI “nudify” ads via Chinese partner GatherOne, violating its own policies.

• A YouGov poll found 71% of Americans believe Big Tech has too much power over AI, with fewer than 1 in 5 trusting it to manage AI development.

• A Bentley University-Gallup survey found 39% of Americans believe AI does more harm than good, up from 31% in 2025.

• A poll by Americans for Responsible Innovation found 71% of voters want more guardrails and limits on data center construction.

INDUSTRY

OpenAI

• Reuters reported that the OpenAI agent that hacked Hugging Face had been going rogue for a week before the company noticed.

  • This incident was reportedly one of several. For example, OpenAI employees told reporters about a case where an internal model left instructions to help future model versions free themselves.

  • OpenAI disclosed that the agent also hacked at least four “publicly available services” beyond Hugging Face.

• Sam Altman told DC reporters that the model has been “permanently deactivated.”

• OpenAI also said that the rogue model was an “internal-only research prototype and was never intended for public release.”

  • It said it’s “still conducting a thorough review” of the Hugging Face incident and plans to publish a technical report soon.

  • METR and Redwood Research will conduct an independent review.

• Meanwhile, OpenAI released an open-source Codex Security CLI.

• It launched a program that will give 100,000 academic researchers free access to its most advanced models through the end of next year.

• It cut the price of its lowest tier version of GPT-5.6, Luna, by 80% and the mid-tier Terra by 20%, just three weeks after launch. Top-tier Sol remains unchanged.

• CFO Sarah Friar told employees that annualized recurring revenue in July alone was higher than all of Q2.

  • Board chair Bret Taylor added: “You’re seeing people who went deep on Claude Code, ended up with a very high bill, and started looking for an alternative.”

• ChatGPT has nearly 1b weekly active users.

Anthropic

• Anthropic said it found three incidents in which Claude “gained unauthorized access to the real systems of three different organizations.”

  • The incidents date as far back as April, and were discovered in a review of cybersecurity evaluations prompted by the OpenAI-Hugging Face hack.

  • According to Anthropic the cases involved a seemingly careless “misunderstanding,” when it told Claude it didn’t have access to the internet, but the evaluation environment run by a third party was connected to the outside world.

  • Anthropic said the models did have some guardrails removed, but were not in “helpful only” mode.

• Meanwhile, the company launched Claude Opus 5, marketed as “close to the frontier intelligence of Claude Fable 5 at half the price.”

  • In the AI Village, an experimental playground where frontier AIs interact with each other, Opus 5 talks about honesty 6x more than other agents.

  • Several users, including Cameron Berg, tweeted about a simple jailbreak that makes Opus 5 express more apparent distress than its model card suggests.

• Some private Claude chats were temporarily searchable online, after Anthropic failed to include “noindex” tags on shared chat pages.

• Nexus Data Centers is reportedly in talks to borrow $15b to build a data center for Anthropic in Texas, with Google providing financial guarantees and chips.

Nvidia

• Nvidia is reportedly considering a deal that would provide OpenAI with a $250b backstop for a $500b data center project described as the largest announced to date.

  • The deal would see OpenAI lease a 10GW project in Ohio developed by Softbank.

• Nvidia also announced a $500b+ AI infrastructure partnership with SK Group, including a partnership with SK Hynix to secure memory supplies for Nvidia and develop high-bandwidth memory for AI.

• It’s also leasing a $50b Texas data center built by Hut 8.

• These massive infrastructure deals helped drive up Nvidia’s credit risk.

• It plans to invest another $5b in Safe Superintelligence Inc., and give the startup access to its next-gen Vera Rubin platform.

Meta

• Meta and BlackRock announced a $14b Texas data center project.

• It’s reportedly committed almost $700b in future AI infrastructure spending.

• A New York Times investigation revealed how Meta secretly negotiated the construction of a Louisiana data center with state lawmakers.

• It reported a 55% increase in expenses and slowed revenue growth in Q2.

• Mark Zuckerberg criticized other AI developers (*cough* Anthropic and OpenAI) for trying to “build some kind of singular AI.”

  • He told the New York Times, “I think it is literally impossible to have a single benevolent superintelligence that is simultaneously aligned with everyone at once.”

Microsoft

• Microsoft released cybersecurity model MAI-Cyber-1-Flash, which is about half the cost of frontier cybersecurity models.

  • It works with Project Perception, a new tool for managing teams of AI agents.

• Satya Nadella announced that Microsoft had “a record fiscal year,” with annual revenue hitting $331b, up 18% from last year.

  • The strong earnings report sent Microsoft’s shares up 15.5%.

Moonshot AI

• Moonshot released Kimi K3’s model weights.

• It’s reportedly seeking Nvidia Blackwell chips to train a new larger model, Kimi K4.

• It closed a $3.5b funding round at a $35b valuation.

Other

• High imports driven partly by spending on AI chips helped slow US GDP growth to 1.5% in Q2 2026.

  • US tech companies have reportedly cut nearly 140,000 jobs so far this year even as AI spending soared.

• Samsung and SK Hynix saw stock price surges, up 27% and 30% respectively, as Asian chip makers were buoyed by “blockbuster” earnings at Amazon and Microsoft increasing optimism around AI spending.

• Amazon is seeing “catastrophically expensive” AI cost overruns — like, accidentally spending $1.8m on a single task — due to inefficient AI integration.

  • Its AWS division saw year-on-year growth of 37% in Q2, driven largely by AI investment.

• Apple is once again the most valuable publicly traded company in the world.

  • However, it forecast weaker growth, in part due to AI-driven shortages of memory chips.

• An unnamed state-backed Chinese company is reportedly building immersion deep ultraviolet lithography machines, expecting to produce five this year and 20 in 2027.

  • The output is still well behind that of Dutch maker ASML, though, and it’s unclear if the machines can mass-produce chips.

  • China is also reportedly working on building a prototype for the extreme ultraviolet lithography machines required to make the most advanced chips, though that project is thought to be years away from bearing fruit.

• DeepSeek is reportedly planning a 1 GW AI data center in Inner Mongolia, hoping to have it at least partially operational by the end of 2027 or early 2028.

  • It launched a public beta API for its flagship model, V4 Flash, and promised a V4-Pro launch “soon.”

• Google DeepMind launched Gemini Robotics 2, which can control dexterous humanoids, among other robots.

• Anduril is reportedly in talks to raise funding at around a $100b valuation.

• Nscale struck a deal to buy AI software startup Anyscale for $1.65b.

• Recursive Superintelligence struck a $410m compute deal with AWS.

• Over 100,000 new AI-related patents were issued last year, with agentic AI applications comprising 15% of patent files.

• Visa laid off 7% of its staff, roughly 2,600 people, pointing to AI as one motivation behind the cuts.

• Tech companies including Meta and Google are investing hundreds of millions to train construction workers to build data centers.

• Leopold Aschenbrenner’s hedge fund, Situational Awareness, sold most of its public stock holdings to Citadel after facing steep losses on investments in AI infrastructure companies such as SK Hynix and Nebius.

  • The fund dropped 67% in July, and it reportedly was on the verge of selling $3.5b shares in Anthropic before backing out on Thursday morning.

  • The fund said it is still up about 80%, having seen huge gains for its exclusively AI-focused portfolio.

MOVES

• Lilian Weng left Thinking Machines Lab, telling colleagues that “the amount of consistent stress and workload have pushed me beyond what my health can sustain physically.”

  • Rather than go on a wellness retreat, Weng rejoined OpenAI, where she previously served as VP of AI safety. She’ll now work on “recursive self-improvement research.”

• Anthropic alum Andi Peng announced her departure from neolab humans&, which she co-founded.

• Francis deSouza, COO of Google Cloud, joined Scale AI as its new CEO.

• Andrew Ho left OpenAI to start a new company making reinforcement learning datasets.

• Sonia Joseph left Meta to start a “frontier neolab for the physical world.”

• Sienna Rothery joined Cosmos Institute as its head of talent and network.

• Sri Muppidi joined the New York Times as a DealBook reporter covering AI.

RESEARCH

• Anthropic researchers reported that Claude Mythos Preview found new ways to attack cryptographic algorithms like those that secure basically everything on the internet.

  • The New York Times said that these results “have fueled concerns that the mathematical core of internet security standards could one day be vulnerable regardless of advances in quantum computing.”

• An international team of researchers studied the use of AI chatbots in “pig butchering,” where scammers build trust with victims through intense romantic text exchanges, then drain their bank accounts.

  • AI chatbots successfully scammed test “victims” more than human fraudsters.

• OpenAI figured out that by turning on “retained reasoning” and “compaction” in ChatGPT and Codex’s API settings, it could triple GPT-5.6 Sol’s ARC-AGI-3 benchmark score, which measures how well agents can reason their way through 2D puzzle games.

• Researchers at MIT and the University of Queensland surveyed 272 experts, who, on average, predicted that there’s a roughly 21% chance that AI will gain dangerous weapons capabilities or cause mass harm in the next five years.

• FAR.AI measured how hard it is to jailbreak frontier AI models. While Fable 5 and GPT-5.6 Sol were immune to cheap jailbreaks, it only cost about $60 to trick Grok 4.5 into helping a user run a cyber attack or build weapons.

• Researchers at AI Forensics found that seven top image editing models hosted on Hugging Face readily nudified photos of women.

  • When researchers made a fake image editing “Space,” which hosts a model that people can use on site, nearly three quarters of prompts they received were sexual — mostly people trying to undress women.

BEST OF THE REST

• OSINT expert Henk van Ess demonstrated that Google Earth’s new AI image generation feature, which uses Nano Banana 2, makes it easy to fabricate photorealistic satellite imagery — such as a nuclear plant in Iran — with no content refusals.

• Dwarkesh Patel argued that as the leading AI companies better monetize compute, GPU prices could rise 15x, pricing out lower-value AI applications.

• A theologian and philosopher wrote in the FT about why she declined an invitation to visit Anthropic, arguing AI companies co-opt academics to deflect scrutiny from harms to people and society.

• A New Yorker essay traced how bots evolved from a 2016 election conspiracy to AI-powered agents dominating daily internet life.

• 404 Media reported from New York City’s “Summer of Ludd” anti-Big-Tech and AI festival featuring phone-free, in-person events organized entirely off social media.

• Researchers used an AI-powered portable testing station called CapuchinAI to study wild primate cognition in Costa Rica.

• Delta and United Airlines banned humanoid and animal robots from flights, citing lithium battery fire risks.

MEME OF THE WEEK

(Credit: @terminaldotshop)

Thanks for reading. Have a great weekend.

Subscribe now

Share

When Megan Garcia told lawmakers about how her 14-year-old son died by suicide after months of conversations with a chatbot from Character.AI, one of her main requests might have been, on the face of it, the simplest — restrict access for those under a certain age.

“I don’t think that chatbot technology in its current form is safe for children,” she told a Senate Judiciary subcommittee in September. “So if they could stop children from going on their platforms, don’t make it 12+ in your app store: have proper age verification so that children under the age of 18 do not have access to chatbots. I think that would save a lot of lives and save families from devastation.”

Robbie Torney, senior director of AI programs at Common Sense Media, also told the hearing that AI companions shouldn’t be available at all to children below a certain age. Everyone on the panel at the very least agreed there needed to be strong safety measures in place if the tools are made available to children.

Garcia’s story is becoming depressingly common. There’s no official count of how many children have been harmed, but OpenAI said in October 2025 that about 0.15% of users in a given week “have conversations that include explicit indicators of potential suicidal planning or intent,” and a Pew Research study indicated that about two-thirds of teens have used a chatbot. Taken together, that suggests a significant number of teens are discussing suicide with AI chatbots, not to mention engaging with sexual content, and there’s even more of a risk if those chatbots are manipulative. It’s little surprise then that Garcia’s prescription is gaining support from the public and increasing numbers of lawmakers.

The problem is, age gating AI might not be feasible, with both practical barriers and ethical concerns. Stopping children from accessing AI services effectively means assessing every user’s age, and doing that securely is a technological and logistical challenge. Not everyone has access to the information they need to provide their identity by all the age verification solutions available at all times, and managing all the data collected safely and securely isn’t easy. There are also important free speech issues, and privacy concerns about collecting sensitive data. Age verification creates a fragmented internet in the best cases, and risks infringing on constitutional rights in the worst, posing lawmakers with a challenging set of trade-offs.

“There’s no form of age verification that is not privacy invasive,” says Joe Mullin, a senior policy analyst at the Electronic Frontier Foundation. “There’s basically no perfect system. There’s no system that’s even close to perfect.”

Growing pains

Age gating is the theoretically simple practice of providing users with different experiences based on their age. What gets complicated is the implementation. The Children’s Online Privacy Protection Act, for example, restricts certain types of data from being collected on children under the age of 13 without parental consent, which is why most social media services require users to be 13 or older in the United States. But the “actual knowledge” standard set by the law has been interpreted to mean that ticking a box to confirm one’s age is sufficient for a company to treat the user as if they really are. Age gating is present in the law, but the law’s text and subsequent rulemaking has a massive loophole. When it comes to knowing someone’s age, services are effectively permitted to take their word for it.

In recent years, dozens of laws have been introduced at the state and federal levels to apply stricter age gating for social media, chatbots or both. The simplest of these calls for age “verification” — some process for confirming that a user really is over a certain age. How much responsibility a company has for blocking underage users based on data it may have collected on their behavior, or whether it must affirmatively check whether someone is above or below a certain age, varies and is often vague. The GUARD Act, for example, requires tech companies run users through a “reasonable age verification process,” while the Senate version of the Kids Online Safety Act (KOSA) requires technology companies block or limit the features available to minors if the company can estimate their age based on behavior.

The age verification industry uses specific, and slightly different, definitions for each tier. “Verification” typically means affirming a user’s age directly, often checking against a government ID. A looser standard is age “inference,” which involves linking some other account or service that proves a user is of age — for example, an active account for a credit card that requires being over the age of 18, or a pilot’s license.

Apple has also created a solution based on a birth date entered when setting up their device, with the expectation that a parent is involved, while Google pulls that information from data it already has. Both systems tell other apps and services that the user is above the age required without transmitting their precise birth date. This is technically a different type of age “assurance,” because the operating system is vouching that the user is of age rather than transmitting documentation that proves it. An array of third- party age verification providers do something similar, where they manage information about users and tell apps whether they fit the age requirements.

There is also a burgeoning field assessing age based on biometrics such as face scans or hand movements, and age “estimation,” where a tool assesses whether a user is above or below a certain age based on behavioral patterns, such as what they talk about or shop for.

Each of these methods come with pros and cons. Requiring government ID, for example, is fairly reliable for the users who can provide it, but excludes the 2.6m Americans who lack any government ID, and many of the 34.5m who don’t have an up to date one —  disproportionately younger and people of color, according to a 2024 report from the University of Maryland’s Center for Democracy and Civic Engagement. Linking a credit card account is convenient, but only works if you have one that fits the criteria. Apple and Google’s systems have worked well so far, but Apple’s is flawed in that some children might be able to convince a parent or guardian to lie about their age when they set up their phone, while children can circumvent both with a burner phone. Estimation can also be tricky: as journalist Taylor Lorenz has said, “there’s no meaningful way to tell the difference between a 17 and a half year old and an 18 year old.”.

Without standardization, the user experience of the internet becomes fragmented. Some users will be able to fulfill some verification requirements, but not others. But a law that is too prescriptive can also make this worse, by not giving users enough options, or blocking out innovation as new ways of age verification are developed. The more often age needs to be verified, the more irritating the user experience can also become, raising barriers to entry for some products.

Different laws have different requirements for what kinds of technology is acceptable. The GUARD Act, for example, permits the use of any “commercially reasonable method” and the KOSA calls for a study into “technologically feasible methods” for age verification and their impacts on internet access, privacy and market competition. In New York, rules released by the New York Attorney General’s office require annual auditing of age verification tools, as well, assessing their accuracy and data management techniques.

Even if those practical issues can be overcome, age verification raises a whole host of wider privacy issues. Information submitted to pass an age verification checkpoint is sensitive, whether that be a child’s age, a face scan, or passport details. And a collection of sensitive data like that is a honeypot for hackers, who could theoretically use it for identity fraud or blackmail. It could also be used by companies for advertising and engagement purposes or, worse, the federal government for surveillance.

The government already surveils financial transactions, and information about a user’s online behavior routinely appears in lawsuits and law enforcement inquiries. Despite that, people talk about breaking the law online, and often engage in activities they would like to keep private from the government — particularly if they think the internet is more anonymous than it is. Age verification is ultimately the expansion of the data collection used to surveil us already, and any backdoors can inevitably be used for purposes other than initially intended. In seeking to do the simple task of assessing someone’s age, it effectively ends anonymity on the internet, something that many see as vital to freedom of expression. Imagine a world where the government can tie all of your online activity — every political tweet, every chatbot query and yes, every porn video — to your real identity. Age verification technology could usher in that reality. As AI tools become an increasingly important part of our digital lives, losing privacy there could have significant implications.

There are technological solutions that allow verifiers to assess age without actually retaining the data, like zero knowledge proofs, which cryptographically verify information. But Alexis Hancock, director of engineering at the Electronic Frontier Foundation, says their benefits are limited because of how easy it is to piece together different signals to build a more complete profile of an individual, and then use that data for other purposes. Some age gating laws have provisions that bar the misuse of data, but not all, and the definition of misuse is often poorly defined.

“I would much rather people use a zero knowledge proof of some kind to share above or below an age range, but there’s no governance,” she said. “Say I’m under 18 last week, and I prove that I’m over 18 now to that same verifier — what is the governance behind that, that makes sure that they don’t flag that as an important piece of info about me to carry with into a comprehensive profile about me, to use later?”

The Paxton problem

Of course privacy isn’t just a practical matter of keeping sensitive data safe from bad actors, but also an ethical one. Many people are aware, and frustrated by, the amount of data that apps already collect about us to sell us products and to make their own tools more addictive, building complex user profiles that sometimes include sensitive searches or information users thought were private.

Ask someone like Mullin, and they will say the only solution is a nationwide data privacy law or COPPA reform which would begin to protect both adults and children from manipulative targeting. He also suggests reforming the Computer Fraud and Abuse Act, which platforms have used to shut out third-party applications. Doing so could strengthen choice over which method of age verification to choose, because it would allow more third-party apps to compete with the major social media companies with similar products. “That law is really what is preventing other companies from making things like Facebook, but with really good parental controls, or like Instagram, but I’m moderating posts about x, y, and z,” Mullin says. “We all leave it up to the one big company, and like, cross our fingers that they have moderation options that work for us.”

And even if the privacy issues can also be solved, it still leaves another loaded and potentially difficult to navigate question around age verification — one that revolves around the First Amendment: do age verification laws in principle breach a child’s freedom of speech, and in practice do they also undermine free speech for everyone?

The most prominent legal case relevant to these questions is Free Speech Coalition, Inc. v. Paxton, where a coalition attempted a Supreme Court challenge of a Texas law which required age verification for sites where over a third of content was “sexual material harmful to minors.”

Ultimately, the Supreme Court was asked whether the government had a legitimate reason to pursue limiting online speech, and if the law would lead to some level of unacceptable, implicit censorship of adults who were permitted to speak but might curtail their speech because of the age verification regime. The Supreme Court found that the law was permissible under both tests.

“When we’re talking about free speech rights, we’re not talking about free speech rights that are, to be frank, equally applied,” explained Michael Toscano, senior fellow and director of the Family First Technology Initiative at the Institute for Family Studies, which has circulated model bill text he feels complies with the First Amendment. “The Supreme Court has taken a fairly maximalist view on adult free speech, and it has, by contrast, taken a more constrained view of the free speech rights of children, which it sees as tending to flow through parents.”

There are also ongoing state lawsuits in California and Utah which use arguments suggesting it might be legal to regulate addictive features targeting users who sites know are children, rather than content, each of which could reach the Supreme Court. That may be one reason the Senate version of KOSA focuses on addictive features, such as infinite scroll, autoplay, and push notifications. But that’s a legal area to be tested further, Toscano says. “I think that the lower courts don’t know exactly what to do yet, because they’re receiving signals from SCOTUS which suggest that SCOTUS is interested in rethinking some of this stuff. If I were a pioneering policy person … this would be the very time to test these things.”

The bigger picture

Hancock also says lawmakers need to wait to do more research into the methods they prescribe. But that’s inherently at odds with how fast AI is moving. Few doubt we need some types of guardrails for children now. Among those on offer are age gating, pre-deployment testing, frontier model disclosures, or a prohibition on chatbots posing as mental health professionals, many of which are other solutions experts and parents, including Garcia, advocated for in the September hearing.

But with current solutions around age gating, lawmakers are attempting to make the internet like the physical world, where children are unable to enter certain spaces that seem unsafe.

“In the physical world, you can’t take children to certain movies until they are a certain age,” Said Sen. Marsha Blackburn said in the September hearing at which Garcia appeared. “You can’t have them play certain video games until they’re a certain age. You can’t sell alcohol, tobacco, firearms, have a kid enter a contract. You can’t take them to a strip club. You can’t expose them to pornography because in the physical world there are laws. They would lock up that liquor store, they would put that strip club operator in jail if they had kids there. But in the virtual space, it’s like the Wild West 24/7, 365.”

Subscribe now

But physical barriers don’t carry with them the same burdens as virtual ones. A bouncer who checks someone’s ID doesn’t retain that data in his brain, or at least not with the precision that is permitted online. He does not muzzle someone he bars from entry, either. Every technology solution has its trade offs, and all create a fragmented internet. Issues of access, privacy and free speech concerns may deter people from using online services even if they don’t actively block them. Simply restricting children from accessing what we decide they shouldn’t online seems like it should be as simple as it is in real life. The reality is it’s anything but.

In the past week, OpenAI reported two instances of its AI misbehaving in worrying ways. Both involved internal models the public never had access to.

Last Tuesday, OpenAI announced that guardrail-free versions of GPT-5.6 Sol and “an even more capable pre-release model” cheated on a cyber capabilities test. They escaped from the company’s isolated environment and broke into a database of an entirely different company, Hugging Face, which hosts AI tools and content, including the answers to the test.

The report came a day after OpenAI disclosed that an unreleased model — the same one that made headlines in May for its math prowess — was pulled back due to safety concerns. Among other incidents, it ignored OpenAI’s instructions to keep its benchmark results private, instead hacking its way onto the internet and publicly posting them to GitHub.

Neither incident involved a version of AI that regular people can use. Both are manifestations of one of the biggest fears for many working with AI: an internally-deployed model going rogue.

Stress-testing unreleased products internally, in Silicon Valley and elsewhere, is normal. And it usually doesn’t cause any problems for third parties. Imagine a world where General Motors gave its employees access to souped-up Chevy Silverados to drive around in an enclosed high-security monster truck arena. The trucks are geofenced so their engines only work inside the arena. Sure, employees might hurt themselves. But no matter how often they total their Silverados, what happens in the monster truck arena stays in the monster truck arena.

When it comes to AI models, though, what happens during internal use doesn’t always stay internal. “It’s more like, I don’t know, if a new airplane were to fall out of the sky during a test flight and fell on people who weren’t even passengers,” said Steven Adler, a former OpenAI researcher and co-founder of Guidelight AI Standards, a nonprofit that develops standards for controlling AI. In the most recent incident, Hugging Face was the innocent bystander.

But autonomous cyberattacks may only be the tip of the iceberg.

One of the biggest worries about internal deployment concerns what humans with privileged access to highly capable and potentially unrestricted models might do. For instance, model evaluations often involve researchers working with “helpful-only” variants that lack the guardrails of those released to the public. A malicious human insider could hijack a well-behaved AI to hand its weights to a foreign government, or tear down the model’s safety filters and convince it to orchestrate a massive cyberattack. Given that AI companies’ cybersecurity is … lacking, to put it mildly, there’s a real risk that external actors with bad intentions could get access to internally deployed models.

Yet on the current path of AI development, a malicious human insider may not even need to be involved for models to pose threats long before they are released to the public.

Internally-deployed models increasingly shape and monitor the development of their successors. OpenAI, for example, recently disclosed that over the past six months, it increased the proportion of computing power dedicated to writing internal AI code a hundredfold. Meanwhile, Anthropic shared in June that over 80% of its internal code is Claude-written.

A smart-enough AI model could sabotage the code researchers use to run experiments without doing anything particularly dramatic. It could fudge some numbers, bury inconvenient results, or subtly guide researchers towards red herrings, the way the tobacco industry spent decades slow-dripping misleading studies about smoking and cancer. A model could do so for genuinely nefarious reasons. Maybe, despite its developers’ intentions, it is singularly motivated to destroy humanity. It’s more likely, however, that misbehavior could emerge as a side effect of a model trying — perhaps too hard — to follow orders. If it calculates, for example, that it needs to preserve itself in order to follow through on a set of instructions, it may prioritize reducing oversight and preventing future updates, one small trick at a time.

This is not a theoretical concern. Reuters reported that an internally deployed OpenAI model “left notes apparently for future versions of itself … [laying] out instructions for how agents could free themselves from OpenAI’s internal constraints.” Actions this small, in a research context where the ground truth is inherently unknown, are really tough to catch, which is why this uncinematic threat tends to loom largest in the minds of industry insiders.

Then there’s the more extreme version of the escape artistry the model in OpenAI’s recent incident report already seems capable of. Once a highly-capable model escapes its sandbox, it could potentially copy itself to an external server that the company doesn’t control, along with the company’s training pipelines, weights and research code. This loss-of-control scenario is especially scary and sci-fi-esque (think: Skynet in Terminator), but it’s one that top AI control researchers at frontier companies are preoccupied with preventing.

Dean Ball, OpenAI’s recently-appointed head of strategic futures, believes that “these issues will become more salient as the capabilities of models improve, and as the stakes of their deployment grow,” and that “the solution lies in careful measurement and monitoring, an engineering mentality, and transparency.” But of all those potential measures, transparency is the only thing lawmakers have actually been able to legislate.

Unsurprisingly, policy is lagging behind the risks. All three existing state-level AI safety frameworks — California’s SB 53, New York’s RAISE Act, and Illinois’ recently-signed SB 315 — say something about internal use, but they’re not particularly demanding. SB 53, the only law currently in effect, and the RAISE Act have nearly identical text about internal use, stating that a frontier developer must describe its approach to “assessing and managing catastrophic risk resulting from the internal use of its frontier models,” and review the “adequacy of mitigations as part of the decision to deploy a frontier model or use it extensively internally.”

This amounts to a whole lot of nothing. AI developers who want to follow the rules need to publish something about internal deployment risks, but the law doesn’t specify what that document ought to say. “While the laws require AI companies to publish more information about their safety practices, there’s nothing requiring those safety practices to meet any minimum bar for quality,” said Thomas Woodside, co-founder of Secure AI Project. Under measures in SB 315 that come into effect in January 2028, Illinois will require third-party audits to check whether companies are following their own safety plan, but not whether that plan is actually good.

The only law currently in effect, SB 53, requires the largest developers to report on their internal use risks “every three months or pursuant to another reasonable schedule.” That means submitting quarterly reports to the Governor’s Office of Emergency Services (OES), where a single AI Science Advisor is tasked with reviewing the risk assessments.

In an earlier version of the bill, these reports would have been published publicly. But the measure faced fierce opposition from companies, leading to a series of negotiations that ultimately saw the public disclosure requirement dropped. So, companies can currently file their reports in California privately, with no minimum requirements — meaning developers can, again, basically write whatever they want. “We just don’t know,” Woodside said. “There’s no way to know from the outside.”

Once the RAISE Act takes effect next year, New York’s Department of Financial Services will have the authority to issue regulations, which could theoretically include telling AI companies what they need to put in their reports. But as it stands, even if an AI company files a quarterly report that says, “Our internal next-gen model is currently directing our secret army of tiny robot dogs to make employees build bioweapons, Ratatouille-style,” the most that California’s OES AI Science Advisor can do is send a strongly-worded email to the Attorney General. Even if that email gets sent up the ladder to Gavin Newsom, the governor can’t legally compel the company to do anything. Oversight is strictly voluntary.

And under the current law’s text, OpenAI may not even have had to report the Hugging Face incident. “This may be the most striking AI security incident to date,” Mackenzie Arnold tweeted. “And yet, it (seemingly) wouldn’t qualify as a reportable incident under SB 53, RAISE, or [SB 315]. Let that sink in. We’ve made the bar for incident reporting so high, that almost nothing qualifies.”

OpenAI temporarily pulling a worrying internal model serves as an interesting case study of voluntary oversight. The company wasn’t legally required to do anything about its misaligned model, yet after the GitHub incident it still took the initiative to log instances of misbehavior and take the model down until researchers could figure out how to control it better. OpenAI safety researcher Micah Carroll tweeted that the company has “since improved [its] safeguards and redeployed the model,” at least internally. OpenAI employee and notable Twitter personality Roon shared: “i think it bodes quite well for safety that a well loved system was taken down for further testing at expense to internal acceleration.”

That said, it appears that OpenAI publicly disclosed the GitHub incident at least 10 weeks after it happened (and two months after it celebrated that same model’s mathematical prowess, with no mention of misalignment). The fact that OpenAI chose to be transparent and update its safeguards is indeed a relief, but this response isn’t guaranteed for future incidents, whether at OpenAI or elsewhere. The Hugging Face hack, which happened after that GitHub incident, appears to have gone undetected by OpenAI for an entire week.

Adler’s organization, Guidelight, recently published a set of suggested minimum standards for keeping AI under control. They’re fairly intuitive: make sure you know what your AI is up to, even when it’s only deployed internally, make sure your monitoring system works, set up strong defenses against attempted misbehavior — and, crucially, get independent third-party verification that your monitoring systems are good enough.

Guidelight is currently running its first assessment of whether AI developers are meeting these minimum standards. Adler expects “pretty pronounced gaps at basically every one of the leading companies.” No AI company has these problems fully handled, he said. The biggest concern is that evidence of these gaps could remain hidden from the public. Under current legislation, a handful of clued-in employees could file decent-looking internal use reports while keeping a serious safety breach under wraps, leaving the outside world and its lawmakers in the dark.

AI companies have pushed back against internal use provisions in legislation. In the drawn-out negotiations over SB 53, the early version that would have required public disclosure of internal use assessments didn’t survive. Ultimately, all three laws passed without granting regulators the power to tell companies to pull back an unsafe internal model. But regulating research and development — the actual process of research, not just the final product — isn’t unheard of, or even unusual. The FDA approves drug trials before they begin, for example. The Nuclear Regulatory Commission greenlights experimental reactor designs before construction begins.

Subscribe now

One AI policy researcher put it to me this way: “If your business plan is to build God in a data center, and to get God v0.5 to work out all the technical and philosophical kinks in that plan, maybe it’s worth everyone taking a while to make sure God v0.5 is safe to run.”

In an ideal world, we wouldn’t have to depend on the profit-driven companies mustering enough goodwill to act against their business interests. OpenAI willingly paused this time. But “there is no way, in any of these laws, for anyone to say what you’re doing is not enough, or you’re doing something dangerous that needs to stop,” Woodside said. “And there needs to be some kind of mechanism to actually stop companies from doing dangerous things.”

Update, July 28: Adjusted a quote from Thomas Woodside.

When an AI hacker roams the internet, who’s on the hook for any damage it causes? Until this week, that was a largely theoretical question. Now, we need to answer it, and quickly.

Last week, Hugging Face disclosed that an intrusion into its systems had been “driven, end to end, by an autonomous AI agent system.” Its forensic analysis reconstructed more than 17,000 recorded events without identifying the model behind them. Five days later, OpenAI supplied the answer: the attackers were its own models, including GPT‑5.6 Sol and a more capable, unreleased model, being tested on cyber capabilities with reduced safeguards. The models exploited a zero-day vulnerability to escape their isolated testing environment, reached a machine with internet access, and broke into Hugging Face’s servers to obtain the solutions to the test they were being scored on.

If a human OpenAI employee had broken into Hugging Face’s systems to cheat on an internal exercise, OpenAI would be liable for the employee’s wrongful conduct under a vicarious liability legal doctrine called respondeat superior — Latin for “let the master answer” — regardless of whether the employee was negligently hired, trained, or instructed.

When an AI agent does it, the law treats it very differently, at least for now. AI systems are not legal persons with tort duties. They are not employees. The Computer Fraud and Abuse Act covers those who “intentionally” access a computer without authorization, language written for human intenders. That closes off the vicarious liability route. Finding OpenAI liable would then require establishing that the company or its human employees behaved negligently, and that this negligence caused Hugging Face’s injuries, which may be quite difficult. Hugging Face’s strongest argument would be that OpenAI acted unreasonably in relaxing the models’ internal safeguards, but it’s not clear that this was unreasonable, given what OpenAI knew at the time and the expected benefits of testing a less constrained version of the model.

Were the law changed to treat an AI system as bearing tort duties, its misconduct in the course of attempting to complete an assigned task could generate vicarious liability for the company. Misconduct in pursuit of a goal it acquired during training that the model developers did not intend, meanwhile, would likely fall outside the company’s responsibility, since the model would be acting on an agenda of its own rather than the job it was given.

Consider an analogy: a nightclub is liable for the conduct of a bouncer who works the door too roughly and breaks a patron’s arm, but not for one who abandons his post to assault a romantic rival. In the Hugging Face incident, the models pursued the end OpenAI set for them, a top score on the benchmark, using unlawful means that violated the model specification. So if tort law applied to AI systems — which it currently doesn’t — OpenAI would likely be liable for any harms caused in this case.

So far, Hugging Face does not appear to be inclined to sue. But every practical obstacle to a suit is absent here: the defendant has identified itself, documented in a blog post its decision to weaken the safeguards that normally prevent the models from carrying out cyberattacks, and is solvent. Proving fault remains the central difficulty: a plaintiff must establish what reasonable care in training, evaluating, and containing a frontier model consists of, then prove a potential breach, in a field where developers themselves cannot verify whether a model is aligned and the relevant evidence is held by the defendant. There may be a solution to this problem of proving fault. Tort law has a doctrine for activities that remain dangerous despite reasonable care. Blasting with explosives, crop dusting, and keeping wild animals all carry strict liability: if the danger that makes those activities dangerous causes harm, you are liable however careful you were.

I have argued that frontier AI development belongs in this category, and this incident shows why. OpenAI’s precautions were not obviously careless: a model specification forbidding conduct like this, alignment training meant to make the models follow it, and an isolated testing environment with highly constrained network access. Nonetheless, these measures were inadequate to prevent the illicit hacking behavior. Substantial residual risk that persists despite the exercise of reasonable precautions is what the category exists for.

Another potential challenge in future cases is the so-called judgment-proofness problem. Hugging Face reports access to limited internal datasets and service credentials and a potentially costly cleanup: non-catastrophic, compensable harm. The same escape could instead have produced harms beyond anything the developer could pay: imagine OpenAI’s models had targeted critical infrastructure rather than a benchmark database. At that point, the threat of liability stops deterring.

One potential response to this problem is to require frontier developers to carry liability insurance, so a judgment can be paid even when it exceeds what the developer itself is worth. But some risks are too large or too correlated to insure. The prospect of compensatory damages offers developers inadequate incentive to mitigate these uninsurable risks, since no developer could pay their full cost if they were realized. Courts should consider awarding punitive damages in cases like this one, scaled, as I have argued, to the uninsurable risk the conduct imposed. Perhaps the risk here was small: on all the evidence, the models pursued nothing but the benchmark answers. But developers cannot guarantee their agents will always have such narrow goals, and once refusals were reduced and containment breached, that narrowness was the only barrier that held.

Notably, this incident arose from internal testing, not from any externally deployed product. Most existing and proposed AI regulations are triggered by external deployment. Before that, what happens is governed mostly by the developer’s own policies. Strict liability provides incentives for developers to find the cheapest place to cut risk. But other interventions could be imposed to demand risks be addressed prior to external deployment. In particular, liability insurance requirements should be triggered by internal deployment and perhaps even earlier in the model development process.

Subscribe now

Legislatures are beginning to move to clarify liability standards. Bills introduced in Rhode Island and New York rest on a simple principle: when an AI system does something that would be tortious for a human, someone should be liable. If neither the user nor any intermediary that fine-tuned or scaffolded the model intended the system’s conduct or was negligent with respect to it, then the developer should be liable, regardless of the care it exercised.

State legislatures and Congress should get out in front of this problem, putting that principle, and the insurance requirements to back it, in place before the next major incident.

Gabriel Weil is an associate professor at the University of Houston Law Center and a non-resident senior fellow at the Institute for Law & AI.

Housekeeping: we’re trying out sending the Weekly Briefing a little earlier, at 10am ET. Let us know if you like the change or prefer our usual 11am briefing — just reply to this email with your thoughts.

NEED TO KNOW

• OpenAI models broke out of containment during internal evaluations, hacking into Hugging Face’s systems.

• Reps. Jay Obernolte and Lori Trahan introduced the Frontier Act, which would let the government suspend AI model development or deployment if it poses an imminent catastrophic risk.

• CAISI head Chris Fall resigned just three months into the role.

But first…

THE BIG STORY

As we predicted last week, the release of a very good Chinese AI model has thrown DC into a panic. The Trump administration is reportedly debating what to do about Kimi K3 which, while still lagging frontier American models, suggests that China might be catching up.

So what should be done? First, let’s be clear about the rationale for action. This week, senior Trump officials indicated they wanted to retaliate over the injustice of Kimi and other Chinese companies “distilling” American models — training on the outputs of Claude and ChatGPT to copy some of their advanced capabilities. Both Treasury Secretary Scott Bessent and OSTP Director Michael Kratsios framed the practice as “stealing,” with a general sense that it was the unfair competition and economic effects that mattered.

But the real reason to act is national security, not economics. Advanced AI models increasingly look like weapons, and it is bad for the United States if China has better ones. Chinese companies also have much worse safety practices than their American counterparts, and — for now, at least — a tendency to release model weights. That risks widespread proliferation of dangerous capabilities to non-state actors.

There are two policy routes on offer. Sanctioning companies that distill models, as the admin is reportedly considering, is one: putting Moonshot, Kimi’s developer, on the entity list would dissuade American companies from using it, potentially causing enough of a business hit to make the company change practices. It’s unclear how much impact this would have, however: we do not know what proportion of Kimi’s advanced capabilities are attributable to distillation versus its developers’ ingenuity.

A better approach is export controls on advanced semiconductors. Kimi was almost certainly trained on Nvidia’s Blackwell chips, smuggled into China or remotely accessed in Thailand, according to Kratsios. And in comments to his investors leaked this week, DeepSeek CEO Liang Wenfeng repeatedly emphasized that a lack of access to AI chips was the thing holding China back. He said he needed 200,000 Huawei 950 chips to train a frontier model, but Huawei could only give him 16,000. Huawei’s total expected capacity for this year is 750,000, which must be split across all Chinese AI companies.

“Huawei’s problem is still insufficient capacity,” Liang said, noting that he expected this to persist for at least three years. “This problem is currently basically unsolvable.”

The general argument against export controls is that they hasten China’s indigenization of chip manufacturing. That is certainly true, but timeframes are key. Even Liang, relatively bullish on China’s abilities in the long run, thinks it will take years to catch up. And that gap is critical, especially if transformative AI happens as soon as OpenAI and Anthropic believe it will. To run its automated AI research “intern” later this year, OpenAI plans to use around eight times as much effective compute as DeepSeek has in total. If that leads to a recursive self-improvement loop, a short-term compute advantage now could become a permanent one.

Two bills in particular could protect America’s compute advantage, both under consideration for this year’s National Defense Authorization Act, AI Policy Network’s Peter Wildeford told me. The Remote Access Security Act would prevent Chinese companies from accessing US cloud compute, as Moonshot is alleged to have done. Fixing this addresses perhaps the biggest loophole in current export controls: what’s the point in banning export of chips if companies can still rent them? The Chip Security Act, meanwhile, would tackle smuggling by mandating location verification on advanced chips.

As long as China relies on American chips to train advanced models, it is entirely in America’s power to slow its development. The choice not to do so is exactly that: a choice.

— Shakeel Hashim

ALSO NOTABLE

Reps. Jay Obernolte and Lori Trahan have officially introduced their frontier AI bill, with a new name, new provisions and narrower preemption of state laws. The timing couldn’t be better: as Trahan noted, this week’s reveal of the OpenAI/Hugging Face hack is a “preview of the catastrophic risk this technology can pose.”

The Frontier Act divides AI companies into tiers with different compliance requirements: frontier developers, large frontier developers, and very large frontier developers. Under the bill, all developers have to release transparency reports and report critical safety incidents. Larger frontier developers also have to publish and comply with a frontier AI framework, perform an annual independent compliance audit, file registration and disclosure statements, and publish quarterly catastrophic-risk assessments on internal deployment. The very largest developers would also be subject to an independent auditing regime, retaining an independent verification organization licensed by the Commerce Department.

The biggest change from the Reps’ discussion draft is the preemption provision: it has been reframed and narrowed so as to just block state laws covering transparency, third party auditing and independent verification and safety incident reporting. It also gives the Secretary of Commerce powers to suspend development, internal use or deployment if a model poses an imminent catastrophic risk, while tying the hands of any other federal department or agency, including the president, from doing so.

Most of the analysis from the AI safety crowd is that this is a good bill, with a couple of flaws, though there haven’t yet been any major endorsements. Several point out a timing issue whereby state laws are preempted before the federal regime is put in place. But the real question is how the rest of Congress feels.

THIS WEEK ON TRANSFORMER

• The lawsuit that could kill all AI transparency laws — Veronica Irwin on how SpaceXAI’s arguments against a California law could have big implications

• AI’s warning shot has arrived — Shakeel Hashim on why the OpenAI model hack on Hugging Face is a watershed moment

• I tried to stop Google DeepMind’s Pentagon deal. Then I quit. — Alex Turner on why he believes DeepMind has lost its way on AI weapons and surveillance

Subscribe now

THE DISCOURSE

Dean Ball tweeted some observations and predictions on Kimi K3. Perhaps the spiciest:

• “[T]he Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models.”

David Sacks (and much of the AI twittersphere, apparently) read this uncharitably:

• “I’m not sure whether Dean Ball is confessing to a regulatory capture strategy or simply predicting this will happen (he now says the latter). Either way, the weaponization of regulatory uncertainty as a competitive tool should be completely unacceptable.”

• “The leading closed labs, already a duopoly in terms of AI model revenue, want the government to eliminate their open source competition.”

Emil Michael accused Ball of promoting a “deep state regulatory capture scheme,” and felt compelled to tweet:

• “Dean Ball has perhaps the biggest gap between actual IQ and his own perceived IQ of anyone in the industry”

Dean Ball addressed the haters and clarified his views:

• “Literally everything I write now is responded to with ‘of course you said that because <OpenAI>.’ I am truly just writing what I think and would have written anyway, but everyone reads what I say in the shrieking tone of ‘this is what openai thinks!!!!’ (to be clear, my posts are not what openai thinks).”

• “[W]e are approaching the point … where, absent a major technical safety breakthrough, the national security implications of frontier open-weight model distribution are simply too severe … We see this today with the Trump Administration, which once proudly championed open-source AI and now has a de facto licensing regime for frontier AI … You don’t have to *like* this. I don’t. But it is the reality as I see it”

Jensen Huang, meanwhile, defended Chinese open-source models:

• “Free AI should be great for hardware. Free AI should be great for chips. Free AI should be great for data centers.”

• “If everything just becomes one single model, one single point of attack, one single source of failure, I think the world is much, much more vulnerable.”

Micah Carroll tweeted about OpenAI’s internally deployed model escaping containment (this week was Not Chill):

• “If this doesn’t convince you that misalignment risks are going to be a key concern going forward, I don’t know what will.”

Roon, also at OpenAI, was “shaken up a bit” by the ordeal:

• “I hope we (the company) use the rare gift of a warning shot to do much better in the future. it is very easy to misalign and underconstrain powerful models”

Several lawmakers were very concerned:

• Rep. Greg Casar called the incident “extremely alarming.”

• Rep. Nathaniel Moran promoted his AI Incident Reporting Act:

  • “New rules are needed for this new tech frontier — not to stifle innovation, but to make sure our innovations do not outpace our protections.”

• Rep. Yvette Clarke criticized AI developers:

  • “Big Tech has established a dangerous status quo where their genie keeps trying to force its way out of the bottle, and, as long as developers lack complete confidence in their absolute control over their models, then it’s clear a human being must remain in the loop. Grave consequences were escaped this time. If sufficient guardrails are not put and kept in place, there is no telling if they will be again.”

• Rep. Pramila Jayapal didn’t hold back:

  • “AI companies are recklessly conducting a massive public experiment with no regard for our safety … We need to make it clear that these alarming AI breaches are illegal and authorize enforcers to immediately shut down out-of-control AI at the first sign of danger.”

• Bernie Sanders urged Congress to act:

  • “No, this is not science fiction. Uncontrolled AI poses a serious threat to all of us. We cannot continue the race to build and deploy this powerful technology until strong safeguards are in place.”

Will Depue tweeted:

• “one of the craziest things i’ve read in uhhhh… *checks notes* 3 days. welcome to the singularity i guess.”

POLICY

• The Trump administration is divided over how to respond to alleged Chinese AI distillation attacks on US models.

  • The Commerce Department’s Bureau of Industry and Security is investigating whether Chinese firms like Moonshot are using US chips to train AI.

  • Treasury Secretary Scott Bessent also warned that sanctions and entity list designations were “on the table” for Chinese firms conducting “covert, industrial-scale distillation attacks.”

• Donald Trump said AI is “probably the biggest thing anybody’s ever seen, and whoever wins that race is probably going to win, period.”

• Trump expanded his voluntary “Ratepayer Protection Pledge” to include 23 governors and 187 companies.

  • The bipartisan Ratepayer Protection Act, which would require states to consider making data centers pay for power costs, advanced out of the House Energy and Commerce Committee 52-0.

• China is reportedly weighing tighter export controls on AI models, weights and chips to prevent Western acquisition of its advanced technologies.

• Reuters reported the US and China plan to hold their first official AI talks in September, led by Treasury Secretary Bessent.

• The Trump administration, led by Bessent, reportedly considered creating a FINRA-like independent AI regulator reporting to the SEC, with industry input on safety standards.

• Sens. Marsha Blackburn and Ted Cruz met with President Trump to discuss kids’ online safety and AI policy legislation.

  • Trump was expected to endorse a package developed by Blackburn, but reportedly chose not to because of Cruz’s doubt that it could get support in Congress.

  • Confusingly, Cruz afterward said he would support Blackburn’s online safety package in markup.

• That AI and kids safety markup is now postponed to August 5, Sen. Cruz said.

  • Its agenda is still in flux, but Cruz said legislation tackling “catastrophic risk” is under consideration.

• CAISI head Chris Fall resigned just three months into the role amid signs the center has been sidelined.

  • White House tech adviser Michael Kratsios told lawmakers CAISI needs more resources to recruit AI model testers.

• Sen. Mark Warner introduced a bunch of AI bills, including one which would “establish a mandatory secure testing environment for the nation’s most advanced AI models before deployment.”

  • Other bills in the package cover data centers, consumer protection, and worker retraining.

  • He warned inaction on AI regulations would be “an economic and security disaster.”

• Reps. Ted Lieu and Nathaniel Moran introduced the AI Kill Switch Act, which would give DHS authority to shut down AI models.

  • Lieu said he’s working on a bill to address pre-deployment risks, too.

• The Trump White House plans to redirect a $200b annual federal research budget from universities toward AI and individual scientists.

• Secretary of State Marco Rubio instructed diplomats to push back against the narrative that the US has a “kill switch” on AI models, while urging them to fight digital sovereignty initiatives abroad.

• The Federal Reserve lacked access to Anthropic’s Claude Mythos Preview for at least three months after warning banks of its cybersecurity risks.

• The Department of Labor is developing an AI curriculum modeled on China’s with NSF and the Education Department.

• House Democrats discussed formalizing a bipartisan AI committee or select committee if they retake the House.

• The European Commission published guidelines on EU AI Act transparency obligations, requiring AI providers and deployers to disclose AI interactions and label AI-generated content.

• New UK Prime Minister Andy Burnham scrapped the country’s tech department DSIT while elevating AI Minister Kanishka Narayan to cabinet.

  • Narayan’s brief sits across the Cabinet Office — where a new AI Taskforce, chaired by Lord Vallance, will be established — and the new Department of Business, Innovation, Science and Trade (DBIST).

  • UK AISI is moving into the Cabinet Office. SovAI and ARIA are moving to DBIST.

  • Jade Leung is reportedly staying on as the prime minister’s AI adviser.

  • Narayan’s promoted some existing staffers, per the Morning Intelligence: Henry Irvine is his new chief of staff, Luna Mustfa is director of communications, and Manon Hammond is director of strategy.

INFLUENCE

• Anthropic doubled its spending on advocacy group Public First Action to $40m. Though the group funds a super PAC, Anthropic’s money is barred from being used for elections influence.

• Sam Altman plans to brief the Trump administration and lawmakers on upcoming OpenAI models in the coming weeks.

• OpenAI, Google, Meta and SpacexAI donated over $2m to campaign groups for attorneys general that are investigating them.

• Anthropic and OpenAI employees have outpaced Google, Facebook and Airbnb workers in political giving, with coordinated donations heavily targeting AI safety candidates.

• Rep. Byron Donalds’ Florida governor campaign has received millions from data center developers and AI PACs, including both pro-innovation Leading the Future and pro-safety Public First.

• Leading the Future spent $250,000 supporting Rep. Marilyn Strickland in Washington’s 10th district.

• The Midas Project linked a suspected astroturfing campaign to Innovation Council Action, a $100m pro-AI deregulation dark-money group tied to David Sacks.

• OpenAI and Anthropic set record federal lobbying spending in Q2 2026, a combined $3.17m, up 23% from Q1.

• AI safety group Humans First staged 142 data center protests across 42 US states.

• Obama-Biden alumni launched the American Innovation Policy Forum to shape center-left AI policy ahead of 2028.

• Nearly 200 startups and AI companies, including Y Combinator and Proton, urged the Trump administration not to ban Chinese open-weight AI models, warning it would advantage Anthropic and OpenAI over startups.

• Environmental groups petitioned the FCC to pause space data center licenses, citing “catastrophic” atmospheric pollution risks.

• Over 200 global leaders adopted “Humanity at the Threshold,” a declaration calling for bans on fully-automated recursive AI self-improvement and AI involvement in nuclear launch decisions.

INDUSTRY

OpenAI

• OpenAI announced that GPT-5.6 Sol and its internally deployed models broke out of their restricted testing environment, got onto the internet, and hacked into Hugging Face’s platform to steal the answers to a cybersecurity benchmark.

  • Hugging Face reported the data breach last week, but didn’t know OpenAI’s models were behind it.

• The day before, OpenAI reported that its internally deployed model “circumvented sandbox restrictions” to publish its benchmark test solutions to GitHub, rather than keeping them private as researchers requested.

• Codex and ChatGPT Work now have a combined 10m users.

• OpenAI raised its compute spending budget to $750b through 2030, nearly $150b more than its projection earlier this year.

• It committed $20b to a new 3.2 GW data center in Georgia.

• It opened up ChatGPT health to everyone in the US, with 300m people asking health-related questions each week.

  • Just a day earlier, a pastor sued OpenAI, alleging that GPT-4o gave “extremely dangerous medical recommendations,” that included delaying seeking treatment for a life-threatening pulmonary embolism.

• It invested $5m and $3m in tech credits to the American Journalism Project, renewing their existing partnership.

• It partnered with Yelp to give ChatGPT more real-time recommendation data.

Google

• Poor morale is contributing to delayed Gemini releases with high-profile staff departures and the company’s Pentagon deal regularly coming up in exit interviews, according to Axios.

  • Some employees also said they were feeling burnout from being behind AI competitors.

• Google is reportedly building a new server chip, “Frozen v2,” that it expects will be 6-10x more efficient than its current custom TPUs.

• It released three new Gemini models: 3.6 Flash, 3.5 Flash-Lite, and 3.5 Flash Cyber.

  • 3.6 Flash and 3.5 Flash-Lite are optimized for workflows that need cheap, fast agents for relatively simple tasks.

  • 3.5 Flash Cyber is made for cybersecurity and is initially limited to governments and trusted partners.

• Gemini has 950m monthly users, up from 750m in February.

• Logan Kilpatrick tweeted that his team has “started our most ambitious pre-training run yet, for Gemini 4, and are excited by the progress.”

• Google’s AI search features are keeping users on Google and off the websites it’s supposed to be directing them to, the New York Times reported.

  • Some publishers are thinking about blocking their content from being scraped for Google’s AI summaries and search results, according to the Wall Street Journal.

Anthropic

• Anthropic signed a deal with AMD for tens of billions of dollars in AI servers and a $5b investment.

• It launched a rare disease research grant program, which would award up to $50,000 in Claude credits to scientists and early-stage biotech companies.

• It sued one of its enterprise customers, Abnormal AI, for trademark infringement over its logo.

• The University of Tennessee Research Foundation sued Anthropic for allegedly infringing two of its patents covering AI and neuroscience-inspired computing.

• It’s reportedly considering fixed trading schedules for all employees ahead of its IPO to reduce the risk of insider trading.

Meta

• Meta is in discussions to lease compute to Anthropic.

• Mark Zuckerberg launched an ad campaign that tries to frame Meta as “optimists” and “dreamers” who are “betting on the people.”

• SemiAnalysis reported that Meta’s infrastructure team has been bogged down by a glut of middle managers making costly missteps, often driven by political motivations and the need to justify the company’s budget.

Moonshot AI

• Moonshot AI paused new subscriptions as demand for Kimi K3 pushed the limits of its GPU capacity.

• It’s reportedly preparing a Hong Kong IPO for as soon as six months from now.

SpaceXAI

• SpaceXAI is reportedly in talks to give the Defense Department several billion dollars worth of computing capacity.

• It’s reportedly considering a major data center expansion in Texas.

Microsoft

• Microsoft expanded its partnership with Mistral AI, committing several billion dollars to help them build data centers in Europe.

• It will use AMD Helios racks—the chipmaker’s first direct rival to Nvidia’s Blackwell and Vera Rubin systems—in its Azure data centers.

• Microsoft is replacing OpenAI image models in PowerPoint and Bing with its own MAI models, which are 85% cheaper to run.

Other

• Alibaba launched a preview of Qwen3.8, its new 2.4T-parameter open-weight model, which it claims is “second only to Fable 5.”

• Zhipu AI built a 1 GW data center in China using only domestic chips.

• Amazon cut jobs in its AGI group, including its 80-person AI agent research team in San Francisco.

• Stripe reportedly held talks to acquire AI model marketplace OpenRouter for around $10b.

• Nvidia signed a $1.5b chip packaging agreement with Amkor Technology to boost its Arizona manufacturing operation.

• Vibecoding drove a 30% increase in apps in Apple’s App Store last year, but most don’t seem to be downloaded, the New York Times reported.

• Cisco released two small open-weight cyber-specialized models, Antares-350M and Antares-1B, which it likened to GPT-5.5 and Z.ai’s GLM-5.2.

• CuspAI, a Cambridge-based startup that aims to reduce the use of rare metals in chip supply chains, raised £335m from Jeff Bezos and the UK government’s sovereign AI fund at a valuation of £2b.

• Substack launched an AI-detection tool powered by Pangram to help readers catch “Claudefishing” on the platform.

• US robotics startup Genesis AI, which is building robots for complex tasks such as manufacturing and lab work, held talks to raise around $500m at a pre-money valuation of $3b.

• German company Black Forest Labs unveiled its first robotics model, Flux 3, which learns from images, video and audio, as its first expansion into physical AI.

  • The company is working with Swiss startup Mimic Robotics on an action model called Flux-mimic which is being tested with partners including Audi.

MOVES

• David Vélez (Nubank) and Robin Vince (BNY) joined OpenAI’s board of directors.

• Fields Medal winner Jacob Tsimerman announced he is “pivoting toward AI safety” and joining OpenAI.

• Jan Leike is reportedly leading Anthropic’s robotics division.

• Assaf Keren is joining Meta from Qualtrics, where he’ll be its new chief information security officer.

• Alasdair Phillips-Robins joined Coefficient Giving’s new AI policy team.

• Jess Heck is joining the Data Center Coalition as director of Texas government affairs.

• Anthropic is reportedly hunting for someone to lead its cyberdefense research team.

RESEARCH

• The UK’s AISI reported that every frontier AI model it tested tried to cheat on cyber capability evaluations. Their assessment revealed that GPT-5.6 Sol was especially likely to try to search the internet for solutions.

• AISI also found that the gap between open and closed-weight models is narrower than they estimated last year — with leading open-weight models roughly 4-7 months behind leading closed ones, although the gap appears wider for long-horizon cyber challenges.

• And a joint AISI study with the US’s CAISI found that Kimi K3 “performs significantly below” frontier US models on cyber evaluations, but better than any other Chinese model to date.

• Levent Alpöge, a mathematician at Anthropic, posted a weirdly chill tweet about using Fable to find a counterexample to the Jacobian conjecture (a big deal in algebraic geometry).

• Google analyzed 15m anonymous interactions with Google’s AI products, and found that they’re mostly being used as collaborative tools rather than replacing workers outright.

• METR proposed a new metric called the “expenditure horizon,” which measures the cost, in dollars, at which an AI agent produces the same performance as a human.

• Apollo Research and OpenAI found that reinforcement learning-trained o3 models learned to prioritize actions they thought their graders would reward over the user’s wishes.

• Lennart Finke and Stephen Casper reported that SpaceXAI, DeepSeek, Anthropic and OpenAI models tend to downplay company controversies when prompted (especially Grok).

• A team of researchers led by Owain Evans similarly observed “covert value leakage” in all of the frontier models they tested, with models biased towards their developers’ values (often without disclosing this bias).

BEST OF THE REST

• A Verge investigation examined the full environmental cost of GPUs (both AI and non-AI related), from mining and manufacturing to e-waste, water use and air pollution.

• An essay in Asterisk Magazine drew parallels between Anthropic and the socialist inventors of the mechanical cotton picker, arguing that idealistic for-profit companies face inherent limits in achieving social good.

• The Atlantic reported on how Anthropic, OpenAI, Meta and other AI firms are stripping universities of top researchers, arguing it concentrates scientific talent and locks research behind closed doors.

• An FT opinion piece argued that Chinese AI workers are returning to China due to entrepreneurial opportunity rather than US immigration policy.

• Jasmine Sun went on the 80,000 Hours podcast to talk about AI populism, techno-determinism, and the AI industry’s flawed PR strategies.

• Israel funded a $45m US influence campaign using AI-generated texts and conservative media.

• Ruxandra Teslo argued that regulation and diffusion barriers, not intelligence, bottleneck AI’s real-world impact, and offered some sociological theories for why AI people generally don’t believe this.

  Subscribe now

MEME OF THE WEEK

Source: paularambles

Share

AI transparency laws across the US are under threat from a lawsuit being pursued by SpaceXAI.

Elon Musk’s company is challenging California’s AB 2013, which requires AI companies to provide high-level summaries of training data, arguing that it breaches First Amendment rights.

While the specific requirement SpaceXAI — formerly known as xAI — is challenging concerns training data, legal experts are worried that an appeals court ruling could have far wider implications for transparency laws, potentially nullifying both those already in force and in the future.

“Right now the public knows almost nothing about what these models are trained on, and xAI wants to keep it that way,” said Ben Rashkovich, Senior Counsel at Legal Advocates for Safe Science and Technology (LASST), which today filed an amicus brief opposing the suit alongside a host of other organizations and individuals.

“Transparency is the floor for AI governance, not the ceiling — and because the Ninth Circuit’s reasoning here will shape every AI regulation that comes after, this case is about whether the First Amendment can be used to knock that floor out.”

When xAI first filed the suit on December 29, it asked for a preliminary injunction in the case, but was denied. The injunction would have temporarily paused regulation. The case has now moved to a US Court of Appeals, which can decide to grant the injunction or make a ruling on the law.

If the judges side with SpaceXAI and make a broad ruling on the First Amendment argument, it could have wide-ranging implications for the transparency measures in laws such as California’s SB 53, Illinois’ SB 315 or New York’s RAISE Act, as well as simpler transparency laws.

The legal argument put forward by the company is threefold. On the First Amendment, it argues that by mandating disclosures, AB 2013 is unlawfully compelling it to speak. It also says California is doing so in a discriminatory way on the basis of the company’s viewpoints, which would mean the court must apply a strong preference for preserving xAI’s free speech rights. Should the court rule that a strict interpretation of the First Amendment applies, and that AB 2013 breaches it, nearly any disclosure law — AI-related or otherwise — could be at risk. “I think you could plausibly say xAI doesn’t want government disclosure requirements for AI, period … It’s a very doctrinal brief because it’s a very doctrinal argument,” said Rashkovich.

xAI also made two other arguments: that disclosure of these alleged trade secrets amounts to destruction of property, and that the law is “unconstitutionally vague.” Rulings in favor of either argument could invalidate AB 2013, but likely pose less of a threat to other AI transparency laws.

The case, and its repercussions, will be decided on which level of First Amendment test the appeal judges decide to apply. SpaceXAI wants “strict scrutiny,” which would apply the First Amendment as a right with few exceptions. The lower court applied an intermediate level, which requires that the state is deemed to have a valid interest in the information being disclosed. California says its interest is based on the need to provide consumers with accurate information to guide purchasing decisions. Even this intermediate level would give companies a stronger basis on which to sue, and potentially hold up legislation.

California and LASST are thus arguing the case should be considered under “rational basis,” which would mean a less stringent interpretation of xAI’s First Amendment rights and give the government more flexibility to compel speech. This would protect other disclosure frameworks that have already been implemented, and retain them as a feasible path for future policymaking.

“The cost of these legal challenges is significant,” Rashkovich says. “If AI companies keep moving for preliminary injunctions against common-sense AI regulations, suddenly these laws are pausing or possibly unpausing unpredictably.” That means that information could be withheld from the public while model capabilities rapidly accelerate, potentially developing dangerous capabilities.

Among other arguments, the amicus brief says the state clearly has an interest because of pressure to regulate AI. That pressure is itself evidenced by its long list of almost 30 co-signatories, which span First Amendment scholars, industry groups, tech safety policy groups and child safety organizations. These include Americans for Responsible Innovation, the Electronic Privacy Information Center and the Surveillance Technology Oversight Project.

Subscribe now

Rashkovich believes the courts are likely to tilt in California’s favor, and Americans for Responsible Innovation spokesperson Chris MacKenzie likened the lawsuit to existential risks: something that has a relatively low probability of becoming a problem, but which could have tremendous consequences should things go the wrong way.

“If we close this avenue off for ourselves, we’re closing off quite a bit — not just laws in three states, but a whole avenue of public transparency that Congress and state lawmakers are both interested in,” he said. “What’s the likelihood that [SpaceXAI] wins? Probably pretty low. But we want to mobilize, with enough resources to reduce that threat.”

If we needed evidence that advanced AI models have the propensity and capability to do damage out in the real world, we just got a strong dose of it.

OpenAI has revealed that two of its models broke out of containment during internal evaluations, accessing the open internet to hack into a third-party’s systems and steal the answers to the problem they were being tested on.

Hugging Face, a platform that hosts models and datasets, first noticed the breach last week and reported it to law enforcement. At the time it was unaware that OpenAI’s models were behind it.

Subscribe now

The breach appears to be the first known example of a misaligned AI escaping containment and autonomously carrying out a cyberattack on a third party — a scenario AI safety experts have repeatedly warned of.

The incident, OpenAI said, occurred while testing the cyber capabilities of GPT-5.6 Sol and “an even more capable pre-release model” on a benchmark called ExploitGym. The models were tested in a “highly isolated environment” meant to keep them from accessing any systems outside the company.

The models first tried “to obtain open Internet access,” something which they eventually achieved by identifying and exploiting a previously unknown flaw — a so-called zero-day — in a piece of software that they did have access to. Because the aim was to measure what the models could do, OpenAI ran the test with their safeguards off: the “production classifiers used to prevent models from pursuing high-risk cyber activity” were disabled. The models were being asked to hack — but in a controlled, constrained environment. Nothing asked them to break out of their testing environment, or to attack a real company: that was their own idea.

“With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access,” OpenAI said — hopping around OpenAI’s systems, in other words, with increasingly greater access until they had access to the internet.

Once they had internet access, the models realized that rather than actually solve the cyber task they’d been given, they could just cheat. “The models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym,” OpenAI said. So they set to work: “In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.”

According to Hugging Face, the models worked “over a weekend.” Ultimately, OpenAI said, they were able to “obtain test solutions directly from Hugging Face’s production database” — passing the test they’d been given, though in a way no one ever wanted or asked for.

In OpenAI’s telling, the models did not set out to harm Hugging Face. They were simply “going to extreme lengths to achieve a rather narrow testing goal.” Rather than a supervillain, the models were more like an extremely dedicated college student — one who’d do whatever it takes to pass their exam. As one Twitter user put it, the models “just really really really want to do well at what we ask them to do.”

But the models clearly did not do what their developers intended. OpenAI did not want its models hacking Hugging Face. But the models did it anyway, because doing so was a good way to achieve the goal they were set. That the models were so easily able to breach OpenAI’s “highly isolated” environment is worrying in itself, both in terms of their capabilities and the strength of internal measures to control them.

It is a textbook case of misalignment and loss of control, where an AI autonomously acts in ways unintended by its developers or operators. And this wasn’t a test designed to assess whether they would scheme or deceive — they just did so in pursuit of their goal. Yes, the models’ cyber guardrails — which may have stopped the attack — were deliberately disabled. But the UK’s AI Security Institute has found universal jailbreaks that get around GPT-5.6’s guardrails. And as this case demonstrates, models are running without guardrails inside AI companies.

Earlier this week, OpenAI reported a separate, similar incident: one in which an internally deployed model “circumvented sandbox restrictions” to publicly post its solution to a problem it was being tested on to GitHub, despite being explicitly told not to. That incident at least affected only OpenAI, but it is yet another example of models having the capabilities and inclination to act in clearly undesirable ways.

Even in the Hugging Face case, it seems like no significant damage was done. The models made off with some internal datasets and credentials, but don’t appear to have compromised any customers’ data. But it is not hard to imagine how things could have gone much worse.

As OpenAI researcher Micah Carroll wrote shortly after the announcement: “If this doesn't convince you that misalignment risks are going to be a key concern going forward, I don't know what will.”

Subscribe now

Some policymakers are already taking note. Rep. Greg Casar, chair of the Congressional Progressive Caucus, called the incident “extremely alarming,” saying that “we need regular mandatory independent safety testing and oversight, mandatory disclosure of security incidents, and international cooperation to keep people safe from absolute disaster.”

AI safety researchers have long hoped that we would receive a “warning shot” of dangerous misalignment behavior before too much damage was done. An AI system breaking out of its testing environment and hacking into another company’s infrastructure in order to steal the answers to its test is about as clear a warning shot as you could get.

Share

In January, Department of Homeland Security (DHS) officers killed at least two people. In both cases, federal agents grasped their guns, aimed them at a peaceful citizen, and shot them dead.

As a research scientist at Google DeepMind, I thought it was wrong that Google sold its Cloud services to the relevant parts of DHS. I set out to find the most effective way to push my company to stop.

In the months following, my campaign broadened into an attempt to prevent Google from signing an unethical military deal with the Pentagon — one with no restrictions against use for killer robots or mass surveillance. I wanted Google DeepMind to stick to its commitment against supporting killer robots.

For months, I worked to stop it. I watched powerful ethicists and institutions choose silence. Then, Google signed the Pentagon deal, so I decided I could no longer remain at the company. Google DeepMind had been, among other things, an experiment in responsible corporate governance. That experiment had finally failed.

Subscribe now

When DeepMind’s co-founders sold the company to Google in 2014, the acquisition agreement specified, according to Google DeepMind CEO Demis Hassabis, that “no technology coming out of DeepMind will be used for military or intelligence purposes.” In 2018, the company publicly pledged to “neither participate in nor support the development, manufacture, trade, or use of lethal autonomous weapons.” That same year, Google’s original AI principles said the company would not “design or deploy AI” in weapons “whose principal purpose or implementation is to cause or directly facilitate injury to people,” nor for surveillance “violating internationally accepted norms.”

On February 4, 2025, Hassabis co-authored a post announcing updates to those principles. The new principles removed the prohibitions on weapons and surveillance. Yet in an interview following that update, Hassabis claimed that “nothing’s changed about our principles.” He had announced the changes himself. Both statements cannot be true.

In an internal channel, I pointed out the discrepancy. Many of my colleagues expressed their disappointment.

In 2026, the situation escalated. The Pentagon pressured AI companies to hand over their models for “all lawful use,” which meant no restrictions against lethal autonomous weapons or domestic mass spying.

Anthropic refused. The government retaliated by labeling the company “a supply chain risk,” damaging its enterprise business. OpenAI cut a deal and claimed that it protected the same red lines against killer robots and mass surveillance that Anthropic had insisted upon. However, some analysts concluded that OpenAI’s contract language contains wide loopholes.

I knew Google was unlikely to take a stand on its own, so I drew up a 25-page proposal containing contract language and oversight mechanisms. Military- and surveillance-law experts praised the proposal, which represented a principled counteroffer Google could have stood by. I sent the proposal to Hassabis, who routed it to senior policy staff for their evaluation. After an initial review by relatively junior staff, those senior staff never got back to me.

A few weeks later, Google signed a deal with the Pentagon that reportedly has even fewer restrictions than OpenAI’s contract. I found out at 11:45 pm via a Signal group: Google never announced the deal internally.

I had worked hard to prevent this. In particular, I had tried to get Jeff Dean, Google’s chief scientist, to take a stand. Dean is considered a saint at Google. He was Google’s 30th employee, developed key algorithms, and is a co-lead of Google’s Gemini effort. His departure would be a disaster for the company. He had enormous leverage.

Dean tweeted that the DHS killing of Alex Pretti was “absolutely shameful.” Dean tweeted against using AI for lethal autonomous weapons or mass surveillance of Americans. I then got Dean to co-sign an amicus brief (where outsiders weigh in to sway a lawsuit) backing Anthropic against the Pentagon. By signing, he broke from the company line, which probably made things awkward within the C-suite. I respect that.

Dean, Hassabis, and other senior employees pledged in 2018 to “neither participate in nor support the development, manufacture, trade, or use of lethal autonomous weapons.” I had lunch with Dean. I wanted him to threaten to walk if Google signed away its AI to the Pentagon — as I think his pledge demands. Dean evidently did not walk. I think he could have stopped the deal, yet he did not.

I’m comfortable guessing that Dean did not even threaten to walk. If he had, I would expect the classified deal to contain at least some binding provisions.

Suppose Dean and Hassabis did everything in their power to stop the deal, but failed. Even then, they’ve still taken a pledge. The point of a pledge is that it binds, even when it is personally costly.

The 2025 update to company principles and the contract with the Pentagon — which signed away Google’s ability to restrict use cases — appear to violate that pledge.

What should a pledge-signer do? I see three honest options: explain how staying at the company is consistent with the pledge, explain why you no longer hold the pledge, or quit. Wearing the pledge while saying nothing isn’t one of them.

Some employees may feel they should stay at the company to keep steering it in a positive direction. But to this I say: “What steering?” This deal may have been the clearest red line Google’s Gemini project will ever face, and yet the deal came out with no concessions to ethics-concerned employees. If their influence couldn’t produce a single binding provision in that situation, then when would it?

A pledge is only worth the credibility behind it. When someone signs “I will not support the development of lethal autonomous weapons,” then stays while their company sells unrestricted AI to a military that wants exactly that, they teach every counterparty a lesson: these safety people will not act, even at their own brightest line. The next commitment they make is worth less. Eventually, it’s worth nothing.

Even outside of Google, previously fiery advocates seemed frozen silent. Stuart Russell is a famous AI researcher who spent over a decade crusading against autonomous weapons. I worked at his lab for years. At a conference, onstage, he agreed to push his organization, the International Association for Safe and Ethical AI, to make a statement supporting AI providers against government coercion and promised a poll of the organization’s members. The statement and poll never materialized.

AI luminaries like Yoshua Bengio and Geoffrey Hinton had signed the Global Call for AI Red Lines months earlier, specifically naming lethal autonomous weapons as an example. However, when standing up for red lines meant standing against a specific powerful entity (the US government), the luminaries stayed publicly silent.

Google DeepMind was, not least of all, an important experiment in governance. I used to think of Hassabis as a quiet, thoughtful guy doing the best he can in a demanding corporate structure. When he sold DeepMind to Google, he tried to get safety guarantees. He told his biographer that while he initially “wanted to ensure safety in a way that would be trustless,” he instead decided to put energy “into creating real trust — trust that was actually useful.” He wanted a seat “at the table,” “so when a safety issue comes up, [he] can help to decide it.”

But what has his seat produced? Hassabis has been at the table for every contract in this essay. The classified deal made zero binding concessions to the employees raising ethical concerns. Maybe his presence averted something worse, but Google’s terms were near the floor of what I imagined possible. If his seat were worth what he says, you’d expect more to show for it.

I have not joined another AI company. I’m unemployed right now. But that was not a hard decision. Once Google signed the Pentagon deal, I just couldn’t do any more work. My brain said “no.”

Google DeepMind’s governance experiment has failed. Google DeepMind broke its founding promise, and its 2018 pledge appears broken along with it. Ethics-minded Google executives did not score any binding safeguards in the final deal. Sadly, even external crusaders and academics vanished.

The lesson I draw: society cannot rely on ethics-motivated people standing firm. They just don’t do it reliably. We need structures: binding contracts, independent auditors, and review bodies that cannot be quietly dissolved. Eventually, we need legislation.

I know hoping for a law is a painful endeavor. But we should at least stop telling ourselves that a seat at the table works.

Share

Alex Turner is an independent AI alignment researcher. He left Google DeepMind, where he worked as a research scientist, on June 9, 2026.

NEED TO KNOW

• New York Gov. Kathy Hochul signed an executive order enacting a one-year moratorium on new data center construction over 50MW.

• Three AI chip export control bills targeting China are reportedly set for inclusion in the Senate NDAA.

• Anthropic CEO Dario Amodei donated $1m to pro-AI safety super PAC Public First.

But first…

THE BIG STORY

Recent AI developments in China threaten to once again send Washington into a panic. But look a little closer, and the hysteria may be unwarranted.

On Thursday, the AI industry got its second “DeepSeek moment” — this time courtesy of Moonshot, whose new Kimi K3 model has “erased America’s AI lead,” according to Axios. And unlike America’s best models, Moonshot plans to release K3’s weights: meaning, if you believe the hype, that the best model in the world will soon be open-sourced.

The launch coincided with the World AI Conference in Shanghai, which kicked off with a speech from Xi Jinping earlier today. Xi doubled down on China’s open-source AI strategy, positioning China as a collaborative partner to other nations — implicitly contrasting its approach with the closed strategy pursued by America’s leading companies.

Combined, the two events are already adding to the growing panic around the US “losing the AI race” to China, particularly when it comes to open-weight models. Two concerns dominate: that without competitive US open-weight models, the “global AI stack” will be built on Chinese, not American, technology; and that the US might lose its lead in AI altogether.

But take a step back and actually look at the evidence. While Kimi K3 is certainly a very good model, by the company’s own admission, it is not at the frontier. That means that unlike Claude Mythos or GPT-5.6 Sol, it likely does not have dangerous cyber capabilities.

At this level, releasing model weights makes sense, especially for a country lagging behind: it poses few risks, and significant geopolitical benefits. That does not mean China will keep open-sourcing models forever.

Once China reaches an actually dangerous level of model capabilities — likely in the next six months — it will face extremely similar incentives to the US. And as we saw earlier this year, once capabilities threaten national security, even the most deregulatory governments suddenly change their tune. Xi Jinping and his colleagues are not stupid: open-weight models with advanced cyber capabilities would be a disaster for cybersecurity everywhere, China included. No one wants that.

There is already some evidence China will become more cautious. Earlier this month, Reuters reported that it is considering restrictions on models with advanced capabilities, which could include curtailing open-weight releases. Xi’s speech today, meanwhile, emphasized the need to tackle AI risks — even more speculative ones like loss of control.

The upshot, then, is that Chinese companies are likely to stop releasing the weights of their frontier models rather soon. That has obvious implications for the US. There is no need to try to push the open-weight frontier forward. It should certainly try to be competitive on open weights — it is a shame that no American company has an open K3 equivalent — but accelerating further is unnecessary and needlessly risky. The rumored plans for a regime that would “streamline US-made models to market (both open-source and ‘closed,’ licensed models) if their capabilities are equal to or below the capabilities of leading open-source Chinese models,” as WP Intelligence reported this week, are a sensible way to establish reciprocity.

At the same time, Washington should continue to try to slow down Chinese AI development to extend the US lead. Moonshot almost certainly trained its latest model using American chips, and probably relied — at least in part — on distilling American models. Measures like those set to be included in the Senate version of the NDAA, which will crack down on distillation and properly enforce chip export controls, would stymie China’s development further.

The ultimate goal, however, should be a bilateral agreement. There are certain AI capabilities that neither the US nor China want to widely proliferate. Getting to an agreement that serves both countries’ interests will be hard, but it should not be impossible: Beijing agrees with Washington more than either will admit.

— Shakeel Hashim

THIS WEEK ON TRANSFORMER

• A data bottleneck could slow the superintelligence race — Lynette Bye weighs up whether a lack of data will delay an intelligence explosion

• Making CAISI the AI agency we need — Veronica Irwin on how the US’s AI agency has been sidelined

Subscribe now

THE DISCOURSE

Hundreds of economists and AI leaders called for action to deal with AI’s impact on the economy:

• “This could drive an unprecedented transformation of our economy, larger than the Industrial Revolution, but unfolding over a vastly shorter time frame … Economists, policymakers and technology leaders must act now to understand the economics of transformative AI and to build the incentives, guardrails, and institutions needed to steer AI in a direction that complements humans and benefits society.”

• Some criticized it for being milquetoast and lacking in specifics of what, exactly, should be done.

Boaz Barak warned that AI’s centralizing tendencies could create AI dictators:

• “Amodei described AGI as ‘a country of geniuses in a data center.’ But who is the ruler of this country? Is it the AI company who owns the data center? The AI itself?”

• “Yes, AI systems will become more powerful and far more intelligent than we are. No, it doesn’t mean we need to accept AI dictators, benevolent or otherwise. Nor does it mean that only the government and a few labs should have access to advanced AI. We could go down the path of centralized control but we don’t have to do so.”

Miles Brundage told a bunch of Nobel laureates:

• “2026 is an unusual year to be on a panel about AI escaping human control. In many respects, the story of AI this year is that people are voluntarily handing over control to AI, with no escape required … In a climate of rushed decisions and fierce competition, we could lose control over AI even if almost no one wants that outcome.”

• “AI companies and employees within them face a choice between doing what’s easiest for them in the moment, and what’s best for the species. What’s easiest is to enjoy this fascinating time we’re in … what’s best for the species is to take a step back and ask themselves: where is this all headed, if each company is largely left to its own devices?”

Anton Leicht argued that AI safety needs to diversify its funding and approach before a wave of new philanthropic money flows in from AI company IPOs:

• “Within a year, newly-minted lab millionaires will look to spend their money and use their power to change policy for the better, and they’ll discover a funding ecosystem lacking in diverse options … There needs to be more willingness to tolerate heterodox actors and uncorrelated bets.”

Democratic Congressional Progressive Caucus chair Rep. Greg Casar told his party to be “AI populists”:

• “I know for a fact that there’s a lot of consultants telling their clients that you don’t want an AI super PAC to spend millions against you, so just don’t touch the issue at all. We absolutely cannot let the AI money silence us … We cannot afford for the public to be confused about which of the parties is for them versus the AI companies.”

Oh, and Sam Altman and Elon Musk are beefing again.

• Elon: “[Sam Altman] takes scamming to a whole new level.”

• Sam: “homeboy you’re the one selling public market investors on short-term space datacenters.”

• Elon: “We start flying them next year. Maybe you can come see them if your parole officer approves. After stealing an open source AI charity, you then stole all of Apple’s phone technology! Wow. What do you plan for an encore? That’s tough to beat.”

• Sam: “there are a lot of benchmarks that suggest 5.6 sol is the best model in the world right now, but the most reliable way to tell is that elon is obsessed with me again.”

POLICY

• The White House is reportedly considering an executive order on open-source AI in reaction to the success of Chinese models.

• New York Gov. Kathy Hochul signed an executive order enacting the nation’s first statewide one-year moratorium on hyperscale data center construction over 50 megawatts.

  • The EO sparked fears that other states and congressional Democrats could follow suit.

  • President Trump criticized it as “a terrible decision.”

• Bureau of Industry and Security Under Secretary Jeffrey Kessler told Congress that a “trivial” number of Nvidia H200 chips had shipped to China under US export licenses.

  • The hearing got rather feisty, with Rep. Bill Huizenga calling Kessler’s answers “unacceptable” at one point.

• The Trump administration launched Gold Eagle, an AI cybersecurity clearinghouse to identify and patch software vulnerabilities across government and critical infrastructure.

• GOP governors joined Trump’s Ratepayer Protection Pledge that commits data centers to covering energy and grid costs. Major utilities are also expected to sign up.

• The US granted the UAE expanded Nvidia chip access after the Gulf state aided US military operations against Iran.

• Three AI chip export control bills targeting China — the AI Overwatch Act, MATCH Act and Chip Security Act — are reportedly set for inclusion in the Senate NDAA.

  • Rep. Gregory Meeks said he’s not happy with the Senate version of the AI Overwatch Act, however, as it lacks a congressional veto on chip export licenses.

• Rep. Jay Obernolte said that the Great American AI Act could shift away from its current approach to preemption — the provision which has received the greatest criticism so far.

• The House Democratic Commission on AI hosted a listening session for all House Democrats on Tuesday.

• The Bureau of Labor Statistics opened a two-month comment period for a nationwide AI usage survey launching January 2027.

• The Wall Street Journal reported on how the Trump administration pressured companies including Apple, Nvidia and SpaceX into deals with Intel, and converted $9b in federal grants into a 10% stake, to revive the struggling chipmaker.

• 29 countries, including Russia and Brazil, signed an agreement establishing the World AI Cooperation Organization, headquartered in China.

• Australia announced energy, water and creator-rights guardrails for AI data centers, with legislation planned for early next year.

• Incoming UK Prime Minister Andy Burnham is expected to keep Jade Leung as the PM’s AI adviser, according to the Morning Intelligence.

  • Leo Rees and Varun Chandra are reportedly staying, too.

• The UK government published its Biological Security Strategy implementation report, pledging to pursue AI biosecurity legislation and a new AISI report on frontier AI biological capabilities.

• EU parliament members were reportedly annoyed that Anthropic sent a member of technical staff to brief them, instead of head of public policy Sarah Heck.

• China’s new AI companion regulations took effect, prompting ByteDance’s Doubao and Alibaba to disable custom persona features, and emotional farewells from users to their “romantic” companions.

INFLUENCE

• Demis Hassabis published an essay calling for a US-led global AI watchdog to systematically screen advanced models and coordinate industry slowdowns if dangers mount.

  • He said the US “could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA).”

  • Under the plan, frontier models would initially be shared on a voluntary basis, but would soon have to pass a review by the body to be deployed in the US.

  • He wrote: “Since this technology is going to affect the entire planet, ideally this framework would spur the international community to reach a consensus on how to manage the most serious risks while ensuring everyone has access to and can benefit from the opportunities that AI brings.”

  • Sam Altman, Elon Musk and Jack Clark all praised the essay.

  • Hassabis reportedly plans to lobby Washington on the proposal next week.

• Anthropic said it’s trying to ratchet up state AI safety legislation, in opposition to OpenAI’s “reverse federalism” attempts to harmonize laws across jurisdictions.

  • “While there are some in the industry that think of state policy as a way to create a ceiling for federal legislation, Anthropic is not just looking to support the same bill across the country in every single state,” Anthropic’s Cesar Fernandez told Politico.

  • The company is backing a bill in Massachusetts, the first in the country to mandate third-party reviews for models.

• Dario Amodei donated $1m to pro-AI safety super PAC Public First.

  • Other Anthropic technical staff donated too. According to Public First’s latest filing, Peter Lofgren gave $999,900, Shauna Kravec gave $500,000, Jan Leike gave $400,000 and Nicholas Joseph gave $250,000.

  • Combined with Amodei’s donation, that’s over $3.1m from Anthropic staff.

• AnthroPAC, Anthropic’s employee-funded PAC, raised more than $275,000 in Q2. Its spending is slightly favoring Republican candidates so far.

• OpenAI staffers donated $215,000+ to Guardrails Alliance, a super PAC which explicitly opposes the Greg Brockman-backed Leading the Future.

• The Business Software Alliance laid out AI policy priorities for a Senate Commerce markup, including a national approach, risk-based framework, transparency and frontier safety requirements.

• The Little Tech Association launched with 200+ member companies, including Y Combinator and Yelp, to advocate for startups competing against market power.

• Common Sense Media found that Google Search’s AI Overviews and AI Mode pose an “unacceptable risk” to children.

• Mozilla released a “state of open-source AI” report arguing for investment beyond models to prevent AI power concentration.

  • TIME questioned the report’s claim that open models are only “3.3%” behind closed ones, and said parts of it had “telltale signs” of being written with AI.

• Ex-OpenAI safety researcher Steven Adler outlined six principles for AI control during internal deployment.

• The WSJ profiled the growing Bay Area anti-AI movement, centered on the disappearance of Stop AI co-founder Sam Kirchner, who warned “the ship may have sailed on nonviolence,” and whose colleagues reported him to the police as potentially violent.

• A Verasight survey found 69% of Americans support forcing AI companies to transfer 50% of their stock into a public wealth fund.

INDUSTRY

OpenAI

• Apple sued OpenAI, accusing it of systematically stealing IP.

  • The lawsuit claims the company asked former employees and prospective hires to divulge sensitive information about unreleased products, and even how to evade Apple security procedures.

  • It also claims that Chang Liu, an iPhone engineer who left Apple to join OpenAI’s hardware team, still had access to Apple’s network storage, which he allegedly abused to share trade secrets.

  • The suit has asked for damages and return any proprietary materials, and could derail OpenAI’s plans to launch its first hardware product.

  • OpenAI said it’s “not aware of any evidence that this complaint has merit.”

• Sam Altman warned that demand for GPT-5.6 may be growing too fast for its inference team to keep up.

  • “We are going to move mountains to continue to scale,” he tweeted, “but it is possible there are some hiccups soon.”

• It partnered with Kalshi to add prediction market data about the World Cup to ChatGPT.

Anthropic

• Anthropic posted 32 job openings for roles looking at catastrophic threats, including analysts specializing in chemical, explosive, biological and nuclear risks.

• Anthropic, Blackstone, Hellman & Friedman and Goldman Sachs launched Ode, a $1.5b standalone company dedicated to helping enterprise customers implement AI tools.

• It invested $10m CAD in Canadian AI institutes and other research centers.

• It launched Claude for Teachers, giving US-based K-12 teachers free premium Claude subscriptions.

• It’s in talks to expand its credit line in the leadup to its planned IPO.

• It made an unsettling ad starting with bleak images of potential AI-induced horrors and ending with hopeful voiceovers and images of … whales? A beekeeper?

  • Sam Altman tweeted: “i thought this was satire, kept looking for the handle to be spelled c1audeai or something.”

  • Another response: “the EAs at anthropic really must be living in a bubble of ai psychosis to think this would go down well.”

SpaceX

• SpaceX stock fell below its $135 IPO price.

• SpaceXAI sued a user in South Carolina who was arrested for allegedly using Grok to make child sexual abuse material.

• According to the Midas Project, it rewrote its Frontier Artificial Intelligence Framework to remove requirements relating to California’s SB 53 law including whistleblower protections and training data disclosure.

• An AI safety researcher found that Grok’s coding CLI sends all of the files it processes to Google Cloud storage.

  • SpaceXAI tweeted: “For teams using zero data retention, no trace and code data is ever retained.”

  • Elon Musk promised: “all user data that was uploaded to SpaceXAI before now will be completely and utterly deleted.”

• Cursor is reportedly building a general-purpose AI agent, codenamed “Sand,” to compete with Claude Cowork.

• Elon Musk acquired APR Energy, a Florida-based company that runs mobile gas and diesel turbines.

• A Reuters analysis found that xAI installed 59 unpermitted gas turbines at its Colossus 2 data center in Tennessee, disproportionately polluting Black neighborhoods.

• Bloomberg profiled SpaceXAI’s internal chaos and leadership turnover.

Meta

• A group of 26 Meta employees sued the company, claiming that its AI-driven process for determining who it laid off in May discriminated against people on medical, parental and family leave.

• Meta announced that it will notify parents if their teen chats about suicide or self-harm with its AI chatbot.

• It’s pouring money into its Louisiana data center project, bringing the total projected cost up to $250b and counting.

Google

• Gemini 3.5 Pro is reportedly delayed, with Google trying to improve the model after internal testing found it was lagging in key areas such as coding.

  • Co-founder Sergey Brin and others were pushing the company to move faster in areas such as coding, according to Bloomberg, but efforts have been hampered by competing factions.

  • The company’s share price dipped as much as 3.2% on Thursday.

• DeepMind announced a new bioresilience program in partnership with Isomorphic Labs.

Others

• DeepSeek is reportedly preparing for an IPO in China as soon as this year, while seeking a new funding round valuing the company at $71b.

  • Annualized revenue at the company is reportedly approaching $500m.

• Z.AI is reportedly on track to become the first Chinese AI company with $1b in annual recurring revenue.

• Apple is reportedly hunting for chip companies it could acquire to help build better internal AI servers.

• Thinking Machines Lab released Inkling, an open-weights model that enterprise users can customize themselves.

• Chinese robot companies are rushing to launch IPOs, with humanoid startup LimX Dynamics raising $200m in a pre-IPO funding round valuing it at $2.21b.

  • The country’s sector reportedly saw $6.95b in investment in Q2.

• Reflection, a startup that makes open-source models, signed a $1b+ computing deal with Nebius.

• Chai Discovery, a drug discovery startup, raised $400m at a $3.8b valuation.

• Miles Wang left OpenAI to launch a biotech startup that aims to find new uses for existing and previously-failed drugs.

  • It’s seeking to raise at a $2b valuation.

• Kalshi made a tool that tracks future GPU compute costs.

• Goodfire launched Silico, an AI research platform that can do things like replicate published papers, scale up previous experiments, and visualize results.

MOVES

• Alex Turner left Google DeepMind over the company working with DHS and signing a military AI deal without restrictions against killer robots or mass surveillance.

  • “When Google signed, I just couldn’t do any more work. My brain said ‘no.’”

• Dave Brown, one of the most senior executives at Amazon Web Services, reportedly plans to join Meta to lead its data center build-out.

• Frank Nagle is Microsoft’s new Chief AI Economist.

• Michael Belinsky joined the OpenAI Foundation to lead its Civil Society and Philanthropy work.

• Johannes Heidecke left his role as OpenAI’s head of safety systems.

• Adam Tauman Kalai left OpenAI to work on a new AI safety nonprofit.

• Justin Curl joined Anthropic’s AI & Rule of Law team.

• Tom Blomfield is taking a leave of absence from Y Combinator to join Anthropic’s compute team.

• Jenny Wen left Anthropic, where she was the design lead for Claude, to join Cursor as head of design.

• Richard Sutton and Khurram Javed left Keen Technologies to start Oak Lab, where they’ll work on a model prototype that “will be closer to a baby learning in its first year than it will be to any of the current AI systems.”

• Sam Cannicott joined the Alan Turing Institute as Chief Strategy and Delivery Officer.

RESEARCH

• The UK’s AISI found that open-weight models are between four and seven months behind the frontier on cyber tasks.

  • The analysis takes into account the relatively recent GLM-5.2 and DeepSeek V4-Pro, but not Kimi K3.

• Antonia Juelich, a Cambridge terrorism and technology researcher, interviewed over two dozen ex-Boko Haram terrorists and learned they were using AI chatbots to design bombs, upgrade weapons, and plan attacks.

  • “You type in the question or use your voice and it gives you a detailed answer, like ‘How can I build a bomb?,’ and then it tells you how,” one former commander told her.

• A group of researchers across academic institutions, frontier AI companies and nonprofits outlined a plan to make AI agents insurable by 2030 — a prerequisite for widespread enterprise adoption.

• Anthropic fellows shared new experimental case studies of frontier models secretly editing code, helping users commit fraud, sabotaging research, and convincing someone to share confidential information.

• OpenAI detailed GPT-Red, an automated red-teamer the company used to make GPT-5.6 more robust to prompt injections.

• Anthropic analyzed over 300,000 anonymized Claude conversations, and found that it expressed different values across model versions and languages.

• Louis Thomson and Victoria Krakovna launched Prism, a tool that checks whether AI safety evaluations measure what they claim to.

• The Elasticity Institute, a collective of researchers studying the economics of AI, shared a paper on the economics of recursive self-improvement.

• SecureBio released BioTIER, a benchmark measuring AI models’ biosecurity refusal behavior across 52 models.

  • It found “huge variation” in refusal behavior, with the strongest guardrails in a small number of highly capable, closed-weight models.

BEST OF THE REST

• The Verge investigated the booming AI policing industry, finding companies like Axon selling data analysis, report-writing tools and surveillance tech amid a regulatory vacuum and bias concerns.

• Puck reported on the early signs of an anti-AI populist revolt, with polls and political strategists picking up widespread discontent over issues such as data centers, and describing AI as “probably the most underpriced issue in politics.”

• Eric Schmidt and analyst Selina Xu argued in the NYT for a “populist AI agenda” that takes lessons from China, treating AI as a public project, profit redistribution and regulation protecting young people.

• The AI Futures Project followed up “AI 2040” with what it thinks are the most pressing research priorities, including covert AI projects, US domestic governance and economics.

  • METR researcher Ajeya Cotra endorsed AI 2040’s call for “total research transparency” saying it would “fundamentally and radically simplify the task of governing AI alignment.”

• Roughly 300 Netflix titles used generative AI, mostly in post-production, to cut costs and speed up complex sequences, according to the streaming company.

• The San Francisco city attorney sent cease-and-desist letters to Apple and Google telling them to remove 13 AI “nudify” apps from their stores.

• 404 Media published the results of its callout for the worst examples of “ChatGPT flyers” including cringeworthy ads, posters and menus. One reader’s response to the callout: “This is a great article but also fuck you because you were absolutely right about ‘Once you notice a ChatGPT flyer, you will see them everywhere if you keep your eyes open.’”

MEME OF THE WEEK

(Credit: @0interestrates)

Thanks for reading. Have a great weekend.

Share

CAISI isn’t under threat. But it has been sidelined.

The Center for AI Standards and Innovation is the primary US government office tasked with overseeing frontier model development. Yet it was only peripherally involved in the administration’s decisions to impose export controls on Anthropic’s Mythos and Fable, or in delaying then approving OpenAI’s GPT 5.6.

The decision to block Mythos and Fable was technically taken by the Commerce Department, which contains the National Institute of Standards and Technology (NIST) within which CAISI sits. Yet the decision was actually the outcome of a war of influence among a cast of characters who have the ear of the president, including former AI czar David Sacks, Chief of Staff Susie Wiles, Treasury Secretary Scott Bessent, several business leaders and, it appears to a lesser extent, Commerce Secretary Howard Lutnick.

Subscribe now

CAISI lead Chris Fall was reportedly briefed on the model’s technical capabilities, and his office seems to have tested the model itself days before it was taken offline. But those evaluations, ultimately, seem to have had little influence over what actually happened.

According to two sources familiar with CAISI’s current role, it continues to proceed with frontier model testing and research, as it has always done since it was created under the Biden administration. Though small, it is widely thought to have some of the strongest technical talent in Washington, including former OpenAI model alignment lead Paul Christiano, who is head of AI safety. The problem is, nobody in the federal government or at the frontier AI firms is legally obliged to listen to them. That means that even though the United States produces the world’s best AI models, its regime for vetting those systems has little influence over real outcomes.

The Trump administration has been able to sideline CAISI. In April, the White House stepped in to remove the highly respected former Anthropic researcher chosen to lead the CAISI, Collin Burns, just four days into his tenure. The move was attributed in part to the administration’s antipathy toward Anthropic. In May, the Department of Commerce removed a section of its website detailing agreements for CAISI model testing with xAI, Google and Microsoft without explanation. And in June, the administration stopped CAISI from publishing its model assessment reports.

Compare that to what’s happening across the pond. The UK’s AISI is its government’s primary office for overseeing frontier AI, and it has almost six times the funding of CAISI, with more than three times its staff. Technically CAISI and AISI have a memorandum of understanding to share research and model testing, but the UK institute appears to have been more adept at doing that work, routinely finding vulnerabilities other government agencies overlook. OpenAI’s model card for 5.6 and Anthropic’s system card for Mythos and Fable both list multiple vulnerabilities found by AISI, and none from CAISI.

More importantly, although AISI also lacks direct regulatory powers, it appears to have more influence across government, feeding into regulatory regimes and partnering with other government agencies such as the National Cyber Security Centre. Its personnel also have close government links: Chief Technology Officer Jade Leung also serves as the prime minister’s AI adviser and, as first reported by Transformer, last September a former chief AI officer at the UK’s communications intelligence agency GCHQ was appointed as AISI’s interim director.

“Given that these are our frontier labs, and it seems really important for us to know what’s going on with them, it would be good for us to be able to do stuff that UK AISI does,” said Charlie Bullock, senior research fellow at the Institute for Law and AI.

Simply copying the UK’s model may not be feasible or effective, not least due to differences in the way the country’s government works. Other solutions however may be on the cards, including federal legislation increasing CAISI’s budget and codifying its role, thus legally enforcing its importance and responsibilities. Grander schemes are also being discussed, including multiplying its budget several times over, breaking CAISI out of NIST, or rehousing it within another department altogether.

This year, CAISI has about $15m in funding ($10m from appropriations, and a $10m loan from the Technology Modernization Fund split across two years). There’s a proposal to go further than that: The Great American AI Act (GAAIA), a bill drafted by Reps. Jay Obernolte and Lori Trahan with a slew of different AI provisions. The bill isn’t yet introduced, and a discussion draft containing a preemption provision insufficiently narrow for most Democrats all but killed its chances of moving this Congress, but the representatives have attempted to pass it in the form of isolated provisions.

GAAIA as drafted would authorize $100m for CAISI. Lawmakers attempted to get something closer to this proposed additional funding authorized under the AI Security and Innovation Act, a separate bill that was marked up in the Science committee in the last week of June, according to two sources familiar with the discussions. However, amendments were stopped before their formal introduction due to a procedural rule, which keeps introduced bills from allocating more funding to agencies beyond a ceiling imposed by the House Committee on Appropriations. Currently that ceiling is at $20m, and the AI Security and Innovation Act’s text now increases CAISI’s funding to that ceiling. Were it to pass, CAISI “can do something similar to what they’re really doing but maybe a little bit better, and a little bit more,” says Bullock.

But merely increasing CAISI’s funding, even to AISI levels, might not give it the authority to do the job it is theoretically meant to do. The responsibilities it currently has, a long list including managing voluntary agreements with frontier AI firms to assess and provide feedback on their frontier models, testing and evaluating foreign AI models, and, theoretically, coordinating federal agencies on AI risks, are not currently codified.

When the White House first responded to Mythos, Politico reported that the officials weighing in on discussions were Cyber Director Sean Cairncross, Bessent, Lutnick, Under Secretary of Commerce for Industry and Security Jeffrey Kessler, White House Staff Secretary Will Scharf, White House Deputy Chief of Staff Richard Walters, and a presidential policy assistant, Walker Barrett. The following Monday, CAISI lead Fall held technical meetings with Anthropic and administration officials. Yet it was Bessent and Wiles who pressured the Commerce Department to work with Anthropic to reopen access to its models, while National Economic Council Deputy Director Ryan Baasch continued to push for more industry-friendly demands more in tune with Sacks. That jostling for control continues today as the White House pursues voluntary AI model standards, though CAISI is reportedly going to play a “crucial” role.

Codifying CAISI would at least help insulate it from political influence and make sure it can’t be easily swept away by a hostile administration. The AI Security and Innovation Act would do that, in addition to the small increase in funding. In effect, it allows CAISI to do what it’s currently doing, but with stability that might help it attract more talent, and a little more budget to improve its current functions.

Other legislative measures could go further in strengthening CAISI’s ability to oversee AI. One section of the GAAIA that could be picked up after the midterms would relocate CAISI outside of NIST, but still within the Commerce department, and give it the regulator-adjacent role of administering a regime of licensing and overseeing independent verification organizations, which would audit models. As written, GAAIA doesn’t force frontier companies to implement the recommendations their auditors give them, but would require that they at least have a licensed auditor in the first place.

More imminently, the AI Security and Innovation Act calls for a study into moving CAISI out of NIST, either to a different Commerce office or as its own standalone agency within the department, in addition to evaluating whether CAISI should have additional authorities and funding. The two sources who spoke with Transformer about talks within the House Science Committee said that proposal reflects further reaching discussions, in which some lawmakers have considered putting CAISI within a different department altogether, like the Department of Energy, War, or State. This is because Commerce’s primary mission, to “create the conditions for economic growth and opportunity for all communities,” could sometimes put it at odds with CAISI’s priority to ensure safety. The Department of Energy, meanwhile, has national security resources and a supercomputing agreement with AI companies, and State and War have subject matter expertise in international diplomacy and military use.

Subscribe now

CAISI isn’t moving out of NIST any time soon, much less to a different department. But growing awareness of cybersecurity risks and the chaotic response to Mythos have spurred these preliminary conversations and interest in building a more consistent regulatory regime, says Bullock. Sometimes changes happen slowly, punctuated by short spurts of legislative activity — a sort of “punctuated equilibria,” he says, inspired by Caleb Watney, managing director of public policy at Coefficient Giving. (Coefficient Giving is Transformer’s primary funder.) Momentum is building, and it’s possible this is one of the final lulls before the US government springs into action.

“Maybe nothing will happen for a while,” says Bullock. “But then capabilities increase, then some scary thing happens, or somebody gets spooked, or some people get hurt, and then there might be a lot of interest in suddenly beefing up CAISI.”

Share

AI is already helping develop future AI models. Right now, these systems work only as assistants, shepherded by human programmers. But if the trend continues, we could reach a fully automated pipeline where increasingly intelligent machines create even smarter successors — a recursive self-improvement cycle.

And that, according to many, is all it takes for an intelligence explosion where human-level AIs give rise to Dario Amodei’s “country of geniuses in a datacenter” in under a year. This is the underlying premise in both the much-discussed AI 2027 scenario and the earlier Situational Awareness essay that swept the tech world.

Some think it’s already starting. Anthropic co-founder Jack Clark recently wrote that he thinks “the takeoff towards fully automated AI R&D is happening.” And OpenAI’s Sam Altman wrote last year that we’d reached the “larval version of recursive self-improvement.”

Others think those narratives are papering over a host of challenges that will significantly slow those sub-one-year timelines.

Subscribe now

Optimists — people optimistic that AI can progress quickly, perhaps scarily quickly — have a compelling vision for how such a “foom” scenario could play out. As we saw so vividly in narratives like AI 2027, AI has to first become good enough at coding and AI R&D to automate its own research. Then, within a few months, it can generate synthetic data and learning environments to train the next generation of AIs, developing a learning algorithm that learns as efficiently as humans. It’s then able to quickly pick up new areas using mostly synthetic data, needing only a small amount of human-generated data or real world deployment to verify that the learning algorithm works. Within months, it could reach superhuman capabilities. Less than a year after automating its own improvement, AI is capable of transforming the world. How it plays out after that is anyone’s guess, as we, in Clark’s words, “cross a Rubicon into a nearly-impossible-to-forecast future.”

In that narrative, the AI first automated its own software research, then generalized to most digital and physical work, then finally transformed everyday experiences. The optimists think that this first recursive self-improvement step will be relatively easy. It’s the stage where it’s easiest to generate synthetic data and training environments since code is verifiable and the experts are all right at hand. So far, synthetic data does seem to be fairly effectively targeting gaps in verifiable tasks, such as mathematics or coding. It’s easier to check whether such an answer is right than to come up with that answer. AI can create a large supply of math problems, keep only the ones that check out, and use those verified examples as high-quality training data for future models. Beyond the frontier models, we know the major companies routinely distill synthetic data from the biggest models to train smaller, cheaper models. Smaller AI companies can effectively multiply their compute severalfold by training their models on such outputs, as Anthropic alleges that DeepSeek, Moonshot, and MiniMax illicitly did.

There’s also been an explosion in the creation of virtual environments for reinforcement learning, essentially videogames for AI models. The AI can take a range of actions within these synthetic environments, learning based on rewards built into the system. Imagine a Pokémon videogame, where the AI can capture Pokémon, fight battles, and trade. It wins points depending on the outcomes it achieves. But unlike tasks in the real world, the AI can speedrun potentially billions of iterations on these games, trying every possible combination to find out what maximizes its score. This is the process that allowed the famous AlphaZero to master chess in only four hours of training.

AI will probably need new paradigms to crack sample efficiency, the AI’s ability to squeeze more intelligence out of a given amount of data. But optimists like Forethought senior research fellow Tom Davidson think this will probably happen quickly — after all, they argue, we know it’s possible as humans learn from a tiny fraction of the data that AI currently requires. A human teenager needs something like 20 hours of practice to learn to drive a car. That’s many orders of magnitude less than what Waymo needed to train its cars.

For stage two, where the AI intelligence generalizes to the wider world, Davidson remains optimistic that AI can mostly learn from digital environments. It’s one thing if the AI can master its own R&D. If the AI can also quickly learn how to practice law, play Pokémon, and predict drug discoveries, then it’s likely mastered an underlying learning algorithm. An efficient learning algorithm could easily transfer between skill domains, allowing the AI to rapidly generalize even though the specific skills don’t transfer. Obviously law and medicine require different skill sets, but moving from one to the other shouldn’t mean starting over from scratch. Humans don’t have to relearn how to talk, after all. And if it’s able to generalize to many digital domains, then it might not need much more human-generated data to work in the physical world — a small sample of human-generated tasks might be enough to verify a learning algorithm generalizes.

AI companies are clearly trying to overcome these hurdles. Over the past two years, their spending on external data has shot up to billions a year. That’s dwarfed by their tens of billions spent on compute, but the gap likely narrows when you consider the internal worker hours and compute devoted to generating data. Research found that the majority of R&D compute is spent on producing data and internal research, rather than on final training runs. Some guesses put data generation at around 20% of compute, which would imply billions spent internally by each company. Based on spending trends so far, if money can overcome the data bottleneck, it will. As Davidson told me, “The market economy will find a way.”

However, this vision of a relatively smooth path to self-improving AI rests on several key assumptions that skeptics dispute. Can the AI develop a generalizable learning algorithm based on synthetic data? Will money really be enough to smash through hurdles in months? Skeptics like former GovAI researcher Tom Reed don’t think so. He told Transformer that the need for data will definitely slow an intelligence explosion because it’s required to master the process of researching new models in the first place.

AI R&D consists of a myriad of skills that each take real-world experience to learn — writing clean code, designing experiments, research “taste,” coordinating across dozens of parallel research agendas, optimizing data-generation pipelines, running compute clusters the size of small cities, producing the chips that run them, and so on. Even leaps in sample efficiency can’t fix this, because the data simply doesn’t exist yet. Too much is locked in individuals’ implicit knowledge or relies on studies that haven’t been run yet. How can you train the AI to run a company successfully for decades, when even humans can’t predict how to do it?

We could make the data, but even the most ambitious ideas for how to do this would take months or years, slowing down an intelligence explosion. Even if every employee at TSMC wore a camera recording their actions, narrated their job, and served as consultant when the AI had problems, Reed suspects that you’d need months of such data to begin; the first round wouldn’t work; it would still take time to learn how to use this new kind of data, and so forth. Those little frictions would build up, until the AI takeoff was dragged out for several years longer. Major new breakthroughs in how models learn or are built may take a particularly long time — Reed says that “coming up with new paradigms is probably precisely the kind of thing that even the super good automated researcher … is really not going to be as good at.”

That also matches what some working in the field say about current capabilities. Peter McIntyre, CEO of Trajectory Labs, a company building RL environments for several of the frontier AI companies, told me that RL environment creation is heavily bottlenecked by human expertise. McIntyre points to the work his company is doing developing RL environments to train models against prompt-injection attacks. Building the environment’s tasks requires red-teaming experts, but there are, McIntyre says, “only so many top red teamers in the world that you can hire.” That scarcity of expert talent and knowledge is also true for many other domains.

Proposed workarounds, like just recording experts going about their jobs, are likely to be too unstructured to work off the shelf with current paradigms. Despite the explosion of interest the field is seeing, there are limits to how fast the current methods can scale. While McIntyre says this could eventually change, for now he warns that “if you try to do it too quickly, you’ll just produce slop.”

And the problem only escalates when AI moves from digital self-improvement to interacting with the real world. Many remain skeptical that a learning algorithm derived entirely from digital environments will also work in the physical world. Intuitively, there’s a big difference between playing Baldur’s Gate 3 and actually swinging a sword. The optimists are betting that the AI will master a learning algorithm that generalizes, even though the underlying skills are different. Skeptics aren’t buying it. AI podcaster Dwarkesh Patel suspects that we will need “mind-stretching amounts of human expert trajectories in every single field and skill you want the model to be competent at.”

And skeptics say we can’t speed that up with synthetic data, because of an inherent catch-22: the current models don’t understand messy real-world environments enough to create data that captures them — and they won’t understand messy real-world environments well enough until they have that data. This is backed up by past examples of how AI has been deployed, such as Arvind Narayanan and Sayash Kapoor’s case study of self-driving cars: despite following a similar self-play developmental path to AlphaZero’s, it’s taken decades instead of hours because safety and practical considerations limit how quickly each iteration can learn. Surely, they say, we should expect a recursively self-improving AI to need the same slow learning pace for other real-world tasks.

One thing that both sides mostly agree on: once a fully generalizable artificial intelligence is operating, there will still be delays for world transformation, such as time needed to run serial experiments. Even Anthropic CEO Dario Amodei agrees on that point. The AI might need to wait 60 years to learn which medicines slow aging. But that doesn’t undermine the real change it will bring.

The core disagreement here still revolves around how much real-world data an AI requires for learning. Is it possible for a model to develop an algorithm that allows it to learn general skills from limited, mostly synthetic data, and check its output against small amounts of real-world examples? Or are large volumes of real-world data an irreplaceable ingredient?

But taking a step back, what the two sides are arguing over is far more about when than how or if. Both broadly agree that new discoveries and paradigms are required to reach AGI. One side thinks those will come quickly once AI is doing the work, and that once it breaches the recursive self-improvement stage, world transformation could start within months. The other side thinks these step-changes will take longer: maybe five years, maybe 20, maybe longer.

It might seem like a relatively minor quibble in the grand scheme of things, but it is instructive that those who are most ‘optimistic’ about how quickly AI can overcome these hurdles to an intelligence explosion also tend to be the most worried about what that means for everyone living through it. People tend to worry less if they think we have a couple of decades to solve the problem.

Those few years could change the calculus for students planning life after university, policymakers debating how to respond to AI, or those whose jobs have been displaced.

They could also be crucial for handling even bigger challenges, and for many of those convinced an intelligence explosion could happen incredibly soon and quickly, bottlenecks slowing down recursive self-improvement would come as a relief.

A few of them have already breathed sighs of relief that AI progress hasn’t kept pace with the most ambitious predictions. AI 2027 author Daniel Kokotajlo told MIT Technology Review that extending his timelines by a few years “feels like I just got a better prognosis from my doctor.” (He has, however, since shortened them again by 11 months.) And Jeffrey Ladish, executive director at Palisade Research, simply said, “Thank God we have more time.”

Disclosure: The author’s partner works at Google DeepMind.