Google breaks its promises

Transformer Weekly: Gemini 2.5 Pro, SB 813, and BIS budget cuts

Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.

One bit of housekeeping first: Transformer will be off for the next few weeks while I take some time off. We’ll be back in late April.

Top stories

• You’ve almost got to feel bad for the folks at Google DeepMind. The release of ChatGPT’s new image mode completely stole the thunder from Gemini 2.5 Pro, a new model which is, by many accounts and benchmarks, now the world’s best AI model.

  • But you shouldn’t feel too bad. Because the release of Gemini 2.5 Pro is one of the most irresponsible in recent memory — and breaks promises Google made to the US government.

  • While Google spent plenty of time touting the new model’s capabilities, it spent absolutely no time talking about its safety. The word isn’t mentioned even once in Google’s announcement.

  • Unlike Anthropic, OpenAI, and even Meta, Google did not release a system card accompanying the new model. Such cards generally include detailed information on safety testing procedures and results, and have in recent months revealed some worrying evidence of dangerous model capabilities.

  • That’s despite Google previously committing to release such information. In the July 2023 White House Commitments, Google promised to “publish reports for all new significant model public releases within scope”, with scope defined as “generative models that are overall more powerful than the current industry frontier”.

    • Those reports, Google promised, “should include the safety evaluations conducted (including in areas such as dangerous capabilities, to the extent that these are responsible to publicly disclose), significant limitations in performance that have implications for the domains of appropriate use, discussion of the model’s effects on societal risks such as fairness and bias, and the results of adversarial testing conducted to evaluate the model’s fitness for deployment.”

    • In Seoul last year, the company made similar commitments, pledging to “publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use” and “provide public transparency” on its risk assessment processes and results.

  • Based on Google’s own announcement blog, which describes Gemini 2.5 Pro as “state-of-the-art on a wide range of benchmarks”, the model is clearly in scope. But the company has not published a report of any kind, let alone one that includes detailed information on safety evaluations.

  • Reneging on voluntary commitments is bad in and of itself. But there’s a bigger problem here than just breaking a promise.

    • In the absence of a model card, we have no idea if Google actually did any safety evaluations, or if the model is safe to release. In a self-regulation regime, transparency is the only way to enforce best practices. Failing to provide it means we have no idea if the company’s behaving sensibly at all — or how much risk society is taking on by having this model out in the wild.

    • These aren’t just abstract concerns. OpenAI and Anthropic have recently warned that they think their models are on the cusp of being able to help bad actors build bioweapons. Gemini 2.5 Pro is supposedly better than those companies’ models — so does it also increase biorisk? We just don’t know!

    • And there’s particular reason to worry about this when it comes to Google. A fantastic piece in Wired this week exposed that the company has a track record of rushing over product testing in order to ship products faster, reporting that “several reviewers quit, feeling their concerns with various launches weren't fully addressed”.

      • (The article also notes that the company’s trust and safety teams added new roles following these complaints.)

  • In fairness, it’s not just Google behaving badly here. OpenAI also failed to release a system card for Deep Research when it was first made available (though they did provide me with some information about their safety testing when I asked). But this is the most significant model release I can think of that was released without any safety information.

  • This should be a wake-up call for policymakers. Companies are reneging on promises they’ve made to governments, and may not be carrying out basic safety testing procedures on whether their models are increasing national-security relevant risks. It’s increasingly clear that the only way to get companies to behave responsibly is to actually regulate them. The question is whether governments will step up and make that happen.

  • (I reached out to Google DeepMind for comment on all of the above; at the time of publication I had not received a response.)

• California State Senator Jerry McNerney introduced SB 813, a bill that I expect you’ll hear a lot about over the coming months.

  • It would establish “independent, third-party panels of AI experts and academics to devise strong yet workable safety standards”.

  • Those panels would be accredited by the AG, and then “certify and monitor AI developers and vendors who meet the standards”. The standards would be voluntary, but companies which “choose to meet the MRO safety standards and receive MRO certification would be eligible for certain legal protections under state law”.

  • The bill has already received endorsements from Gillian Hadfield and Dean Ball (it closely mirrors the “private governance” approach Ball recently wrote about).

  • Meanwhile, Scott Wiener’s SB 53 bill unanimously passed out of committee. Politico has a good run down of a couple other AI-relevant bills in California here and here.

The discourse

• Alibaba chairman Joe Tsai said he’s worried about a data center bubble:

  • “I start to see the beginning of some kind of bubble … I start to get worried when people are building data centers on spec … People are talking, literally talking, about $500 billion, several 100 billion dollars. I don’t think that’s entirely necessary.”

• Wired published a long-read on Anthropic, with extensive quotes from Dario Amodei. Not much new for Transformer subscribers, but this bit stood out re: the OpenAI split:

  • “A number of employees began to worry about where the company was headed. Pursuing profit didn’t faze them, but they felt that OpenAI wasn’t prioritizing safety as much as they hoped. Among them—no surprise—was [Dario] Amodei. ‘One of the sources of my dismay,’ he says, ‘was that as these issues were getting more serious, the company started moving in the opposite direction.’ He took his concerns to Altman, who he says would listen carefully and agree. Then nothing would change, Amodei says … As one member of the group put it, they began asking themselves whether they were indeed working for the good guys.”

  • Also notable re: the decision to let OpenAI start the AI race with ChatGPT: “‘It was costly to us,’ Amodei admits. He sees that corporate hesitation as a ‘one-off.’ ‘In that one instance, we probably did the right thing. But that is not sustainable.’ If its competitors release more capable models while Anthropic sits around, he says, ‘We’re just going to lose and stop existing as a company.’”

• Alex Chalmers wrote an excoriating takedown of the Alan Turing Institute — and why it’s been such a failure:

  • “The ATI’s decision essentially to ignore much of the cutting edge work coming out of DeepMind and US labs meant that its leadership was asleep at the wheel as the generative AI boom got underway.”

• Thomas Friedman didn’t mince words in his latest NYT piece about US-China cooperation on AI:

  • “There is an earthshaking event coming — the birth of artificial general intelligence … Whatever you both may think you’ll be judged on by history, I assure you that whether you collaborate to create a global architecture of trust and governance over these emerging superintelligent computers, so humanity gets the best out of them and cushions their worst, will be at the top.”

Policy

• The Trump administration blocked $20m in funding for the Commerce Department’s Bureau of Industry and Security, which famously already struggles to enforce export controls due to its limited budget.

  • In hilarious timing, BIS also imposed export controls on a bunch of new entities (including Nettrix, which has reportedly helped companies evade chip controls, and the Beijing Academy of Artificial Intelligence).

• Virginia Governor Glenn Youngkin vetoed the state’s AI bill.

• NIST announced a “zero drafts” pilot project to “accelerate the creation of [AI] standards”. It also released new guidance on adversarial machine learning, co-authored with the UK AI Security Institute.

• Reps. Obernolte and Beyer introduced a revised CREATE AI Act to codify the National AI Research Resource.

• Sens. Rounds and Heinrich reintroduced a bill to “leverage AI for national pandemic preparedness”.

• Sen. Cruz said he wants the federal government to tackle “specific, concrete” AI problems rather than creating a “comprehensive regulatory regime”.

• The Intelligence Community’s Annual Threat Assessment listed China’s AI ambitions as a threat.

• The Senate Armed Services cyber subcommittee discussed getting the DOD to accelerate AI use to “outpace our adversaries in the cyber domain”.

• North Korea reportedly set up a new unit to do AI-enhanced hacking.

• Germany’s new government might seek a revision of the EU AI Act, MLex reported.

• A group of EU lawmakers strongly criticized the new draft of the GPAI Code of Practice, seemingly taking issue with the new draft’s sharpened focus on catastrophic risks.

• Other EU lawmakers called for a second EU Chips Act with a focus on AI.

• Chi Onwurah, chair of the UK House of Commons technology committee, urged the government to bring forward its AI safety bill soon.

• Taiwan is reportedly investigating SMIC for allegedly poaching engineers to access advanced chip technology.

• Leaked data revealed a Chinese AI system designed to automatically flag content to be censored.

Influence

• Bloomberg has a piece on how foreign governments and tech companies alike are lobbying Trump to ditch the AI diffusion rule.

  • Malaysia, though, said it will increase surveillance on chip flows to appease the US.

• Stephen Orlins, president of the National Committee on US-China Relations, said that China and the US need to cooperate on AI.

• TechNet held a fly-in to discuss AI and other issues with lawmakers. Google and Amazon representatives were reportedly there.

• The Hacking Policy Council advised against requiring AI developers to share red-teaming results with the government, saying that could “inadvertently create significant security risks”.

• Reid Hoffman said the “Biden executive order was directionally right, meant as it was to tackle major harms rather than any harm you could think of”.

• Seán Ó hÉigeartaigh has a great piece on how international AI governance can continue post-Paris.

• A bunch of experts called for the creation of an Australian AI Safety Institute.

• Control AI said its plan is to “inform every relevant person in the democratic process [about the risks of artificial superintelligence] – not only lawmakers, but also executive branch, civil service, media, civil society, etc –, and convince them to take a stance on these policies”.

Industry

• OpenAI is reportedly close to finalizing a $40b funding round led by SoftBank, valuing the company at $300b. It would be the largest venture funding round in history.

  • OpenAI reportedly expects to triple its revenue to $12.7b this year. It’s also reportedly talking about building a 5 exabyte data center.

• OpenAI added much better image generation tools to GPT-4o, sparking a viral trend of making every image look like a Studio Ghibli still. Unsurprisingly, that’s led to a huge argument about copyright and the value of art.

  • The new tool is also much more permissive, and allows users to generate images of adult public figures. OpenAI’s model behavior lead said the company is “shifting from blanket refusals in sensitive areas to a more precise approach focused on preventing real-world harm”.

• GPT-4o also got an upgrade; people are very impressed.

• Microsoft has reportedly abandoned data center projects equivalent to 2 gigawatts of electricity capacity, according to TD Cowen analysts.

  • Alphabet and Meta reportedly grabbed some of the newly available leases.

• H20 chip inventory in China is reportedly depleting amid surging demand from Chinese tech companies.

  • New energy efficiency rules, meanwhile, might soon ban Chinese companies from buying H20s.

• The FT has a piece on how Chinese companies are “overhauling their business models” in response to DeepSeek, with 01.ai shifting from a focus on training its own models to reselling fine-tuned versions of DeepSeek’s models.

• DeepSeek released a new and improved version of v3, this time under an MIT license.

• Alibaba released a new open weight vision-language model and a new open-weight multimodal model which it says can run on edge devices like phones.

• Microsoft announced six new AI security agents to handle tasks like processing phishing alerts and prioritizing incidents.

  • It also announced two new "deep reasoning" AI agents for Microsoft 365 Copilot.

• Google started rolling out real-time AI video features to Gemini Live, allowing it to see your phone’s screen and camera feed.

• Apple reportedly wants to build its own AI models for its Visual Intelligence features, and is considering adding cameras to Apple Watches and AirPods to enable the feature on them.

• Anthropic and Databricks struck a five-year, $100m partnership to sell each others’ enterprise AI tools.

  • Databricks also announced a technique called Test-time Adaptive Optimization, which “uses some relatively lightweight reinforcement learning to basically bake the benefits of best-of-N into the model itself”.

• OpenAI and Meta are both reportedly talking to Reliance Industries about partnering on AI projects in India.

• xAI integrated Grok with Telegram.

• CoreWeave priced its IPO at $40 per share, below its expected range, raising $1.5b. It sold fewer shares in the offering than expected too. It starts trading publicly today.

• Nvidia is reportedly close to acquiring GPU server rental startup Lepton AI for “several hundred million dollars”.

• AI chip startup FuriosaAI reportedly rejected an $800m takeover offer from Meta.

• Butterfly Effect, the Chinese company behind Manus, is reportedly trying to raise at a $500m valuation. It’s reportedly approaching US investors.

• Crusoe raised $225m in debt.

• Browser Use, which makes websites more readable for AI agents, raised $17m.

• Cerebras’s IPO has reportedly been delayed due to an ongoing CFIUS review of G42's investment in the company.

• Chevron is pushing for AI data centers to be powered by on-site natural gas generators.

• Cassava Technologies partnered with Nvidia to build Africa's first “AI factory”.

Moves

• OpenAI is not hiring a replacement for former CTO Mira Murati, and instead said Sam Altman will now do more technical work.

  • COO Brad Lightcap is taking on more business ops work, Mark Chen is now chief research officer, and Julia Villagra is chief people officer.

• The Senate confirmed Michael Kratsios as director of White House OSTP.

  • In a letter supposedly written by Trump, the president said that “just as FDR tasked Vannevar Bush, I am tasking you with meeting the challenges below to deliver for the American people”.

• Alex El-Fakir joined OSTP as senior adviser for strategic operations.

Best of the rest

• New interpretability research from Anthropic made a bunch of breakthroughs. Notably, it found that Claude plans ahead and sometimes fabricates reasoning.

  • DeepMind's mechanistic interpretability team published some updates too, saying that sparse autoencoders aren’t as effective as they hoped.

• Anthropic also published research on how malicious AI models can subtly sabotage AI research tasks in “ways that are hard to detect”.

• The Arc Prize Foundation released ARC-AGI-2, a new benchmark on which the best models only get around 4%, vs. humans’ 60%.

• A federal judge rejected OpenAI's request to dismiss The New York Times' copyright lawsuit. Another federal judge rejected music publishers' request to block Anthropic from using song lyrics to train Claude.

• OpenAI said it will adopt Anthropic’s Model Context Protocol.

• AI sales startup 11x falsely claimed companies like ZoomInfo and Airtable as customers on its website, according to TechCrunch. They’re backed by a16z, which doubled-down on its support for the company.

• OpenAI and MIT published new research which found, per Platformer, that “heavy chatbot usage is correlated with loneliness and reduced socialization”.

• Character.AI launched a “Parental Insights” feature that lets kids send weekly reports about their chatbot usage to their parents.

• Anthropic published new data on how people are using Claude, finding that the release of 3.7 Sonnet didn’t change much.

• New research found that “areas with higher adoption of Google Translate experienced a decline in translator employment”.

• The FT has a good piece on the impact of AI data centers on water supplies.

• Reed Hastings donated $50m to Bowdoin College to create an “AI and Humanity” research initiative.

• Jack Clark has some thoughtful discussion of what might need to happen in a short timeline world — and what might go wrong if we do take those actions.

• Forethought published new research arguing that automated AI R&D could cause a “software intelligence explosion”, which “could potentially create dramatically more advanced AI systems within months, even with fixed computing power”.

• Dwarkesh Patel’s got a book out: "The Scaling Era: An Oral History of AI, 2019-2025”. It’s good — I highly recommend reading it while you wait for Transformer to return.

Thanks for reading; have a great weekend and see you in a few weeks.