Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
Corrections
Last week, I said the Remote Access Security Act would “expand export controls to cloud computing services”. It would actually just give the President/Commerce Dept authority to do so: read CAIP’s excellent newsletter for more detail.
I also included Axios’ report that Senate Commerce would hold a markup session on Wednesday. That report seems to have been false; no markup session happened.
Top stories
Lots of stuff happened at and around the AI Seoul Summit.
16 companies, including Amazon, Anthropic, Google, IBM, Meta, Microsoft, OpenAI and Zhipi.ai, signed the “Frontier AI Safety Commitments”.
27 nations, including the US and UK, plus the EU, committed to working on “severe risks” from AI.
That includes risks coming from a system which could “meaningfully assist non-state actors in advancing the development, production, acquisition or use of chemical or biological weapons” …
… and risks “posed by the potential model or system capability or propensity to evade human oversight, including through safeguard circumvention, manipulation and deception, or autonomous replication and adaptation conducted without explicit human approval or permission”.
Perhaps the most important section is this: “We note the importance of gathering further empirical data with regard to the risks from frontier AI models or systems with highly advanced agentic capabilities, at the same time as we acknowledge the necessity of preventing the misuse or misalignment of such models or systems, including by working with organisations developing and deploying frontier AI to implement appropriate safeguards, such as the capacity for meaningful human oversight.”
China didn’t sign, but the press release says “China engaged in today’s discussions on AI safety which covered areas of collaboration ahead of the France AI Action Summit, and we look forward to continued engagement with China on this work”.
The US, UK, Japan, Singapore and EU (plus others) said they’ll establish an “international network” of AI safety institutes, which will “promote complementarity and interoperability” between the institutes.
That could include “sharing of information about models”, “the monitoring of AI harms and safety incidents”, and “the exchange or joint creation of evaluations, data sets and associated criteria”.
They also made a pretty generic statement to “affirm our common dedication to fostering international cooperation and dialogue on artificial intelligence”, which namechecks the Hiroshima Process, the OECD AI principles and the UN Global Digital Compact.
Talk of the town, though, was the Scarlett Johansson-OpenAI story.
To recap: on Monday, Johansson put out a statement revealing that OpenAI had twice asked her to be the voice of GPT-4o, which she declined. She was then “shocked” and “angered” that one of the new voices, Sky, sounded “eerily similar” to her.
Coupled with Sam Altman’s “her” tweet (seemingly referencing the ScarJo movie) alongside GPT-4o’s launch, it seemed OpenAI was intentionally copying her voice.
OpenAI had previously said that “Sky's voice is not an imitation of Scarlett Johansson”, and execs had said similar things to journalists.
The reality seems to be more complicated. The Washington Post spoke to the agent of the actress who is actually “Sky”, and reports she was “hired in June to create the Sky voice, months before Altman contacted Johansson”, and “neither Johansson nor the movie ‘Her’ were ever mentioned by OpenAI”.
That doesn’t rule out that she was chosen for her similarity to Johansson’s voice, though.
That complicated reality didn’t stop the subject dominating the discourse, though. People really hate OpenAI.
Katie Notopolous: “Are these really the people we should believe are such geniuses and have such great judgement that they should be in charge of this potentially life-changing technology?”
Charlie Warzel: “The Johansson scandal is merely a reminder of AI’s manifest-destiny philosophy: This is happening, whether you like it or not.”
“It’s not stealing to build the future if you believe it has belonged to you all along.”
Ed Zitron: “Sam Altman is full of shit”.
Tino Gagliardi, president of the American Federation of Musicians: “If someone can try and do this to one of the most famous actresses in the world, they can absolutely do it to anyone.”
As for Johansson’s legal threats: Wired has a good piece on whether she’s got a good case or not (legal experts seem split).
And because OpenAI’s incapable of having a normal week, there was yet another controversy, this time around how it forced former employees to sign a nondisparagement agreement under threat of clawing back their vested equity.
Perhaps worse than the (very bad) practice itself is that OpenAI execs seem to have feigned ignorance about it.
Altman said he didn’t know anything about it, but his signature’s on a document giving the power to clawback the equity, according to excellent reporting from Vox’s Kelsey Piper; while OpenAI’s COO, Chief Strategy Officer and VP of People all signed the actual termination docs drawing on the clause.
On the back of Piper’s reporting, OpenAI seems to have backed down: they’ve now emailed former employees saying that the company “has not cancelled, and will not cancel, any vested units” and will release former employees from NDAs (“unless the nondisparagement provision was mutual”).
(It’s still unclear, though, if those former employees can participate in tender offers, which lawyers at one point said they would be banned from.)
And one more controversy, just for fun: Fortune reported that OpenAI never followed through on its promise to give the Superalignment team access to 20% of its compute.
“Instead … the team repeatedly saw its requests … turned down by OpenAI’s leadership, even though the team’s total compute budget never came close to the promised 20% threshold.”
Notable: “the problems with accessing compute worsened in the wake of the pre-Thanksgiving showdown between Altman and the board of the OpenAI nonprofit foundation.”
The California Senate passed SB 1047, and now the Big Tech lobbying effort against it has geared up.
Meta and IBM’s AI Alliance came out against it, saying it was “technically infeasible” and “would slow innovation, thwart advancements in safety and security, and undermine California’s economic growth”. The Alliance says it’s “deeply concerned about the potential establishment of an anti-open-source precedent”.
In particular, the Alliance says it’s worried that “The bill requires developers of AI models of 10^26 FLOPS or similar performance (as determined by undefined benchmarks) to implement a full shutdown control that would halt operation of the model and all derivative models. Once a model is open sourced and subsequently downloaded by a third party, by design developers no longer have control over a model.”
Sen. Scott Wiener seems to suggest the Alliance has misread the bill.
In an open letter addressing many concerns (worth reading in full), he writes that the “open source developers training these extremely powerful models must be able to shut down their models — but only while those models remain in their possession, not after they have been open sourced”.
But while the shutdown provision wouldn’t cause problems for open source developers, other provisions would.
Wiener notes that “If a developer finds [extremely hazardous capabilities] in its model, the developer must install reasonable safeguards to make sure it doesn’t cause mass harm”. The bill says one mandatory safeguard for such models would be to “prevent an individual from being able to use the model to create a derivative model that was used to cause a critical harm” — and it’s currently impossible to put irreversible safeguards on open source models.
The problem here, which ought to be obvious but which people don’t want to say out loud, is that it’s impossible to safely open source a model capable of doing catastrophic harm, but companies want to keep making them anyway.
The discourse
Tyler Cowen said the AI safety movement “is dead”:
“Top scientists are not publishing new models in Science or Nature, two of the better-known journals, suggesting AI will bring about the end of the world.”
The day before Cowen’s column, in Science, an all-star group of AI experts (Bengio, Hinton, Yao, Song and others), said we’re not ready for transformative AI:
“Society’s response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness and barely address autonomous systems.”
Founders Fund’s John Luttig poured cold water on the “open-source is great” narrative:
“Open-source AI will become a financial drain for model builders, an inferior option for developers and consumers, and a risk to national security … When open-source model builders release the model weights, they arm our military adversaries and economic competitors. If I were the CCP or a terrorist group, I’d be generously funding the open-source AI campaign.”
In The Atlantic, Tyler Austin Harper said AI risks making us less human:
“The new AI products coming to market are gate-crashing spheres of activity that were previously the sole province of human beings … Artificial intelligence could significantly diminish humanity, even if machines never ascend to superintelligence, by sapping the ability of human beings to do human things.”
Yann LeCun thinks LLMs are “intrinsically unsafe”, but he’s got a better approach:
“We are at the point where we think we are on the cusp of maybe the next generation AI systems.”
Wired’s Steven Levy said “it’s time to believe the AI hype”:
“Sceptics might try to claim that this is an industry-wide delusion, fueled by the prospect of massive profits. But the demos aren’t lying.”
Kevin Scott thinks we’ll soon see much more powerful AI:
“We are nowhere near the point of diminishing marginal returns on how powerful we can make AI models as we increase the scale of compute.”
The Guardian’s Alex Hern commented on the exodus of safety-minded OpenAI staffers:
“If OpenAI, one of the biggest and best-resourced AI labs in the world, which claims that safety is at the root of everything it does, can’t even keep its own team together, then what hope is there for the rest of the industry?”
Policy
The EU AI Act is finally, completely done. It comes into force next month.
The European Commission threatened to fine Microsoft for failing to respond to a request for information about the risks posed by its AI tools.
Gov. Jared Polis signed Colorado’s AI bill into law.
The US AI Safety Institute released its strategic vision. It will “advance the science of AI safety” and “articulate, demonstrate, and disseminate the practices of AI safety”.
That will include testing models pre-deployment for things like “enabling chemical, biological, or cyber attacks and risks to human oversight or control”, and promoting “adoption of AISI guidelines, evaluations, and recommended AI safety measures and risk mitigations”.
It also said it would organise a meeting of different countries’ AISIs in the Bay later this year.
The UK and Canada signed an AI safety agreement.
Lina Khan said AI scraping might violate antitrust laws.
Sens. Cantwell and Moran introduced a bill on AI education, which would authorise the National Science Foundation to award AI scholarships.
China released an LLM trained on Xi Jinping Thought.
The UK announced £8.5m in grants for research into systemic AI safety. It’s also launching a semiconductor institute.
South Korea announced $19b in funding for chip companies.
The European Chips Act will spend $2.8b to set up a pilot line for “sub 2-nanometre” chips.
ASML reportedly told Dutch officials that it can remotely disable its EUV machines in the event of a Chinese invasion of Taiwan.
Emmanuel Macron laid out his AI vision ahead of the VivaTech conference. TL;DR: “The city of lights will become the city of artificial intelligence”.
UK culture secretary Lucy Frazer said last week that the government would bring in rules requiring transparency on how models are trained; but with this week’s election announcement that plan’s probably dead.
Influence
On Transformer: Alliance for the Future director Brian Chau has a history of racist and sexist remarks. In a post on AI regulation, Chau falsely claimed George Floyd was a "domestic abuser". Read the full story here.
Amazon and Meta joined the Frontier Model Forum.
Politico has a great longread on the “coming war between DC and Silicon Valley”. Hard to summarise, but a couple standout bits:
“‘In no way did the [executive order idea] come from them,’ says Ben Buchanan, the senior adviser for AI at the White House who was one of the drafters of the order, referring to the tech industry and suggestions that the rules were written in ways that help existing players. ‘I reject wholeheartedly the notion of regulatory capture.’”
Rohit Chopra: “We struggle to see how it doesn’t turn, absent some government intervention, into a market structure where the foundational AI models are not dominated by a handful of the big tech companies.”
“The coming of AI offers a chance to revive competition and innovation, [Chris Dixon] adds, through more robust government action. ‘Is tech just five big companies? Or is the tech economy going to be a vibrant economy?’”
Chamath Palihapitiya and David Sacks are hosting a Trump fundraiser on June 6.
A consortium of advocacy groups, including AI Now, Color of Change, and EPIC, released a “shadow report to the US Senate AI policy roadmap”. It mostly lambasts Congress for moving slowly and serving corporate interests.
Politico reported on how Faculty has got a bunch of UK government contracts for AI safety work, “without any competitive bidding process”.
The Tech Transparency Project found that 15 out of 22 academics who participated in Rep. Khanna’s AI roundtable “are tied to Big Tech companies”.
Politico analysed different companies’ lobbying language; it’s an interesting snapshot of their priorities.
Industry
Google’s new “AI Overviews” appear to be another disaster for the company; they’re telling users to put glue on their pizza and that Obama is Muslim.
Lots of Microsoft announcements. Most importantly: Copilot+ PCs (Arm powered, to start with) that track all your activity (and which are already being investigated by data regulators); Copilot Agents; and Phi-Silica, a 3.3b parameter model built for Copilot+ PCs.
Brad Smith also said Microsoft’s deal with G42 could lead to the transfer of model weights to the UAE. The two companies are building a $1b data centre in Kenya, too.
Stratechery has long, complicated interviews with Satya Nadella and Kevin Scott on Microsoft’s AI strategy.
Also, the UK CMA said it won’t investigate its Mistral investment further.
Chinese companies are selling on-prem “AI-in-a-box” products, with Huawei’s help, the FT reports.
There’s an AI price war going on in China, with Alibaba, Baidu and ByteDance all massively cutting prices. ByteDance’s Doubao bot is now China’s most popular.
Amazon is reportedly upgrading Alexa with an LLM, and might start charging for it.
OpenAI signed a $250m licensing deal with News Corp.
Entrepreneur First announced “def/acc”, a new program that will fund “technology to protect us from the biggest threats we face” (especially AI).
Scale AI raised $1b at a $14b valuation.
DeepL raised $300m at a $2b valuation.
H, formerly known as Holistic AI, raised $200m in initial financing. Eric Schmidt’s an investor.
Suno raised $125m.
Groq is reportedly trying to raise $300m.
The Snowflake-Reka AI deal seems to have collapsed.
Humane is reportedly trying to sell itself.
Counterpoint Research says global foundry revenue was up 12% YoY in Q1.
The FT has an interesting piece on the Triton alliance to break Nvidia’s CUDA moat.
Moves
Gretchen Krueger, a policy researcher at OpenAI, resigned. She cited similar concerns to Ilya Sutskever and Jan Leike.
Meta’s creating a new “product advisory council” which will advise on its AI products. It’s made up of Patrick Collison, Nat Friedman, Tobi Lütke and Charlie Songhurst. Many have noticed they’re all white men.
Jun YoungHyun is the new head of semiconductors at Samsung.
Audrey Tang joined the Collective Intelligence Project as a senior research fellow.
Inflection filled out its new exec team: alongside Sean White as CEO are Vibhu Mittal (CTO), Ted Shelton (COO) and Ian McCarthy (head of product).
Indent CEO Fouad Matin and his team are joining OpenAI to work on “AGI-ready security”.
Dhruv Batra, who was head of Facebook’s embodied AI team, has left.
The UK AI Safety Institute is opening an office in San Francisco.
Best of the rest
Anthropic published some cool new interpretability research; they’ve “identified how millions of concepts are represented inside Claude Sonnet”. They’ve also figured out how to manipulate those features, “artificially amplifying or suppressing them to see how Claude's responses change”. That has led to, for instance, a version of the model which is hilariously obsessed with the Golden Gate Bridge.
Some other interesting papers: Advancing Human Interaction Evaluations for LLM Harms and Risks; Societal Adaptation to Advanced AI; Visibility into AI Agents.
The new Foundation Model Transparency Index was released.
A new global survey found that “globally, people feel positive towards AI”, particularly in China, India, Indonesia and Kenya. “Countries including France, the U.S., Canada, and the U.K. display the most negative views.”
“Technology companies are trusted to regulate AI and are the preferred actor for doing so. However, when asked directly if they trust tech companies to self-regulate, respondents are more hesitant: only 21% say yes.”
The UK AISI said its tests show “all tested LLMs remain highly vulnerable to basic jailbreaks, and some will provide harmful outputs even without dedicated attempts to circumvent their safeguards”.
The political consultant who deepfaked Biden’s voice for a robocall was fined $6m. He faces 26 criminal charges.
Wired’s got a great deepdive into how AI deepfakes are being used for voter outreach in India’s election, with millions of robocalls and videos of dead politicians. The Guardian reports that Meta “approved a series of AI-manipulated political adverts during India’s election that spread disinformation and incited religious violence”.
The Guardian also covered how Chinese state-linked groups are using deepfake news presenters to spread propaganda.
A man in Wisconsin was charged with making AI-generated child sexual abuse images. He allegedly used Stable Diffusion to do it.
Forbes wrote about the disturbing flood of AI generated images of highly-sexualised children on TikTok and Instagram, often accompanied by extremely gross comments from older men.
ABC profiled the gig workers helping train AI, and their not-great working conditions.
Geoffrey Hinton thinks we need universal basic income.
Eric Schmidt said that if we develop AI with free will, we can just “unplug them”.
The WSJ has a good explainer of retrieval-augmented generation and vector databases.
Cade Metz took a look at the books in OpenAI’s library.
Thanks for reading. If you’re in the US or UK, have a great long weekend. See you next week.