Transformer Weekly — Aug 9
1047 opposition heats up | GPT-4o system card | Schulman leaves OpenAI
Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
First, some housekeeping: I’m taking a few weeks off from the weekly newsletter to work on some exciting new projects and travel. I’ll be back in your inboxes in September.
Top stories
Once again, the fight over SB 1047 is the week’s big story.
The bill’s opponents landed some hits: Fei-Fei Li wrote an op-ed in Fortune criticising the bill, while Rep. Zoe Lofgren told Scott Wiener that the bill “would not be good for our state”.
Three notable things here.
1. Li’s op-ed doesn’t disclose her significant conflict of interest: that her new AI startup has raised $100m from investors including Andreessen Horowitz (the organisation leading the anti-1047 campaign).
2. Google, which also opposes the bill, is Lofgren’s biggest donor.
3. Lofgren’s letter was leaked through a tweet from Andreessen Horowitz GP Martin Casado — perhaps suggesting the firm had some involvement in its creation and/or distribution.
A group of University of California faculty and students also opposed the bill, as did SF mayoral candidate Mark Farrell.
And Andreessen Horowitz wrote a 14 page letter criticising the bill.
On the other side, Geoffrey Hinton, Yoshua Bengio, Stuart Russell and Lawrence Lessig sent a letter to Gavin Newsom and other California lawmakers supporting the bill, saying that not passing its basic measures would be a “historic mistake”.
“There are fewer regulations on AI systems that could pose catastrophic risks than on sandwich shops or hairdressers … Relative to the scale of risks we are facing, this is a remarkably light-touch piece of legislation.”
Notion co-founder Simon Last also endorsed the bill:
“Product safety testing is standard for many industries, including the manufacturers of cars, airplanes and prescription drugs. The builders of the biggest AI systems should be held to a similar standard.”
And new AIPI polling found that “65% of Californians support the bill as currently written”.
Scott Wiener, meanwhile, went on Lawfare’s podcast to defend the bill:
“This is not about guaranteeing that your model is not going to cause harm. It's not about certifying that it can't cause harm. It is about conducting reasonable safety evaluation, determining whether there is a real, actual risk of catastrophic harm, and then, if so, taking reasonable steps to reduce that risk.”
The discourse
Gary Marcus said OpenAI should worry us:
“We simply can’t trust giant, privately held AI startups to govern themselves in ethical and transparent ways. And if we can’t trust them to govern themselves, we certainly shouldn’t let them govern the world.”
Daron Acemoglu wants AI regulation:
“Since any technology can be used for good or bad, what ultimately matters is who controls it, what their objectives are, and what kind of regulations they are subjected to.”
A report from the Ada Lovelace Institute found that AI evaluations have significant issues:
“Evaluations alone are not sufficient for determining the safety of foundation models, the systems built from them and their applications for people and society in real-world conditions.”
EU AI Act architect Gabriele Mazzini appears to be showing some regret:
“The regulatory bar maybe has been set too high … There may be companies in Europe that could just say there isn’t enough legal certainty in the AI Act to proceed.”
Policy
The UK Competition and Markets Authority opened a merger inquiry into Amazon’s investment in Anthropic.
Sen. Schumer said he’s going to prioritise deepfake election legislation in the next Senate session, possibly by attaching bills to next month’s stopgap funding bill.
He also said the NDAA would likely include some AI legislation.
Sen. Warren and Rep. Trahan asked Sam Altman for “information about OpenAI’s whistleblower and conflict of interest protections in order to understand whether federal intervention may be necessary”.
They also asked if OpenAI has “changed its process for ensuring that new products undergo a predeployment safety review”.
Last week, Sen. Grassley also asked OpenAI for evidence that it’s no longer “stifling” whistleblowers.
Google lost its antitrust case, which some think might have implications for AI competition.
Five secretaries of state asked Elon Musk to stop Grok giving people false election information.
An NYT investigation found that vendors are smuggling in thousands of Nvidia H100s into China. One vendor said he recently sold a cluster of 2000 H100s for $103m.
Chinese companies are reportedly stockpiling HBM chips in anticipation of new export controls, too.
The FEC said it won’t issue rules on AI in political ads this year.
Influence
The NYT reported on how Nvidia’s beefed up its lobbying operation in preparation for antitrust investigations.
130 tech VCs (including some big AI names like Reid Hoffman) gathered on a “VCs for Kamala” call, raising $150,000.
On Transformer: Twitter is overrun with bot-like accounts praising the UAE's approach to AI, raising questions of whether the country is running an influence campaign.
Industry
Elon Musk filed a new federal lawsuit against OpenAI and Sam Altman, alleging they violated racketeering laws and deceived investors about the company's mission.
“Altman, in concert with other defendants, intentionally courted and deceived Musk, preying on Musk’s humanitarian concern about the existential dangers posed by AI,” the lawsuit says.
Nvidia’s new Blackwell chips are reportedly delayed by at least three months.
Interesting tidbit from the story: Google’s reportedly ordered more than 400,000 GB200s, while Meta’s ordered $10b+ worth of chips.
Semianalysis has a good, technical explanation of what’s going wrong at Nvidia (much of the issues seem to be to do with the new CoWoS-L packaging technology).
OpenAI released the GPT-4o system card. Some interesting snippets:
“During testing, we also observed rare instances where the model would unintentionally generate an output emulating the user’s voice.”
“Persuasive capabilities of GPT-4o marginally cross into our medium risk threshold from low risk.”
It worked with Apollo Research to evaluate the model’s scheming capabilities, and METR to assess autonomous capabilities.
OpenAI also released a new, cheaper version of GPT-4o.
It also reportedly has an AI watermarking and detection tool internally, but won’t release it.
And it said GPT-5 won’t be announced at DevDay, which will instead be a series of “educational” sessions in San Francisco, London and Singapore.
Alibaba released Qwen2-Math, which it says outperforms GPT-4o and Claude 3.5 Sonnet on the MATH benchmark.
Anthropic launched an invite-only bug bounty program for testing its “next-generation system [it’s] developed for AI safety mitigations”.
Mistral released “an early version of Agents, that wraps models with additional context and instruction”.
Nvidia scraped videos from YouTube and other sources to train an unreleased video generation model, according to leaked internal documents obtained by 404 Media.
ByteDance launched a new text-to-video model.
Google DeepMind’s built a table-tennis-playing robot which plays at a human level.
Perplexity says it’s now handling 250m queries a month. It reportedly has $35m in annualised revenue.
Super Micro revenue missed estimates.
Hugging Face bought XetHub, which it says will help developers make big models.
AI chipmaker Black Sesame had a rough IPO in Hong Kong, with shares dropping 27% on day one.
Groq raised $640m at a $2.8b valuation.
Glean, which makes AI-powered search tools for businesses, is reportedly raising $250m at a $4.5b valuation.
ProRata.ai, a new AI chatbot company, came out of stealth with a $25m Series A and revenue-sharing partnerships with the FT, Axel Springer, and The Atlantic.
Figure unveiled its new robot.
Black Forest Labs teased a very impressive video generation model.
Moves
John Schulman left OpenAI for Anthropic. He said he’s leaving to “deepen” his focus on alignment work, but that he’s “not leaving due to lack of support for alignment research at OpenAI”.
Zico Kolter, an AI safety and alignment researcher, joined OpenAI’s board. Kolter runs Carnegie Mellon’s Machine Learning Department, and recently cofounded Gray Swan, an AI safety and security startup.
Yoshua Bengio joined the UK’s ARIA as scientific director for the Safeguarded AI project.
Nitarshan Rajkumar, who many credit with catalysing the UK policy response to AI in the past year, is leaving the Civil Service.
DeepMind robotics researcher Alex Irpan is moving to the company’s AI safety team, in part because he’s “not sold that superhuman systems will do the right thing without better supervision than we can currently provide”.
Peter Hase is doing an AI safety residency at Anthropic.
James Baker is now an AI policy and governance manager at Meta.
Seth Lazar is joining the Knight Institute as a senior AI advisor.
Sal Rodriguez is joining CNBC as deputy tech editor.
Josh Koen joined The Information to cover “the most powerful/interesting people in San Francisco tech with an emphasis on politics & public policy”.
Best of the rest
METR found that AI agents can do a bunch of things “substantially cheaper” than humans can (though it varies from task to task).
MITRE said the government should require AI red-teaming as part of the procurement process.
Bloomberg has a deep-dive into Gryphon Scientific and its work on assessing the risks of AI-generated bioweapons.
SCMP has a piece on how Chinese semiconductor equipment companies are improving (but still struggling to nail lithography).
Elliott Management said AI’s a bubble.
Utility stocks are getting an AI bump.
Indian outsourcing companies are pretty worried that AI will replace their work.
Popular YouTuber Kurzgesagt published a video on the risks of superintelligent AI.
Perplexity gave a $250k grant to Northwestern University to fund research on how AI can be used in journalism.
Timothy Lee published an unusually thoughtful critique of AI risk arguments.
The Hollywood Reporter has a good overview of the fight between showbiz unions and studios on AI usage.
Someone found the system prompts for Apple Intelligence.
Nicholas Carlini published a great look at how he uses AI tools in his daily work.
Reuters reported that in 2017 and 2018, Intel had a chance to buy a 15% stake in OpenAI for $1b, but turned it down. Oops!
Humane is reportedly seeing more returns than sales, and only around 7,000 units remain in circulation.
Thanks for reading: I’ll see you in September.