Transformer Weekly — July 26
Zuck on open source, Anthropic on SB 1047, and Google’s IMO solving AI
Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
Before we dive in: I’m going to be in the Bay Area from August 5th-9th, and would love to meet up with readers and get your thoughts on what I should be writing about. If you’re around and would like to chat, email me: shakeel@tarbellfellowship.org. Thanks!
Top stories
Meta released Llama 3.1 models, including Llama 3.1 405B, which the company claims is “the first frontier-level open source AI model”.
Accompanying the launch, Mark Zuckerberg published an essay touting the benefits of open source AI.
In the essay, Zuck claims that “open source AI will be safer than the alternatives”.
It will prevent unintentional harm (such as loss of control scenarios), he says, because “the systems are more transparent and can be widely scrutinised”.
Misuse risks, meanwhile, won’t be an issue because “as long as everyone has access to similar generations of models – which open source promotes – then governments and institutions with more compute resources will be able to check bad actors with less compute”.
He also says that Meta has the “goal of mitigating risks before release”, though doesn’t acknowledge that it’s trivial to undo these risk mitigations.
And US-China competition, he says, is irrelevant because China will just steal closed source models anyway.
In Time, Andrea Miotti and Hamza Chaudhry pointed out Zuckerberg’s sloppy logic.
“Zuckerberg admits that advanced AI technology is easily stolen by hostile actors, but his solution is to just give it to them for free,” Miotti said.
Also notable: Llama 3.1 models are not currently multimodal.
Anthropic came out against SB 1047 in its current form.
The company said that “the current version of SB 1047 has substantial drawbacks that harm its safety aspects and could blunt America's competitive edge in AI development”.
Instead, Anthropic has proposed to “refocus the bill on frontier AI safety and away from approaches that aren't adaptable enough for a rapidly evolving technology”.
It wants the bill to shift from “pre-harm enforcement” to “outcome-based deterrence”, and to get rid of the sections establishing a Frontier Model Division in California.
On Transformer: What might Kamala Harris mean for AI regulation?
Harris is now all-but-certain to be the Democratic nominee for president. For advocates of AI regulation, that’s probably a good thing.
As vice-president, Harris has been extremely vocal about the need for the government to tackle the potential risks from artificial intelligence.
In a November 2023 speech, she acknowledged that AI might “endanger the very existence of humanity”, citing “AI-formulated bioweapons” and “AI-enabled cyberattacks” as particular concerns.
In that speech, Harris said voluntary commitments on AI safety weren’t enough.
“As history has shown, in the absence of regulation and strong government oversight, some technology companies choose to prioritise profit over the wellbeing of their customers, the safety of our communities, and the stability of our democracies,” she said, arguing that the US needs “legislation that strengthens AI safety without stifling innovation”.
Along with tackling potential catastrophic risks, Harris has also shown an interest in broadening the definition of “AI safety” to include tackling AI’s present-day harms. “To define AI safety,” she said, “we must consider and address the full spectrum of AI risk”.
The discourse
Sam Altman said the US needs to control the future of AI, and called for investment in security, AI infrastructure, a better commercial diplomacy policy, and an international governance regime:
“If we want to ensure that the future of AI is a future built to benefit the most people possible, we need a U.S.-led global coalition of like-minded countries and an innovative new strategy to make it happen.”
Alondra Nelson and Ami Fields-Meyer said Trump’s AI policies would be a disaster:
“Trump’s AI pledge is not a policy solution. It’s a campaign promise to Big Tech–one which would unleash an already out of control industry, with harms that would permeate nearly every aspect of our lives.”
Microsoft’s Teresa Hutson agreed:
“We do think we need some rules of the road. And also would prefer not to have this regulated at the state level. Fifty states regulating this will make business impossible."
Roger McNamee called out a16z’s recent marketing push:
“The Big Lie of Andreessen and Horowitz is that they represent “little tech.” Their startups are smallish, but designed to produce global monopolies. Decentralisation in crypto and open source in AI are total BS. The harms are huge.”
Policy
The CCP’s Third Plenum program calls for “establishing an AI safety supervision / regulatory system”.
China expert Matt Sheehan says this is “the clearest indication we've seen that concerns about AI safety have reached top CCP leadership, and that they intend to take some action on this”.
Sens. Schatz, Lujan, Welch, King and Warner asked OpenAI how it plans to “meet its public commitments on safety”, in response to recent claims by whistleblowers that the company’s approach is rather lax.
The EU is reportedly asking companies to make voluntary commitments on AI safety until formal rules kick in.
The Senate passed the DEFIANCE Act, which would let victims of nonconsensual deepfake porn sue whoever made it.
The UK’s Department of Science, Innovation and Technology launched an AI Opportunities Action Plan, to be headed up by Matt Clifford.
It will “identify how to boost uptake of AI across the public and private sectors to increase productivity”.
Lina Khan acknowledged the harms that open models enable (such as fraud and deepfake nudes), and said that “openness is just one way to promote a level playing field”.
The FTC, DoJ, UK CMA and European Commission issued a joint statement on competition in generative AI, warning of various risks and saying they will “work to ensure effective competition and the fair and honest treatment of consumers and businesses”.
Ursula von der Leyen endorsed a “CERN for AI” project, though details are scarce, according to Tarbell Fellow Jacob Wulff Wold at Euractiv.
Influence
Nvidia’s Josh Parker said he recently met the House AI task force to argue that AI’s climate impact is overstated.
A new poll found that 59% of Californian voters support SB 1047, while only 20% oppose it. Tech workers like it even more than the average voter.
Another poll found that 73% of US voters think “Congress should not fund AI innovation until it passes laws that require AI companies to implement consumer protection guidelines”.
And yet another poll found that voters want Trump to prioritise AI safety. Only 3% of Trump voters want to see AI regulations eliminated.
The Manhattan Institute’s Nick Whitaker published a “Playbook for AI Policy” with a Republican bent to it.
He argues that though AI policy mustn’t be “overly broad”, the US “needs to seriously reckon with the national security risks of AI technology”.
Bloomberg reported that Meta lobbyists Dustin Carmack and Rick Dearborn helped write Project 2025.
Dan Hendrycks said he’s divesting his equity stake in Gray Swan AI. He’d previously been accused of having a financial interest in promoting AI regulation.
Chamber Hill Strategies and Akash Wasil are no longer registered lobbyists for Control AI. Franklin Square Group no longer lobbies for Stability AI, either.
Industry
Google DeepMind has built an AI that can achieve a silver-medal standard at International Mathematical Olympiad problems. The system combines AlphaProof and AlphaGeometry 2, and experts think it’s a big deal.
Google also said its “AI infrastructure and generative AI solutions for cloud customers have already generated billions in revenues”. But its stock dropped, seemingly in part due to nervousness about whether AI will generate profits for the company.
Apple signed onto the White House voluntary commitments on AI safety.
Nvidia is reportedly working on a “B20” chip — a version of the B200 which is designed to get around export controls.
OpenAI announced SearchGPT, its Perplexity and Google rival. It’s only available to a small group of users for now — and its demo video seems to contain a bunch of mistakes.
Fine-tuning GPT-4o mini is free for the next couple months. OpenAI also seems to have solved the “ignore all previous instructions” jailbreak method.
An analysis from The Information argues that OpenAI might lose “as much as $5 billion this year”. The company’s reportedly spending $4b a year on inference and might spend $3b this year on training.
Amkor is receiving ~$600m under the CHIPS Act to build a new chip packaging facility in Arizona.
SK Hynix is building a new chip plant in Yongin. It reported its highest profit since 2018 this week.
Nvidia has reportedly greenlit Samsung’s fourth-gen HBM3 memory chips for use in its China-focused H20 GPUs.
Stability launched “Stable Video 4D”, which takes video footage and generates new angles for it.
Google has reportedly discussed a partnership with EssilorLuxottica.
Cohere raised $500m at a reported $5.5b valuation. It laid off 20 employees the next day.
Harvey, the AI tool for law firms, raised $100m at a $1.5b valuation.
Saudi Aramco’s venture arm invested $15m in Rebellions, an AI chipmaker.
Moves
Aleksandr Mądry is out as head of OpenAI’s Preparedness team. It sounds like he wasn’t happy with the change.
The Information reported that OpenAI’s also restructured the team to distance it from leadership: while Mądry reported directly to Mira Murati, new manager Tejal Patwardhan will report to Joaquin Quiñonero Candela, who will in turn report to Lilian Weng — who ultimately reports to Murati.
Mądry, meanwhile, is working on a “new and v important research project”, Sam Altman said.
Adam Eisgrau is now senior director of AI, creativity and copyright policy at Chamber of Progress.
Aviv Ovadya is launching the AI & Democracy Foundation, which is hiring.
Gian M. Volpicelli is leaving Politico.
Best of the rest
Video game performers are going on strike, having not been able to agree on AI rules with studios. A Wired investigation this week found that game studios are already using AI tools for game development.
A new paper finds that AI models “collapse when trained on recursively generated data”. Over at MIT Tech Review, Tarbell Fellow Scott Mulligan has a nice explainer of what this means.
An all-star group of researchers published a monster paper on “Open Problems in Technical AI Governance”.
An AI alignment paper won a “best paper award” at ICML for the first time.
OpenAI published new alignment research on “rule-based rewards”.
MIT researchers published new work on automated interpretability.
Europol said it expects AI-generated CSAM to continue to proliferate.
Meta’s Oversight Board said the company needs to do more to tackle deepfake porn on its platforms.
In Time, Hamza Chaudhry argued that it’s very bad that AI evals are only really done in English — because models might give much bigger capability uplift for things like bioweapons development for non-English speakers.
Academics are mad that Taylor & Francis sold their work to Microsoft for use in AI products.
The IMF launched an “AI Preparedness Index”.
Thanks for reading; I’ll see you next week.