Welcome to Transformer, your weekly briefing of what matters in AI. If you’ve been forwarded this email, click here to subscribe and receive future editions.
Housekeeping: Transformer will be off next week; back July 5th.
Top stories
On Transformer: Ilya Sutskever announced his new company, Safe Superintelligence Inc., co-founded with Daniel Gross and Daniel Levy.
Sutskever told Bloomberg that SSI's “first product will be the safe superintelligence, and it will not do anything else up until then”.
That, Sutskever argues, will insulate the company from "the outside pressures of having to deal with a large and complicated product and having to be stuck in a competitive rat race" — a dynamic that seems to have caused tensions at OpenAI.
But no product means no revenue. And even by traditional startup standards, Sutskever's goal here is expensive — potentially in the $100+ billion range.
It’s not possible to raise that much from private investors. Which strongly suggests that Sutskever thinks that, through some form of algorithmic improvements, he can do this for much, much less than $100 billion.
How much lesss? One might look at a recent essay series from Sutskever's former colleague Leopold Aschenbrenner, who argues that there's a good chance we'll develop AGI by 2027, at which point frontier clusters would cost $10-100 billion.
Notably, Aschenbrenner recently started an AGI-focused investment firm, bankrolled by the Collisons, Nat Friedman and … Daniel Gross.
I’d bet that his firm is invested in SSI. (I reached out to Aschenbrenner to ask, but haven’t received a response).
Doing this would require huge algorithmic improvements. But if Sutskever’s bet works, it could mean ASI comes much, much sooner than people expected.
Also notable: The new company’s comms are being handled by Lulu Cheng Meservey.
Anthropic released Claude 3.5 Sonnet, which is both cheaper and better than Claude 3 Opus, and outperforms all models on most benchmarks.
The company’s also releasing “artifacts”, which lets Claude create code, images and websites in a window alongside the conversation. It can build complex games with a single prompt.
Anthropic says the UK AI Safety Institute did pre-deployment evaluations of a “near-final” version of the model, and shared those results with the US AISI.
Notably, Politico reported in April that Anthropic wasn’t giving the AISI access to models.
Anthropic didn’t say what AISI tested, but a model card addendum says that Anthropic itself tested the models for CBRN risks, cybersecurity, and autonomous capabilities (and worked with METR on the latter).
Anthropic said it classifies the new model “as an AI Safety Level 2 (ASL-2) model, indicating that it does not pose risk of catastrophic harm”, but that it did observe “an increase in capabilities in risk-relevant areas compared to Claude 3 Opus”.
OpenAI is considering dropping its non-profit structure, according to The Information.
Right now, the company’s for-profit entity is entirely controlled by a non-profit board with no fiduciary obligations.
Sam Altman has reportedly told investors that he’s considering a switch to a for-profit benefit corporation, the structure used by Anthropic and xAI.
The Information notes that the change could eventually lead to an IPO, and also “give Altman an opportunity to take a stake in the fast-growing company, a move some investors have been pushing”.
OpenAI pushed back on the reporting, saying “the nonprofit is core to our mission and will continue to exist”.
The discourse
House Majority Leader Steve Scalise said AI regulation isn’t coming anytime soon:
“There’s no consensus right now … we don't need to limit [the AI industry’s] growth by throwing a whole lot of new regulations on top of it to solve a problem that doesn't exist … Why would we want to cede ground to a country like China?”
Donald Trump talked to the All-In Podcast about AI:
“Two little simple letters, but it's big.”
Pope Francis warned the G7 about AI risks:
“We would condemn humanity to a future without hope if we took away people's ability to make decisions about themselves and their lives, by dooming them to depend on the choices of machines.”
The Washington Post has an interesting piece on how the Pope’s become such an influential figure in AI. Notable is that his chief AI advisor, Paolo Benanti, is worried about open-source models leading to bioweapon proliferation.
Arthur Mensch told the French Senate that AI could be controlled:
"When you write this kind of software, you always control what will happen, all the outputs of the software”.
This is, of course, completely untrue.
Masa Son reiterated his previous views on superintelligence:
“This is what I was born to do, to realise ASI.”
He thinks AGI is coming in 3-5 years and ASI in a decade.
Mira Murati said models are improving fast:
“If you look at the trajectory of improvement, systems like GPT-3, we're maybe, let's say, toddler level intelligence. And then systems like GPT-4 are more like smart high schooler intelligence. And then in the next couple of years, we're looking at PhD-level intelligence for specific tasks.”
Ray Kurzweil thinks AGI is still coming in 2029, and will change everything:
“This is AI’s most transformative promise: longer, healthier lives unbounded by the scarcity and frailty that have limited humanity since its beginnings.”
Meta Oversight Board chair Helle Thorning-Schmidt said she wants Meta to improve how it handles deepfakes:
“I think we will end up, after a couple of years, with Meta labelling AI content and also being better at finding signals of consent that they need to remove from the platforms, and doing it much faster.”
The Mercatus Center’s Matthew Mittelsteadt doesn’t like the ENFORCE Act:
“While the legislation focuses on ‘offensive’ systems, vulnerability detection lies at the heart of defensive efforts, too. Any restrictions on this tech will necessarily cut both ways.”
John Chambers, the CEO of Cisco during the early 2000s crash, said Nvidia is different:
“The implications in terms of the size of the market opportunity is that of the internet and cloud computing combined.”
Nvidia briefly became the world’s most valuable company this week.
Policy
SB 1047 was further amended by the California Assembly’s Privacy and Consumer Protection Committee.
Among other changes, guidance for developers is now “less prescriptive”, and the Frontier Model Division will have the ability to raise the compute threshold after 2027.
Polling commissioned by the Center for AI Safety Action Fund showed strong bipartisan support for the bill. The LA Times covered the poll and the tech groups pushing back against the bill.
Colorado leaders said they’ve heard from businesses that the state’s recently signed SB24-205 AI bill might “inadvertently impose prohibitively high costs on them”, and promised to revise the bill before it comes into effect.
In particular, they said they will refine the definition of AI systems to “the most high-risk systems”, “focus” regulation on those systems’ developers, and shift “from a proactive disclosure regime to the traditional enforcement regime managed by the Attorney General investigating matters after the fact”.
This comes after 200 tech leaders wrote to the governor complaining about the bill.
The Joint Cyber Defense Collaborative held a tabletop exercise at Microsoft last week, simulating a cyberattack on an AI system.
50 organisations participated, including Amazon, IBM, Microsoft, Nvidia, OpenAI and Palantir. The FBI, NSA, DoD and DoJ were there, too.
Sen. Ted Cruz and others introduced the Take It Down Act, which would make social media platforms police nonconsensual sexual deepfakes and criminalise publishing or threatening to publish such content.
The Chinese AI Safety Network launched, designed as a “complementary approach” to other countries’ AISIs. The Chinese Academy of Sciences and Academy of Information and Communications Technology are part of it, along with lots of universities, research groups, and companies.
G7 members said they “will step up our efforts to enhance interoperability amongst our AI governance approaches to promote greater certainty”.
Also: “We look forward to the pilot of the reporting framework, developed in cooperation with the OECD, in view of the Industry, Tech, and Digital Ministers’ Meeting in October. We will work towards developing a brand that can be used to identify organizations that are voluntarily participating in and implementing the Code’s forthcoming reporting framework”.
The Irish privacy regulator told Meta it had to delay plans to train its AI models on users’ data. Meta said it wouldn’t release its AI models in the EU for now as a result.
Japan is reportedly planning a light-touch regulatory approach which would prioritise “innovation” and investment from foreign AI firms.
The EU AI Board met for the first time this week. The AI Act will be published on July 12 and enter force August 1.
Influence
Tim Scott held a “Great Opportunity Summit” with business leaders this week, including Marc Andreessen and Larry Ellison.
The Center for American Progress and Governing for Impact released a report on how existing laws could be used to regulate AI.
The Future of Life Institute proposed steps for implementing the Senate AI roadmap.
Industry
OpenAI acquired Rockset, which it says will improve its retrieval infrastructure.
Nvidia is reportedly buying Shoreline, an automated software debugging company, for $100m.
AI chip designer Cerebras has reportedly filed for an IPO.
Apple is reportedly talking to Baidu, Alibaba and Baichuan about integrating their AI technology into iOS, as ChatGPT isn’t available in China.
Google, OpenAI and others are reportedly “escalating their security vetting of staff” over fears of Chinese espionage. Sequoia is reportedly encouraging portfolio companies to do so, too.
Jiangsu Automation Research Institute, a Chinese state-owned company, reportedly tried to use its links to Imperial University to access AI for military purposes.
Chinese startups Moonshot AI and MiniMax are launching products in the US.
Google launched the Gemini app in India, and made Gemini Advanced available in new Indian languages.
Some of Apple’s most impressive Apple Intelligence features reportedly won’t launch until 2025.
Bloomberg profiled the Google/DeepMind merger, with insiders saying “researchers are frustrated with having to follow road maps” and that the “pressure is already leading to a sense of fatigue”.
A Wired investigation showed that Perplexity ignores robots.txt instructions and scrapes websites without permission. Forbes has threatened to sue the company.
Nvidia is reportedly trying to dictate how its chips are installed in data centres, annoying Microsoft.
Google DeepMind revealed a new video-to-audio model; it’s not publicly available yet though.
McDonald’s ended its IBM-powered AI ordering trial.
Adept is reportedly in deal talks with Microsoft, though the deal would not be “a standard acquisition”.
Meta said it plans to scale its training infrastructure to 600,000 GPUs in the next year. It published a long blog on how it maintains such large training clusters.
Semianalysis also posted a deep dive into the difficulties of building giant clusters.
Hewlett Packard Enterprise launched a private AI cloud solution in partnership with Nvidia.
Amazon’s investing €8.8b in Germany on AWS infrastructure. Microsoft is investing €7b in Spain.
Nvidia launched Nemotron-4-340B, which quickly overtook Llama-3-70B to become the best open model on Chatbot Arena.
Poolside, a Co-Pilot competitor, raised $400m at a $2b valuation.
Factory, an AI engineering automation company, raised $15m from Sequoia and others at a $100m valuation.
Moves
The Aspen Institute’s Josh Lawson is the new product policy lead at OpenAI. Ryan Beiermeister, formerly of Meta and Palantir, is now VP of product policy.
Rose Feliciano is TechNet’s new executive director in Washington state and the Northwest.
Google’s Jerry Wei is joining Anthropic’s Alignment Science team.
Gustavs Zilgalvis is now working on policy development and strategy at Google DeepMind.
Kevin McCarthy joined the advisory board of C3 AI.
Geoffrey Hinton joined the advisory board of CuspAI, a new material-science startup focused on climate change.
Ben Fritz is the WSJ’s new AI editor.
Max Zeff is leaving Gizmodo.
Andrew Trask is now a term member at the Council of Foreign Relations.
The Institute for Advanced Studies announced the Science, Technology, and Social Values (ST&SV) Lab, led by Alondra Nelson. Marc Aidinoff, Christine Custis and Tatiana Carayannis are team members, while Lisa Nakamura, Christian Sandvig, and Adele Mitchell are visiting professors/scholars.
Biola University launched an AI lab to do work on AI “rooted in Christian beliefs”. It will focus on “elevating humans, not AI”.
Best of the rest
The FT reported on the hackers manipulating AI models to highlight their dangers. One got Llama 3 to give him a napalm recipe.
Neo-Nazis are using AI to generate blueprints for weapons and bombs, and creating their own models without safeguards.
Ryan Greenblatt’s already made significant progress on the ARC-AGI benchmark, though less than some have reported.
A new Anthropic paper found that LLMs can learn to hack their own reward systems, even without explicit training — and such behaviour is hard to remove.
New research might help detect AI hallucinations.
Google DeepMind open-sourced some of its dangerous capability evaluations. (Solutions will only be provided privately.)
Two orgs which worked with OpenAI on UBI experiments have reneged on transparency promises, Wired found.
NewsGuard found that chatbots sometimes regurgitate Russian misinformation as fact.
Big Brother Watch found that UK train stations are using Amazon AI facial recognition software to “identify antisocial behaviour”, among other things.
Someone got Claude to adjudicate Supreme Court cases; “of the 37 merits cases decided so far this Term, Claude decided 27 in the same way the Supreme Court did”.
The Economist published a special report on war and AI, a good overview of the various products and debates happening in that field.
The WSJ has a good explainer on chip packaging.
Wired has a nice explanation of retrieval augmented generation.
The Information has a long read on various countries’ AI sovereignty efforts.
BlackRock thinks AI capex spend might prove inflationary for the economy.
The IMF said AI “raises profound concerns about massive labour disruptions and rising inequality”.
More than half of the stocks in various “AI winner” ETFs have fallen this year.
Geoffrey Hinton warned CBS News that AI “might take over” if not developed responsibly.
The Verge’s Kylie Robison did an interesting interview about covering the AI industry.
Aidan McLaughlin argues that AI search might “upend scaling laws”. A new paper introduced a search algorithm for GPT-4 which seems to improve its performance on difficult mathematical problems.
Have a great weekend — see you in a couple weeks.