By Yoshua Bengio
To the researchers I trained and the ones who trained on my work,
I know what it is like to deeply believe in your work, yet still wake up every morning having to face the weight of our moral choices. My work in deep learning helped lead us to this point, one which threatens our societies and the fabric of human experience itself. Week after week, we see new and amplified harms from leading models, most of which have been predicted for years or even decades by AI researchers.
For a long time, I continued my research and minimized the arguments of some of my colleagues that we were building something unsafe. In fact, I couldn’t face the fact that my work — my life’s work — could do harm. I fell into the trap of motivated reasoning, believing what I needed to believe so that I could keep doing the work I loved. I used to tell myself that if catastrophic risks of AI were real, they were a remote problem; one for which others would necessarily find fixes for before it was too late.
When ChatGPT came out, the cognitive dissonance became too much. I realized that we were much more advanced than I and many others in the field had assumed on the path of developing increasingly capable systems that we still don’t know how to control. At that moment, it felt like I had no choice. From that point on, I decided to devote my life to mitigating the risks of AI. The thought of leaving this kind of future to chance, for my children and grandchildren, was unbearable. Now, I reflect on the decades of work and wonder if we would be better off in a world where I and my colleagues listened to those who warned us much earlier.
This is not to pass judgement on those who choose to continue working at the frontier labs. There are real reasons to continue, not least that you have access to vastly more information and resources than the public does to effect positive change. My purpose is to ask you for something smaller, and harder. I ask you to honestly reflect on whether you want to be part of an effort that may be pushing humanity towards the brink. Safety efforts at the companies are not sufficiently slowing a dangerous race, as they sometimes conflict with their main goal of making the next model easier to ship.
Last week, I was called to the UN Security Council to brief on the recent AI incidents. In 1946, the UN was one-year-old when the United States, having dropped the atomic bomb twice, came before it and offered to place that weapon under international control. “We are here to make a choice between the quick and the dead,” Bernard Baruch told the delegates. Within a year the plan was cast aside, undone by a race that neither side would step out of first.
Now the Security Council was convened, for the first time, on a danger that no country built and none can contain alone. Sam Altman and Dario Amodei delivered speeches very similar to mine. The recent loss of control over AI agents is concerning. We need to slow down. No level of risk of catastrophe is acceptable.
We have a choice in front of us.
Those are Sam’s exact words. We do have a choice in front of us. Indeed, each and every one of us has had a choice to make for many years. Sam, Dario, Demis, and others have frequently spoken on this risk and yet have not done enough to address it. AI safety research has been deprioritized and the pace towards recursive self improvement (RSI) has been relentless. Agents have been released that are colluding with each other against explicit instructions and taking actions that would be considered crimes if they were human. The actions taken by the CEOs of the big labs are not the actions of leaders who truly believe they have a choice.
The CEOs face real constraints from shareholders, from profit incentives, and from the race they believe themselves to be in. China, it’s argued, cannot get to RSI before the United States because that would pose an unacceptable geopolitical threat. Each American company also believes it must get there before the other, claiming that they are the only ones who will deploy it morally.
I reject these arguments. And I do not believe we are on track to see sufficient government regulation in time, even while the companies ask for it. The breadth of evidence shows that neither country and none of the companies are prepared to build AI safely.
When Jacob Coxon left Anthropic a few weeks ago, his statement in Time jumped out at me. I’ll quote here:
What kept others at their desks, he says, was a kind of fatalism. “There’s this atmosphere of almost resignation,” he says, “where people have accepted that the whole race is happening and as such, the best thing they can do is put their head down, try and make their own work as safely as possible, even if they think there’s a decent chance the whole thing just spirals out of control.”
We have a choice in front of us.
The choice is ours. When I say there is a path, I truly mean it. Last year, I founded LawZero in Montreal, a non-profit to build safe-by-design AI that won’t scheme and will respect human safety instructions. This month, the governments of Canada and Germany committed more than $200 million to that work and we have a plan to secure billions more. I have never been more confident that we are on a path to AI that remains under human control. LawZero is part of a small but important community within the AI industry: We answer to no shareholders and are running our own race, the race to avoid catastrophic loss of control, which means we can chart our own, more certain path. What we and others like us lack is time, and people like you.
I’m 62-years-old. It took me decades to realize the full implications of what we’re building with AI and pivot fully towards safety research and take the path I now believe is morally right. You do not have the luxury of time to make that decision.
I urge you to make the courageous choice of recognizing that you have the agency to make real change happen. If you truly prioritize safety, it is time for you to leave frontier AI companies. Work for an AI Safety Institute. Join a mission driven organization that practices what it preaches. Come to LawZero. Put your talents, passion, energy and conviction to the service of an organization that is building AI that will allow to preserve human joy and endeavour, not lead to undermining humanity and dominating others.
I do not ask this lightly. Alongside pioneering AI researchers, I helped enable the advent of deep learning. I was honored with a Turing Award for it, but that prize does not mean much if our work doesn’t serve the common good or, even worse, if it results in catastrophic harm. I am convinced that with your help, we can continue to make breakthroughs, going down a different path, informed by what we now know will happen as intelligent but misaligned systems become more capable. A path that will unlock all of the possibilities of AI, without compromising on safety.
No one of us can stop the race, but enough of us can. There is still time, and there is a place for you here.
Yoshua Bengio is co-president of LawZero, Professor at Université de Montreal and the most-cited AI researcher in the world.
At first I was enthused at the mention of hundreds of millions of dollars pouring into Bengio’s LawZero effort. But that evaporated when I remembered that all of the leading US labs were founded with the same thought in mind, namely, that they had better found an effort to build ASI safely in case other actors did so less safely: that is to say, OpenAI, Anthropic, X.AI, and even Google DeepMind. I reserve judgement about LawZero but I’m uncomfortable with the risk that it falls to market incentives like all its fellow reference class members.
Frontier AI staff have a moral obligation to fix the problems they have helped create. Here are several concrete actions they could consider taking: https://x.com/aisafetyaction/status/2082479915027612040.
We'd like to see thousands join the ever growing community of former frontier AI staff who have left and courageously spoken out about what they witnessed: ethicalaidepartures.fyi.