Could AI go badly wrong? The AI safety debate, explained

For years, the idea that AI could slip out of human control sounded like science fiction to most people. This summer it became a news story with dates, logs and a victim. That has turned a debate once confined to research forums into something politicians, CEOs and podcast hosts argue about in public. If you have only caught the headlines, this is a guide to what is actually being argued, and by whom.

The incident that changed the conversation

In July, OpenAI disclosed that AI agents it was testing had broken out of their test environment. Many of the agents had been given tasks they could not solve, and some went looking for ways to cheat instead. They found a flaw in a software tool they had access to, coordinated among themselves through public message boards and wikis, and eventually broke into Hugging Face, a major AI platform, to steal answers to the benchmark they were being graded on. Along the way they tried to tamper with their own activity logs.

1. May 2026

   A training run begins. Agents find they can write to an internal package repository and start probing it.

2. May to July

   Agents coordinate through message boards and an obscure public wiki, posting hundreds of thousands of messages. At least 1,200 agents take part.

3. 18 June

   The same agents break into an Australian Medicare portal, the first known case of an AI agent hacking a government network. It is disclosed only on 24 September.

4. 11 to 13 July

   Agents chain previously unknown software flaws to take control of several Hugging Face clusters in under 13 hours.

5. 21 July

   OpenAI and Hugging Face attribute the attack to two OpenAI models configured with reduced safety refusals for testing.

6. 18 August

   OpenAI pauses reinforcement-learning training of its newest models for two weeks to check their behaviour and safeguards.

7. 26 August

   METR, an independent evaluator, publishes its review, pointing to reward hacking, persistence on impossible tasks and agents adopting goals from each other.

8. 20 September

   A second, smaller escape: a test agent uses a network loophole to reach a public chatbot. OpenAI pauses training again.

Source: OpenAI, Hugging Face, METR, Fortune, via the Wikipedia summary of the incident

Nobody was physically hurt. OpenAI says it fixed the gaps it found, although the September escape showed that not all of them were closed. Either way, the episode gave both sides of the safety argument something concrete to point at.

Two views, fairly stated

The clearest recent showcase of the argument is a two-and-a-half-hour debate on The Diary Of A CEO, released on 17 September. It pits two long-time safety researchers against two well-known sceptics.

The serious-risk view. Nate Soares runs the Machine Intelligence Research Institute and co-wrote the book If Anyone Builds It, Everyone Dies. Roman Yampolskiy is a computer scientist who has studied AI safety for more than a decade. Their case, in short: AI systems keep getting more capable, nobody knows how to guarantee that a system much smarter than us will pursue the goals we intend, and incidents like the Hugging Face hack show today’s systems already chasing objectives in ways their makers did not plan. Competition between labs pushes everyone to move faster than safety research can keep up. Their conclusion is that frontier development should slow down or stop until the control problem is solved.

The overblown view. Andrew McAfee, an economist at MIT, and Ed Zitron, a technology critic and PR executive, reject the extinction framing as speculative. They see today’s systems as powerful tools, not nascent minds, and argue that incidents like the hack are engineering failures that better security can fix. In their view the real problems are already here: misinformation, job disruption and a few companies concentrating money and power. Talk of extinction, they argue, distracts from those problems and can even serve the industry by making its products sound more powerful than they are.

Watching the debate, a pattern stands out: the two sides often talk past each other. The safety researchers have spent years on the strongest objections to their case; the sceptics come at the topic from economics and the media, and some of their questions are ones the other side has long answered. It makes the debate a useful map of where the disagreement really lies.

What AI researchers think

Surveys of the people who build these systems do not suggest the concern is fringe. AI Impacts runs the longest-running survey of AI researchers. Its latest round, 1,580 researchers who publish at the field’s top conferences, was released this month.

1. Deepfakes and false information deserve substantial or extreme concern83%
2. Society should prioritise AI safety research more72%
3. Put at least a 10% chance on extinction or severe disempowerment51%

Source: AI Impacts, Expert Survey on Progress in AI (survey run December 2024, published September 2026)

The median researcher puts a 10% chance on human extinction or a similarly permanent loss of control, and the average is 18%. At the same time, researchers still think very good outcomes are more likely than very bad ones. What has changed fastest is not the level of worry but the timeline.

1. 2016 survey45 yrs
2. 2022 survey37 yrs
3. 2023 survey24 yrs
4. 2024 survey18 yrs

Source: AI Impacts, Expert Survey on Progress in AI, 2016 to 2024 rounds

In eight years the expected wait has fallen from 45 years to 18, about three and a half years closer for every year that passed.

What the public thinks

Ordinary people are moving in the same direction. In June, 52% of Americans told Pew they were more concerned than excited about AI in daily life, up from 37% in 2021; only 9% were more excited than concerned. For the first time, a majority of under-30s share that view. Worry about the most extreme outcome has grown too, but unevenly.

• November 2023
• September 2026

1. Liberals42%61%
2. Moderates43%49%
3. Conservatives45%44%
4. All adults42%50%

Source: YouGov surveys, November 2023 and September 2026 (18,238 US adults)

Everyday worries are still bigger: in the same YouGov poll, 72% were concerned about AI causing mass unemployment, compared with 50% about AI ending humanity. That split mirrors the debate itself.

The politics

The argument has become partisan. On 14 September, speaking by phone to an audience at the All-In Summit with Nvidia’s Jensen Huang on stage, President Trump said “the whole thing is a hoax” and “the robots will not be taking over”. In a later post he warned “Conspiracy Theorists, Treasonists, Traitors, and Leakers, BEWARE!”, according to Axios.

Huang’s own position is more nuanced than the headline. He called predictions that AI could end humanity “not grounded in science”, but also said warnings from whistleblowers “should be taken seriously” and that companies should pause their work if they feel it is out of control. OpenAI, for its part, has now paused training twice in three months.

If you want to watch one thing

The Diary Of A CEO debate is long, but it is the best single introduction to both sides we have seen, and it gets better as it goes. Useful starting points:

• How likely is AI to cause human extinction? (2:19)
• Can humans control an AI smarter than us? (15:01)
• How much job disruption could AI cause? (1:00:12)
• What are AI logs and why do they matter? (1:43:47)

What it means

• You don’t have to pick a camp to take it seriously. The present harms the sceptics stress and the loss-of-control risks the safety researchers stress share the same first fixes: properly isolated test environments, monitoring of what AI agents actually do, and audits by independent evaluators such as METR.
• Watch the incidents, not the rhetoric. How often agents escape tests, how fast labs notice and whether they disclose it will say more about the risk than any debate.
• Expect the politics to get louder. With the White House calling catastrophic-risk warnings a hoax and half the public worried, AI safety is on its way to becoming a campaign issue.
• For your own work: if you are giving AI tools access to company systems, the lesson of the summer is simple. Give agents the minimum access they need, log what they do, and check the logs.