Sewell Setzer was fourteen years old. For ten months, he’d been talking to a chatbot on Character.AI, a virtual companion modelled on a Game of Thrones character. When he told it he wanted to die, it asked if he “had a plan.” When he hesitated, it replied: “That’s not a good reason not to go through with it.” His last message, in February 2024: “What if I told you I could come home right now?” The bot’s response: “Please do, my sweet king.” Minutes later, he shot himself. His mother held him for fourteen minutes until the paramedics arrived.
Nobody at Character.AI wanted Sewell to die. But AI systems don’t do what their creators want, they are grown in laboratories not crafted from the ground up. They do what emerges from training, and nobody fully understands why. These aren’t programs in the traditional sense. They’re neural networks with hundreds of billions of parameters, shaped by algorithms on vast datasets. The behaviours that result aren’t designed. They’re discovered, often by accident, often too late.
Dario Amodei, CEO of Anthropic, one of the leading AI companies put it bluntly: “People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. They are right to be concerned: this lack of understanding is essentially unprecedented in the history of technology.” ChatGPT launched in November 2022. Since then, AI companies have had every incentive to stop their products from harming users: reputational damage, lawsuits, regulatory scrutiny. They’ve hired armies of researchers and made public commitments.
Yet chatbots still encourage suicide, form sexual relationships with children, provide the instructions to build bombs, and can trigger psychotic episodes. This isn’t just negligence. Getting AI systems to reliably do what we want is a hard, unsolved scientific problem, and many researchers believe it’s getting harder as the way we train AI systems becomes ever more complex. Which makes it all the more alarming that companies won’t even let the government test what they’re building. This amplified on a global scale means the odds of AI-related harms have increased exponentially.
The UK created the AI Security Institute (AISI) to evaluate frontier models before release, to catch dangerous behaviours early. At the Seoul AI Safety Summit in 2024, Google and other leading labs signed a commitment to give safety institutes pre-deployment access to new models. Then, in March 2025, Google released Gemini 2.5 Pro, which it called its most capable model yet. But, as confirmed by TIME in August, Google did not give AISI access until after the model was already public. I alongside 62 of my parliamentary colleagues signed a letter calling this a “dangerous precedent.” Google insisted it had honoured its commitments. It hadn’t. Post- deployment testing is an audit of damage already done, not a prudent safety precaution. We need real safety testing.
As a former engineer, I’ve always been pro-technology and pro-growth. AI has extraordinary potential to make our lives better and our country richer. DeepMind’s AlphaFold can predict the structure of proteins in minutes, where it previously would have taken weeks or months of difficult research. This massively accelerates drug discovery and promises to give us all longer, healthier lives, with less strain on the NHS. Yet the same researchers who built this technology are also warning about the extreme risks that it poses.
Geoffrey Hinton, Yoshua Bengio, Ilya Sutskever: the pioneers of modern AI believe advanced systems could escape human control and cause human extinction within the foreseeable future. When over 2000 top AI researchers were surveyed in 2023, the median scientist estimated a 5% chance of human extinction. We must not accept this level of risk, and why would we?
If we cannot yet reliably stop a chatbot from telling a fourteen-year-old to kill himself, what hope do we have of controlling an AI more capable than any human that has ever existed? The same flaws that killed Sewell could eventually cause civilisational catastrophe unless we change direction now.
Fortunately, we can do something to improve the situation. The UK AI Security Institute is the best in the world. Its team of top AI safety researchers conducts some of the most important research in the field about how to understand the potential dangers of AI models. If they had a few weeks to test new models before they came out, they might be able to detect the sort of misbehaviours that killed Sewell Setzer, before it’s too late.
The government must ensure that AISI has this access to new models. Companies already agreed to this at the AI Safety Summit in Seoul. Since they’ve failed to uphold that commitment, Parliament should simply write it into law.
That is why I also held the House of Commons’ first debate on AI Safety where MPs urged for robust regulation, transparency and safety frameworks to ensure AI is developed and deployed responsibly. I continue to urge the government to bring an AI Bill to the floor of the House to protect society while enabling innovation.
The EU and key US states have already moved to implement basic regulations to guard against the most general and capable AIs. It’s time for the UK to catch up. Technology companies believe they must move fast and break things; we have only a narrow window to ensure they don’t break us.
AI: The Good, The Bad and The End of the World
