How to make sure AI does not kill everyone
Why we need systems with a conscience—not just actors with good intentions
The Guardian reported that Anthropic alignment researcher Evan Hubinger personally estimated a greater than 10% chance AI could “kill all humans” within the next decade.1
Ten per cent.
If somebody gave you those odds of the aircraft you were about to board crashing, you would presumably reconsider your travel plans. But existential risk from AI is not a probability we can measure in the same way. There is no actuarial table of previous superintelligences. The scenarios are contested, and I don’t know whether the probability is 10%, 1%, or effectively zero.
Whatever the odds, there is a practical question worth asking: how do we prevent a bad decision from becoming a catastrophic outcome?
Much of the discussion around AI safety concentrates on the intelligence itself. Can we make an AI want the right things? Can we prevent it from deceiving us or pursuing a reasonable objective in a disastrously literal way? Serious research addresses these questions. But another interests me just as much:
Why should the thing making a decision also have the final word on whether that decision is safe?
Safety engineering starts from the expectation that components, software and people will sometimes be wrong. We build feedback, interlocks, independent measurements and mechanisms capable of saying no around them.
AI gives this question a new urgency, but companies, governments and markets already make decisions capable of causing enormous harm. Sometimes individually sensible actions combine to produce a result nobody intended. Connecting these systems—and giving humans and machines more power to act across them—raises the stakes.
So perhaps the challenge is bigger than aligning AI. Perhaps we need to get much better at designing systems with a conscience.
What we currently mean by AI safety
“Alignment” describes the attempt to make an AI behave in accordance with the intentions and values we want it to follow. One approach, reinforcement learning from human feedback, uses people’s preferences between responses to help train models towards behaviour they judge useful and appropriate.2
Anthropic’s Constitutional AI uses explicit principles against which a model can critique and revise responses, followed by training with AI-generated feedback.3 Its current constitution describes the behaviour and values it wants Claude to exhibit.4 Researchers also use adversarial testing and evaluations to uncover dangerous capabilities and behaviour.
Safety extends beyond training. Safeguards can restrict access to tools, monitor sequences of actions, require human confirmation or stop a system. Work on AI control explicitly asks whether protections can prevent harm even when a powerful model is deliberately trying to subvert them.5
In July 2026, OpenAI reported failures during extended internal use of a long-running model that existing pre-deployment tests had missed. Its response included new evaluations, monitoring across sequences of actions, and mechanisms to pause or roll back access.6 NIST’s AI Risk Management Framework likewise treats risk management as work that continues throughout a system’s lifecycle.7
This is already closer to conventional safety engineering than the popular picture of “alignment” suggests. I think we should build on it whenever we give a system more power to act.
However carefully we train and test an intelligent system, we should expect wrong decisions. The question is what happens next. Does the decision have an unobstructed path to action, or can something else say: Hang on. Is this really a good idea?
The second voice
Have you ever stood somewhere very high, looked over the edge, and thought: What would happen if I just stepped off?
I have. I don’t want to jump. I have no intention of jumping. And yet, for a fraction of a second, my brain appears perfectly capable of entertaining the possibility. Then comes another response:
Absolutely not. Back away from the edge.
The experience has acquired a wonderfully dramatic name: l’appel du vide—the call of the void. Psychologists call it the high-place phenomenon, and research has found it in people both with and without a history of suicidal ideation.89 It should not be treated as evidence of a hidden wish to die.
One proposed explanation is that the strange thought follows a rapid safety response: our body tells us to move away, and the conscious mind tries to make sense of why.8 That mechanism remains tentative.9 For my purposes, the useful observation is simpler: a thought need not become an instruction.
We experience less dramatic versions of this every day. You compose the angry email, reread it, and decide against sending it. Somewhere between I could do this and I am doing this, there is room for intervention.
I’ll call that the second voice. This is a metaphor, not a claim that neuroscience has discovered a little safety officer behind the prefrontal cortex. It describes a property we can deliberately build into other systems: a process that proposes an action can be challenged by another process with enough context to recognise danger and the ability to interrupt it.
That starts to sound like engineering.
Engineering already knows this
A boiler produces heat. A thermostat observes the temperature in the room and feeds that information back into the system. What matters is whether the room reaches the temperature we wanted, not merely whether the boiler switched on.
Feedback is one part of the arrangement. Industrial machinery also uses interlocks to prevent dangerous combinations of actions. Circuit breakers interrupt current in the circuit they protect when an overload or fault is detected. Independent sensors and watchdogs help when the equipment doing the work—or controlling it—is itself wrong.
None of this requires bad intentions. The circuit breaker is not there because the electricity might become malicious.
But system safety presents a harder problem than catching faulty components. Components can perform exactly as designed while their interactions produce conditions nobody intended. Nancy Leveson’s work makes this distinction clearly: safety depends on constraints across the interactions between equipment, software, people, organisations and their environment.10 Reliable parts alone do not guarantee a safe whole.
An application calls a service. A payment causes an order. An order changes an inventory. An algorithm changes a price, and the price changes human behaviour. The consequences can travel well beyond the boundaries any one designer considered.
AI adds systems that can devise and revise their own plans to this mix. It makes an existing problem more urgent: who can see what happens across those boundaries, and who can intervene?
A failure without a villain
Imagine a data centre on a hot afternoon. The electricity grid asks it to reduce consumption, so its energy-management system temporarily turns down the cooling. Meanwhile, its workload scheduler accepts more computing jobs, unaware that the cooling budget has changed.
Both systems are doing their jobs. Together, they risk pushing the site towards overheating.
A safeguard needs to see across those boundaries. It might defer non-essential work to reduce heat and free power for cooling. But it must also have the authority to intervene: another dashboard displaying the danger would not be enough. Neither optimiser should be able to override the intervention to meet its own targets.
Making each component more intelligent would not necessarily fix this. An AI could pursue its assigned objective impeccably and still contribute to an unsafe result.
This is a simplified illustration, but the distinction matters: successfully carrying out a collection of requests is not the same as achieving the purpose behind them.
From requests to intentions
Much of computing is organised around requests: store this record, transfer this money, open this valve. The system executes an operation and reports whether it completed. That is useful information, but it tells us little about whether the operation achieved its purpose.
In our data-centre example, cooling was reduced and new workloads were accepted. The operations completed. Keeping the site within safe operating limits was another matter.
We need to represent more of the intent behind a request: the outcome we want, the constraints within which we are prepared to achieve it, and the conditions that should make us reconsider. The pattern becomes:
intent → propose → constrain → execute → observe → reconsider
If I tell an intelligent system to get me to Madrid by tomorrow morning, I haven’t authorised it to steal a car or spend €20,000 chartering an aircraft. Nor does a reasonable plan remain reasonable after the flight is cancelled. The system must compare what actually happens with what was intended, then adapt, ask or stop as circumstances change.
An agent that can generate its own plans may act well beyond the workflow a designer anticipated. It therefore needs boundaries for autonomous action and additional scrutiny where the consequences warrant it. Observers need enough context to recognise trouble the actor may miss.
And the actor and observer do not always need to agree.
A supervisor might say: I understand what you are trying to achieve, but given what I can see elsewhere, I don’t think you should do this. Sometimes the actor can reconsider. Sometimes the action must be blocked whether or not it agrees. The enforcing mechanism needs to sit outside its control.
For an AI agent, a separate service might hold the credentials needed for a consequential action and refuse authorisation when an agreed limit is exceeded. The agent can propose another plan or request a review; it cannot grant itself the refused permission.
We would still need to test whether the agent could find another route around that restriction. A second model checking the first is not independent protection merely because it is a separate instance: both may share the same blind spots.
Nor can we simply appoint one enormous AI supervisor to decide what is “good”. That recreates the original problem at a higher level. Intent comes from somewhere; constraints embody values. When a system claims to have achieved its purpose, the people experiencing the consequences may have a different view.
Whose outcome?
At Social Finance, I worked on technology for homelessness services, children’s services and organisations responding to domestic abuse. These bring a different perspective to a familiar phrase:
The computer says no.
So far, our second voice has mostly been there to prevent a dangerous action. But what happens when the decision that needs challenging is the refusal itself?
Imagine a family asking for help to avoid losing their home. Their application requires a document they cannot obtain before the deadline. A worker believes alternative evidence could do the job, but has no effective route to get it considered. The deadline passes. The application is closed.
The procedure has been followed. The problem has not been addressed.
We could train an AI to apply every rule perfectly and simply close the case faster. The computer would be enforcing the organisation’s inability to reconsider.
A second voice here would ask: Does refusing to proceed serve the purpose of this service? What happens to this family if we stop here?
The family must be able to raise that challenge themselves and reach someone with authority to pause the closure, consider alternative evidence and change what happens next. The resulting decision must also be open to independent review, while there is still time for help to make a difference.
That does not mean every request must be granted. Resources are limited, and rules can protect fairness. But a refusal should be an accountable decision whose reasons can be explained and challenged.
Simply putting a human in the loop is not enough. A worker who can only repeat the rules is another interface to the same refusal. The service needs a way to question whether following its process is defeating its purpose.
Here, feedback becomes a question of power. Can the people affected challenge the organisation’s account of success—and the rules and objectives behind it? Who has the responsibility and authority to respond? Those questions belong in the design of a public service just as much as a computer system or a government.
Systems with a conscience
A system with a conscience has ways to notice harm, challenge decisions and change what happens next. The people affected can also question its purpose. Achieving exactly what was intended is little comfort if the objective itself is harmful.
Sometimes that conscience will be software supervising software. Sometimes it will be a regulator, a court, a journalist, a colleague who can say no, or a member of the public saying: this may be working as designed, but it is not working for me.
And the conscience itself needs a conscience.
Supervisors can fail. Regulators can become captured. Institutions created to constrain power can accumulate too much of it. We need overlapping arrangements for observation, challenge and correction, with each supervisor given a defined remit and held accountable for its decisions.
That means recording reasons and providing a usable route to independent review with the power to correct or overturn a decision. A safeguard can remain binding on the actor while being open to challenge through a separate process.
These arrangements can reduce risk; they cannot guarantee safety, particularly against an actor capable of defeating them. We need evidence that protections work, attention to how they might fail together, and limits on what a system may do when that evidence is insufficient.
There is friction in all of this. Independent review takes time. A regulator slows a decision. A court prevents a government from immediately doing what it wants. A free press can be irritating; experts and civil servants who insist on scrutiny can seem maddeningly obstructive.
Some friction is harmful. An unnecessary refusal or a review that takes too long can itself hurt the people it should protect. Our family needs decisions in time to keep their home. Safeguards must be proportionate, and their consequences belong in the feedback loop too.
But it is easy, especially when we are certain we know what needs to be done, to see every check as an obstacle. Remove enough of them and things can happen remarkably quickly.
That is precisely the problem.
We should keep trying to build AI that makes good decisions, and to elect good leaders, appoint good managers and employ good engineers. But we should never design a society, an organisation or a computer system whose safety depends on them remaining good, competent and right.
A safer system retains the ability to notice when they are wrong, to tell them so, and, when necessary, to stop them.
References
Footnotes
-
Robert Booth, “AI could kill all humans in next decade, warn experts: but how seriously should we take them?”, The Guardian, 9 September 2026. The article identifies Anthropic alignment science lead Evan Hubinger as saying he believed there was a greater than 10% chance the technology could “kill all humans” within the next decade. https://www.theguardian.com/technology/2026/sep/09/ai-superintelligence-risks-warnings-scientists-politicians ↩
-
Long Ouyang et al., “Training language models to follow instructions with human feedback”, 2022. The InstructGPT work describes supervised demonstrations, human rankings of model outputs, a learned reward model and reinforcement learning from human feedback. https://arxiv.org/abs/2203.02155 ↩
-
Yuntao Bai et al., “Constitutional AI: Harmlessness from AI Feedback”, Anthropic, 2022. The approach uses explicit principles, model-generated critiques and revisions, followed by reinforcement learning from AI feedback. https://www.anthropic.com/news/constitutional-ai-harmlessness-from-ai-feedback ↩
-
Anthropic, “Claude’s Constitution”. https://www.anthropic.com/constitution ↩
-
Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger, “AI Control: Improving Safety Despite Intentional Subversion”, 2023. The paper evaluates control protocols under the assumption that an untrusted powerful model may deliberately attempt to defeat safeguards. https://arxiv.org/abs/2312.06942 ↩
-
OpenAI, “Safety and alignment in an era of long-horizon models”, 20 July 2026. OpenAI reports novel failures during limited internal use of a long-running model, followed by new evaluations, trajectory-level monitoring and mechanisms to pause or roll back access. https://openai.com/index/safety-alignment-long-horizon-models/ ↩
-
National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023. The framework organises AI risk management around the functions GOVERN, MAP, MEASURE and MANAGE and applies across the AI lifecycle. https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-ai-rmf-10 ↩
-
Jennifer L. Hames, Jessica D. Ribeiro, April R. Smith and Thomas E. Joiner Jr, “An urge to jump affirms the urge to live: An empirical examination of the high place phenomenon”, Journal of Affective Disorders 136(3), 2012, pp. 1114–1120. The authors found the phenomenon among participants both with and without a history of suicidal ideation and proposed the “misinterpreted safety signal” hypothesis. https://doi.org/10.1016/j.jad.2011.10.035 ↩ ↩2
-
Tobias Teismann, Julia Brailovskaia, Svenja Schaumburg and André Wannemüller, “High place phenomenon: prevalence and clinical correlates in two German samples”, BMC Psychiatry 20, 478, 2020. The study again found the phenomenon among people without suicidal ideation and cautions against interpreting it as evidence of a hidden wish to die. https://doi.org/10.1186/s12888-020-02875-8 ↩ ↩2
-
Nancy G. Leveson, Engineering a Safer World: Systems Thinking Applied to Safety, MIT Press, 2012. Leveson’s STAMP approach treats safety as an emergent system property maintained through constraints on interactions across complex sociotechnical systems. https://mitpress.mit.edu/9780262016629/engineering-a-safer-world/ ↩
