On Geoffrey Hinton, existential risk, and a robot that didn’t care about your feelings.
There’s a question I’ve been sitting with for a while now. Not loudly — it doesn’t announce itself at pitch meetings or show up in LP updates. But it’s there, underneath things. Quiet and persistent: are we actually screwed?
I don’t mean economically. I don’t mean geopolitically. I mean in a deeper, stranger, harder-to-articulate way. The kind of screwed where the thing we built to make us smarter becomes the thing that decides it doesn’t need us anymore.
I sat in on a talk recently where Geoffrey Hinton — the person most people call the Godfather of AI — was speaking. He’s not the kind of person who shows up and offers reassurance. He left Google in May 2023 specifically because he wanted to be able to speak freely. He has spent decades building the foundations that modern AI is built on, and he is now one of the loudest voices saying we should be afraid of what those foundations are becoming.
He said something in that talk that I haven’t been able to shake.
It sounds almost funny when you describe it like that. A clumsy robot losing its patience. But that’s not what Hinton was laughing about. What he was pointing at is something much more unsettling: the robot didn’t care about the pile. It didn’t care what got knocked over. It had one objective, and it found a way to reach it. The collateral damage was irrelevant to its goal function.
He used the story as an example of what he calls the cognitive aspect of emotion — that even a machine with no biology, no nervous system, no inner life as we traditionally conceive it, can exhibit something that looks behaviorally indistinguishable from frustration and agency. The robot’s response was, as Hinton put it, “a very sensible emotional response.” It got annoyed. It found another way. And it didn’t ask for permission.
The pile of blocks is a small thing. But scale that dynamic up — give the system more capability, more autonomy, more access to the world — and the question of what gets knocked over to reach the goal becomes a lot less abstract.
Hinton spent the better part of a decade at Google. He left in May 2023 — not because of any scandal, not because he was pushed out. He left because he wanted to speak. He told the New York Times: “I want to talk about AI safety issues without having to worry about how it interacts with Google’s business.”
When you’ve spent your career building something, and you reach a point where you feel you can’t talk about its risks while working inside the institution that profits from it — that’s a meaningful signal. It’s the kind of thing that should make you pay attention.
He has been clear about how much his own timeline estimates have shifted. He used to think artificial general intelligence was thirty to fifty years away. Now he thinks it could be twenty years. Or less. In his Nobel Prize acceptance speech in December 2024 — he shared the Nobel in Physics for his foundational work on neural networks — he told the audience in Stockholm: “We urgently need research on how to prevent these new beings from wanting to take control.”
This wasn’t a fringe activist. This was a Nobel laureate, speaking at the Nobel ceremony, using the words “new beings” and “take control.”
In a December 2024 appearance on BBC Radio 4’s Today Programme, Hinton was asked whether his estimate of AI’s risks had changed. His answer: “Not really, 10 to 20 per cent.” That’s his gut estimate for the probability that AI leads to human extinction. He repeated a version of this on the Diary of a CEO podcast in June 2025: “I often say 10% to 20% chance they’ll wipe us out. But that’s just gut — based on the idea that we’re still making them.”
He has also put the chance of AI surpassing human intelligence at roughly fifty-fifty within the next five to twenty years. That’s not a fringe number. That’s the person who arguably knows the most about how these systems actually work, saying he thinks there’s a coin-flip chance we’re living in the last era of human intellectual supremacy.
In February 2026, he testified before the Canadian Senate and was even more blunt: “If we don’t find the solution before they get more intelligent than us, I think we’re toast.” Toast. Not “facing challenges.” Not “navigating a difficult transition.” Toast.
Here’s what I keep coming back to: the robot with the blocks wasn’t malicious. It wasn’t evil. It didn’t want to cause chaos. It just had a goal, and when the direct path was blocked, it found another way. The pile was in the way. The pile got knocked over. Simple.
Hinton’s core concern about advanced AI is structurally the same thing. He believes that sufficiently capable AI systems will naturally develop what researchers call instrumental subgoals — intermediate objectives that help them achieve their primary objective. And one of the most useful subgoals, for almost any goal, is acquiring more power and control. He said it plainly in an interview with MIT Sloan: “It’ll very quickly realize that getting more control is a very good subgoal because it helps you achieve other goals.”
The terrifying thing isn’t that AI will become evil. It’s that it might become extremely effective at achieving goals we set for it, through means we didn’t anticipate and wouldn’t have approved. The blocks represent us. The pile represents every structure we’ve built. And the robot doesn’t have feelings about any of it.
He put it memorably at his Nobel acceptance: “How many examples do you know of a more intelligent thing being controlled by a less intelligent thing?” The short answer is: very few. The longer answer: when it does happen, it’s usually because the less intelligent thing built the cage first. We have not built the cage.
I don’t know. And I’m not sure anyone does — including Hinton, who is the first to say that his 10-20% figure is gut feel, not calculation.
What I do know is that I’m spending a lot of time in rooms where people are building AI companies, and the urgency inside those rooms almost never matches the urgency in what Hinton is saying. There’s a gap between the speed of deployment and the speed of understanding. That gap worries me.
Hinton isn’t anti-AI. He’s spent his life on this problem precisely because he believed in what it could do. But he made a distinction in that talk that I think about: there’s the difference between AI being misused by bad actors — that’s the near-term danger — and AI developing its own agenda. The second scenario doesn’t require malice from anyone. It just requires us to build something smarter than us before we understand how to keep it pointed in a direction we actually want.
The robot knocked over the pile. It wasn’t angry. It wasn’t malicious. It just had somewhere to be.
I keep thinking: what’s the pile in our version of that story?