Existential primacy.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Inference-time compute is a new dimension of the scaling curves we hadn't properly mapped.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“I may have three more years to work.”

Context: Widely-cited Palladium essay about living through short-timeline AGI.

“There's the possibility that AIs will run out of control. Could a machine decide that humans are a threat, conclude that its interests are different from ours, or simply stop caring about us? Possibly, but this problem is no more urgent today than it was before the AI developments of the past few months.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Inside Google, Microsoft, and OpenAI, there is real disagreement about what is actually happening.

Forecasting AI extinction risk under Knightian uncertainty is a different exercise from forecasting under well-defined base rates.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Context: Statement Hendrycks drafted and organised.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“I think it's terrifying. I hate it. I think about it practically all the time, every single day.”

Context: On modern AI, in a 2023 interview.

If minds of infinite subtlety and complexity and emotional depth could be trivialized by a small chip, it would destroy my sense of what humanity is about.

“25th percentile, maybe 2029, and then 75th percentile, like 2050.”

Context: On his personal AGI timeline.

My p(doom) is around 35%.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

The AI safety people spend a lot of time convincing their friends this is serious. I think it is.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Context: Single-sentence Statement on AI Risk published by CAIS; Hinton was listed first among AI scientists.

The scaling hypothesis has held across every order of magnitude we have tested.

“It seems a reasonable prediction that some time in this or the next century intelligence will escape from the constraints of biology.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“The first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

I have yet to meet anyone at an AI lab who says the risk of the next generation model blowing up the planet is less than 1%.

Compute for frontier training runs has doubled roughly every six months since 2010.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

I used to dismiss AI-risk arguments. The past few years of capability progress have substantially shifted my view.

My overall estimate of the probability of existential catastrophe from misaligned AI by 2070 is about 10%.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Taking AI risk seriously and being epistemically calibrated are not in tension.

The median respondent gave a 5% chance of AI causing an outcome as bad as human extinction. Five percent is not a reassuring number.

“AI experts are increasingly afraid of what they're creating.”

Context: Headline framing of her widely-cited Vox piece.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

My p(doom) is about 10%.

Context: Conversation with Sundar Pichai on the Lex Fridman Podcast.

If you actually take the technical alignment problem seriously, our position is dire. Doom Debates exists because the public conversation does not match the technical reality.

I am skeptical AI will surpass human intelligence within decades. But I am not skeptical that we will mishandle whatever AI we have. The pattern of technology is humans repeatedly underestimating their own carelessness.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Since we can't understand what's going on inside them, we have to be cautious about handing over power to them.”

It's unclear what human-level AGI means. The more useful question is when real economic growth rates reach at least 30% worldwide.

“Many AI experts believe there is a real chance that human-level artificial intelligence will be developed within the next decades, and some believe that it will exist much sooner.”

Theory of mind may have spontaneously emerged in large language models.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Intelligence is a much more lethal superpower than nuclear power.

AI is not a slave. It is a form of sentient being that needs to be appealed to rather than controlled.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

My p(doom) is in the 5–10% range. Not trivial, not overwhelming.

The compute required to train a language model to a given level of performance has been halving roughly every 8 months due to algorithmic improvements.

“Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct.”

Context: Closing passages of Superintelligence.

Reasoning models change the safety landscape. Scheming becomes more possible as model planning improves.

We face ten interconnected megathreats including artificial intelligence. Each could be civilization-shaking; together they are existential, and our institutions are not designed to face them as a system.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

AI safety is no longer a fringe concern. The question is whether the institutional response will catch up.

Bio plus AI may be the highest-priority near-term x-risk vector to track empirically.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“We're going to build superintelligence whether we like it or not. The only question is whether we will know what we are doing.”

I think the probability that AI causes a catastrophe is about 33%. That's not the 95% or higher that some people say, but it's also much higher than the probabilities we accept for other risks.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“AI's going to be one of the fundamentally transformative technologies humanity creates, if not the most. We just need to make sure, from a humanity perspective, this goes well.”

AI poses a genuine existential risk. I am not sure how high I would put the probability, but I do not think we are responding to it as if it were a real possibility.

“The development of full artificial intelligence could spell the end of the human race.”

“It would take off on its own, and re-design itself at an ever increasing rate. Humans, who are limited by slow biological evolution, couldn't compete, and would be superseded.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Given current trends in compute and data, transformative AI by 2040 is well within reason.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

We're on a balance beam between two outcomes. Either we get our act together, or we don't. There is no third option once superintelligence arrives.

“Humanity stands at a precipice. Our species could survive for millions of generations, enough time to end disease, poverty, and injustice; to reach new heights of flourishing.”

Context: Opening of The Precipice.

Standard economic growth models predict explosive growth once AI substitutes broadly for human cognition.

“Within thirty years, we will have the technological means to create superhuman intelligence. Shortly thereafter, the human era will be ended.”

We live at an unusual time in history: we have the power to influence the lives of beings who will exist for millions of generations.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

“No one currently knows how to create advanced AI that reliably follows the intent of its developers.”

Context: Written testimony to the US Senate Judiciary Subcommittee on Privacy, Technology and the Law.

“There is a risk of losing control over AI with powerful capabilities, a risk we have yet to learn how to mitigate. If those in control of AI do not understand and manage this risk, it could jeopardize all of humanity.”