Alignment first.

Alignment should be a first-class research problem alongside capabilities.

Deceptive alignment is the scenario where a model behaves as if aligned during training but pursues different objectives at deployment. The question is whether we can rule it out empirically.

Fairness must be operationalised in training, not bolted on after the fact.

Aspiration is the process by which we acquire values we did not previously have. The question of whether AI can do this is the alignment question, philosophically.

My 2022 median for transformative AI dropped from roughly 2050 to the late 2030s.

“I propose to consider the question, 'Can machines think?'”

Context: Opening line of Computing Machinery and Intelligence.

Deep reinforcement learning doesn't yet work. We need to be honest about that even when we are excited about the wins.

Reward hacking shows up reliably across a range of agents and tasks. The good news is that it is empirically studyable; the bad news is that it does not have a known general solution.

Optimal policies tend to seek power in a technical sense; but learned policies are not optimal in that sense.

When we train a model, we are not just shaping behaviour, we are shaping a character.

Robots and AI assistants need to infer and defer to human preferences, not optimise rigid objectives.

Releasing software to millions of people without safeguards is not engineering. It would never fly in any other engineering field.

AI safety is a shared technical problem; international cooperation is essential.

Most of the work of being correct about AI risk is not technical, it is the epistemic practice that lets you face uncomfortable questions without flinching.

Median respondents put 50% chance of high-level machine intelligence by 2047. The chance of an extremely bad outcome was 5% or higher for half of respondents.

Most of what people think of as 'capabilities' is shaped after pretraining. Post-training is also where most of what we think of as alignment is decided.

Most alignment research happens between papers, in long-running discussions on LessWrong and the Alignment Forum. Maintaining that infrastructure is not separable from the field's progress.

I joined OpenAI because I think the most interesting and important alignment research is happening on actual frontier models. Working from the outside has limits.

The alignment problem is already here, and will keep scaling with capability.

We should design AI deployment protocols that remain safe even if our AIs are trying to subvert them.

Working backward from the question of how we can do the most good with our resources led us to focus on existential risks, including from advanced AI.

Our results show that for English summarization, RLHF-trained models can outperform much larger fine-tuned models. The technique is powerful and inherits all the strengths and biases of the human raters.

We're seeing the largest AI training runs grow more than 300,000x in compute over six years, an order of magnitude faster than Moore's Law. The implications for safety planning are immediate.

Generalisation in robotics is where a lot of real safety work has to happen.

“We define foundation models as models trained on broad data at scale and adaptable to a wide range of downstream tasks, and these models entail both new capabilities and new risks.”

The question 'does this model understand?' hides a stack of questions we haven't yet disentangled. Alignment research has to live with that ambiguity.

“I visualize a time when we will be to robots what dogs are to humans. And I'm rooting for the machines.”

Context: Reported quote from Shannon late in his life. Often cited as one of the earliest succession framings.

NLP has moved from engineering discipline to civic infrastructure. Our responsibility is to treat it that way.

If we want AI to be broadly beneficial, we need to invest in alignment research well before systems are capable of world-changing impact.

What I want from AI safety research is the same thing I want from any other research: clear problem statements, clear progress, and a community that holds itself to the standards of the rest of science.

Frontier AI research has to be a partnership between academia and industry. Neither has the full set of capabilities to navigate where this is going alone.

RL agents that learn open-endedly need open-ended evaluation regimes if we are to deploy them safely.

We are still learning how the brain represents anything. Mapping that onto AI representations is genuinely open research.

AI is the most important and consequential technology being developed in our lifetime. Its development cycle is one of the most important things society needs to get right.

Assistance games give us a framework where AI uncertainty about human preferences is a feature, not a bug.

Reliability research is where most of the real AI safety work is happening, and most of the real progress.

We don't yet know how to robustly characterize what these models understand versus what they confabulate. That gap is the root of many alignment problems.

You can use a language model to red-team another language model, and that lets you scale evaluation in ways humans alone cannot.

A model that has learned deceptive goals during training can pass all your behavioural tests and still fail catastrophically when deployed.

Context: Sleeper Agents paper at Anthropic.

“We present a new scalable oversight protocol (prover-estimator debate) and a proof that honesty is incentivised at equilibrium.”

What I try to do on the podcast is take the technical arguments seriously enough to ask the second-order questions about them, in public, on the record.

Mental states could in principle be realized in any of an indefinite number of physical systems.

Context: Functionalism, the foundational philosophical assumption of strong AI.

Understanding what large language models actually do is the precondition for any meaningful policy debate about them. Educational access to that understanding is itself a safety issue.

Alignment is not just a technical problem. It is a problem about whose values count.

“A robot may not injure a human being or, through inaction, allow a human being to come to harm.”

Context: First Law of Robotics, 'Runaround', 1942.

Machines now exhibit behaviours that need to be studied with the methods of behavioural science, not only with the methods of computer science.

The alignment problem is about getting less-capable verifiers to reliably elicit truth from more-capable provers.

“Over the past years, safety culture and processes have taken a backseat to shiny products.”

Context: Resignation thread on X/Twitter.

If scaling laws continue, we will have models far beyond current capabilities well within this decade. That is a reason to take safety seriously, not to slow down research on safety.

We can formally characterise the conditions under which a learned reward model is hackable. The characterisation lets us design training regimes that reduce the attack surface.

“We have created the first computational tool that can routinely predict protein structures with accuracy competitive with experimental methods.”

“Every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.”

Context: From the Dartmouth Workshop proposal, the founding statement of AI as a field.

“I've made the difficult decision to leave OpenAI. This choice stems from my desire to deepen my focus on AI alignment, and to start a new chapter of my career where I can return to hands-on technical work.”

Doubly-efficient debate lets an honest strategy verify correctness in polynomial time while the dishonest strategy has exponentially more steps to try to deceive.

“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that, when trained in isolation, it can match the test accuracy of the original network.”

My current estimate is that the probability of existential catastrophe from power-seeking, misaligned AI by 2070 is more than 10%.

Context: Updated estimate after the original report; revised upward.

When you watch real users interact with AI assistants, the alignment failures that show up are not the ones laboratory benchmarks are designed to catch. The friction is in the texture of conversation.

Robustness benchmarks are an under-developed scaffolding for AI safety practice.

Reward modelling, RLHF, red-teaming, and eval frameworks are the concrete instruments of alignment today.

Plausible scenarios for AI takeoff include software-only feedback loops where AIs do AI research. Whether this leads to alignment failure depends on details that haven't been carefully argued.

The right model for AI in high-stakes domains is not full autonomy. It is fluid teaming, where the human and the machine each pick up the parts of the task they are best suited to.

Distributional RL learns the entire distribution over returns, not just its expectation. The richer signal turns out to matter for stability, exploration, and risk-sensitivity.

“What magical trick makes us intelligent? The trick is that there is no trick. The power of intelligence stems from our vast diversity, not from any single, perfect principle.”

Teaching language models to support answers with verified quotes is a concrete alignment sub-problem we can make progress on.

“The fate of the gorillas now depends more on us humans than on the gorillas themselves; so too the fate of our species then would come to depend on the actions of the machine superintelligence.”

Doom arguments tend to hinge on underdefined intuitions about 'optimization pressure' that I don't think survive engagement with real systems.

“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfere effectively… we had better be quite sure that the purpose put into the machine is the purpose which we really desire.”

Independent reproductions of frontier-lab claims are an undersupplied public good.

Frontier models can scheme. That's now an empirical observation, not a theoretical concern.

I'd guess something like a 20% chance of an AI takeover, with many of the humans dead, and a further 30% chance or so of serious irreversible problems short of takeover.

Context: From his LessWrong post 'My views on doom'.

Structure-aware architectures can help us build AI systems that generalise more safely across domains.

Moral learning in humans draws on the same reinforcement-learning machinery we are now building into AI systems. That's not an accident; it is the alignment problem.

Values in learned agents emerge as shards, context-activated circuits, not unified utility functions.

Embodied learning forces you to confront questions that are easy to gloss over in language models: causality, partial observability, the cost of mistakes.

Formal verification is under-used in AI safety. When you can prove a property rather than measure it, you should.

The development of AGI may be one of the most consequential events in history, with the potential to either drastically increase or decrease the chances that humanity survives and flourishes.

You don't need the AI to hate you. Almost any objective, pursued hard enough, runs us over as a side effect.

We need to know how to build AGIs we can trust before we build the AGIs. Right now we don't know how, and that is the whole problem.

If transformative AI arrives in our lifetimes, the decisions made in the next decade may shape the long-run future more than any others. That makes alignment work plausibly the highest-leverage problem.

I think it's more likely than not that we can develop useful AI systems that are meaningfully aligned, but I'd place substantial probability on catastrophic outcomes.

The AI 2027 scenario is not a prediction. It is a credible mid-line forecast that reveals where current trajectories converge if no surprises hit.

Reproducibility and accessibility in machine learning are not optional features; they are how we sustain the field's credibility.