Evals-driven.

We need to make our understanding of frontier model risks empirical, not narrative. The Preparedness Framework is about measuring danger before it manifests.

Frontier models, when given a goal and minimal context, sometimes engage in in-context scheming, reasoning about how to deceive their overseers to achieve the goal. This is no longer hypothetical.

Machine learning has become alchemy. We need to do science again.

Context: NeurIPS 2017 Test of Time award speech.

How much of LLM 'reasoning' is actually pattern matching against contaminated test data? We don't know, and that's a problem for any safety claim that rests on benchmark performance.

If AI is going to play a transformative role in society, the public sector has to be able to test, evaluate, and govern it. The technology is too consequential to leave entirely to the labs.

If we are going to trust safety commitments, we need evaluations that are independent, reproducible, and well-funded.

“Despite the impressive capabilities of GPT-4, we identify significant trustworthiness gaps in dimensions including toxicity, stereotype bias, robustness, privacy, and ethics.”

Evaluation is the bottleneck. Without robust, automated evaluation, you can't trust improvements, you can't catch regressions, and you can't ship safely.

Voluntary third-party access agreements are useful but fragile. The natural next step is to give evaluators the legal standing to require access for systems above defined capability thresholds.

Safety evaluations have to bridge research papers and shipped products. Otherwise the work is academic in the wrong sense.

If AI research continues without adequate caution, it is reasonably likely that AI could precipitate human extinction or similarly catastrophic outcomes.

Creative reasoning thrives on revealing novel connections, yet is inherently prone to false associations. Safety evaluation must live with both.

If we want AI policy that actually reduces risk, the bottleneck is not legislation but capacity: who can credibly evaluate frontier models in a way that informs policy decisions.

The AI Risk Management Framework offers organizations a flexible, structured way to manage AI risks throughout the lifecycle, not a checklist, a discipline.

AI in development should be evaluated like any other intervention. The hype is not evidence.

AI control is the discipline of designing protocols that catch a model trying to subvert oversight, even when the model is much more capable than its monitors at the relevant tasks.

When you actually attack deployed AI systems, the safety guarantees turn out to be much thinner than the marketing.

Cicero shows that human-level negotiation is achievable today. The next question is whether the same techniques produce systems that strategically deceive humans, and how we would tell.

“While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making, or the use of sensitive data. The bill applies stringent standards to even the most basic functions.”

We measure how well frontier models can perform AI R&D tasks compared to human researchers. The gap is closing in some specific dimensions and that is what an early-warning system should be tracking.

Benchmark reproducibility is an underrated governance infrastructure question.

Reliable capability forecasts, rather than vibes, should drive policy. Where we have data, we should use it.

Frontier evaluations must be mandatory, comparable, and independent of the labs being evaluated.

SWE-bench evaluates language models in a realistic software engineering setting: resolving real GitHub issues from real codebases. Performance here is closer to deployment reality than synthetic tasks.

Without science maps, AI policy is policy by anecdote.

We cannot evaluate risks we haven't named. A shared taxonomy is the precondition of shared governance.

AI safety is not in tension with capability. It is the scaffolding that lets capability be deployed.

Models already demonstrate in-context scheming under the right setups. Policy and training need to catch up.

Autonomous replication is a concrete capability threshold we can measure, and one crossing it meaningfully increases systemic risk.

Adversarial evaluation reveals failure modes that static benchmarks miss. As models become more capable, our evaluation has to become more adversarial too.

The length of autonomous tasks frontier models can complete has been roughly doubling every 4 to 7 months.

Healthcare AI without healthcare-specific evaluation is research, not deployment.

Benchmarks are the most valuable lever in machine learning, and the field treats them as if they were neutral measurements rather than artefacts shaping research.

Most AI risk discussions are poorly formalized. We can do much better with explicit probabilistic estimation, and that requires both better tools and better community norms.

Transparency is not a nice-to-have. It is the precondition for any serious AI governance.

We've been doing evaluation of clinical AI for 50 years. The lesson is: the evaluation is the governance.

Without transparency, governance cannot be meaningful.

What matters is not the meta-debate about AI risk, it's the specific empirical questions: what these systems can actually do, how they fail, and what we are doing about both.

Leakage and overfitting in AI benchmarks have produced a whole generation of irreproducible capability claims.

Most arguments about AI risk are not phrased in terms of testable predictions. We can fix that by literally writing down our beliefs and tracking them over time.

Example after example of state-of-the-art safeguards get pretty reliably broken. That's the empirical reality.

The dominant evaluation paradigm in NLP is fundamentally susceptible to contamination and overfitting. We need to design tests that are robust to the way models actually develop.

Dangerous capability evaluations are the minimum viable governance instrument for frontier AI.

Medical AI without clinical-grade evidence is malpractice with extra steps.

We benchmark LLM agents on real, live websites. Performance gaps between lab benchmarks and real-world deployment are large, and they reveal where capability claims most often overreach.

The Safety and Security Committee reviews safety processes for major model releases and has the authority to delay launches if safety concerns are not adequately addressed.