Acceleration.

Regulation of AI is regulation of inference. Regulation of inference is regulation of thought.

We believe the next decade of AI is action, not just text generation. Adept's bet was that the agents that take real-world action will be the most consequential AI systems.

“We've got to let the private sector cook.”

Stargate is a new American company that will invest $500 billion, at least, in AI infrastructure.

Humanoid robots running large neural network policies are the embodied analogue of GPT-style language models. The same scaling laws that produced reasoning in language are starting to produce skill acquisition in physical action.

Effective accelerationism wants to propel humanity up the Kardashev gradient.

Context: Lex Fridman podcast episode 407.

“I'm not here this morning to talk about AI safety... I'm here to talk about AI opportunity.”

I think a single individual could probably do the entire AGI training pipeline. The bottleneck is not budget or compute, it is engineering insight.

“Any deceleration of AI will cost lives. Deaths that were preventable by the AI that was prevented from existing is a form of murder.”

Context: The Techno-Optimist Manifesto.

“We believe that we are, have been, and will always be the masters of technology, not mastered by technology.”

The 'AI is going to kill us' people are remarkably aligned with the 'AI should be regulated by us' people. Convenient.

“The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.”

“We contend that GPT-4 could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system.”

If we want machines that can do useful things in the physical world, we need to scale up real-world data and self-supervision. Internet text gets you far, but not into a kitchen.

AI regulation is a Trojan horse for incumbent protection.

DALL·E 2 generates more realistic and accurate images with 4x greater resolution. The visual reasoning that emerges from large multimodal training continues to surprise us.

“We propose Mamba, a selective state space model that achieves Transformer-level performance with linear scaling in sequence length.”

“We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text.”

“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”

Test-time compute can be more effective than scaling model size for certain reasoning tasks. The trade-off between training-time and test-time scaling is far richer than headline metrics suggest.

Reasoning is one of the most important capabilities of LLMs. Chain-of-thought is the simplest demonstration that scale plus reasoning prompts unlocks much more than either alone.

We're using the same architecture that powers language models to design RNA medicines. The substrate matters, but the underlying learning machinery generalizes.

“Chain-of-thought prompting elicits reasoning in large language models. We find that scaling matters: chain-of-thought is an emergent ability of scale.”

“The accelerating progress of technology and changes in the mode of human life give the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue.”

Context: Reported by Stanislaw Ulam in his 1958 obituary of von Neumann; widely cited as the first articulation of a technological singularity.

“Multi-headed self-attention enables the model to jointly attend to information from different representation subspaces at different positions.”

Our agent, AlphaStar, learned by playing against itself, scaling self-play to a domain at the edge of professional human ability.

GPT-4o is our new flagship model that can reason across audio, vision, and text in real time. The end-to-end training over modalities is what unlocks low-latency interaction.

Score-based generative models learn the gradient of the log data distribution. The framework unifies many seemingly disparate generative model families and underpins modern diffusion models.

Sora generates video by predicting future frames from a sequence of input frames. This formulation lets us scale data and compute in the same way that text models do.

FlashAttention computes attention with no approximation in linear memory, by being aware of GPU memory hierarchy. The same engineering carefulness can keep delivering capability for years.