To Understand: The Transformer and Scale

AlexNet could learn to see. It could not hold a conversation, or follow an idea across a paragraph. That required an architecture actually suited to language and context, not just images.

Key Milestones

• The Transformer architecture, 2017, Google, "Attention Is All You Need." A new architecture let a model weigh the relationships between every word in a passage at once, rather than reading strictly left to right, unlocking a far deeper grasp of context.

The Energy, Social, and Economic Impact of Understanding

The Transformer made language models worth scaling up substantially, and scaling meant adding parameters. A parameter is one small adjustable number inside the model, a dial that gets tuned, bit by bit, as the model reads through training text, until the whole collection of dials together can predict language well. More parameters mean more capacity to represent nuance: the difference between a word's ten most common meanings instead of just its first, or between a flat instruction and one that picks up on tone.

OpenAI's GPT-1, in 2018, had 117 million of these parameters. GPT-2, a year later, had 1.5 billion. GPT-3, in 2020, had 175 billion, roughly 1,500 times larger than the model that started the line two years earlier. That jump mattered because capability did not scale in a straight line with size, it scaled in jumps: skills such as basic arithmetic, translating between languages it was never explicitly taught to translate, and following multi-step instructions only became reliable once the model, and its energy bill, crossed a certain scale. Bigger, in this narrow sense, kept buying genuinely new behavior, not just a marginally better version of the old one.

View chart in the full report

We believe this architecture made the rest of this series possible. Every large language model since, including the ones many of us use today, descends from the core idea published in that 2017 paper.