The map: AI, machine learning, deep learning and generative AI

The vocabulary of AI is a pile of overlapping words, and people use them loosely. A single picture sorts it out: each term is a subset of the one before it. Learn the nesting once and every headline becomes easier to read.

Nested circles#

• Artificial intelligence (AI) is the broadest term: any system that does something we would call intelligent. It includes hand-written chess programs and rule-based expert systems that involve no learning at all.
• Machine learning (ML) is the part of AI where the system learns its behaviour from data instead of being given rules (chapter 1).
• Deep learning (DL) is the part of ML that uses neural networks with many layers. It is what made image recognition, speech recognition and translation work well.
• Generative AI is deep learning that creates new content: text, images, audio, video, code. It includes language models and diffusion models.
• Large language models (LLMs) are generative models, usually transformers, trained on enormous amounts of text to predict the next token.

The circles are nested, but the methods differ a lot: a gradient-boosted tree (ML, not deep learning) is still the best tool for many tables of business data, while a transformer (deep learning, generative) powers chat assistants.

A short history, with the turning points#

Dates are approximate and the boundaries are fuzzy; each wave built on the last rather than replacing it.

How big is a model?#

"Parameters" are the adjustable numbers a model learns. Counting them is just arithmetic. A dense layer with n_in inputs and n_out outputs has n_in × n_out weights plus n_out biases. A transformer block has roughly 12 × d² parameters, where d is the width of the model (chapter 89), and the word embeddings add vocabulary size × d. The script uses those rules on a line, on a small image classifier, and on two famous language-model shapes.

def dense(n_in, n_out):
    return n_in * n_out + n_out

line = 2                                                             # y = w x + b
mlp = dense(784, 128) + dense(128, 64) + dense(64, 10)               # a small network for 28 x 28 images and 10 classes

def transformer(layers, width, vocab):
    blocks = 12 * layers * width ** 2                                # attention (4 d^2) + feed-forward (8 d^2) per block
    embeddings = vocab * width
    return blocks + embeddings

gpt2_small = transformer(layers=12, width=768, vocab=50_257)         # the published GPT-2 small shape
gpt3_scale = transformer(layers=96, width=12_288, vocab=50_257)      # the published GPT-3 shape

for name, n in (("a straight line", line), ("a small image classifier (MLP)", mlp), ("GPT-2 small shape", gpt2_small), ("GPT-3 shape", gpt3_scale)):
    print(f"{name:32s} {n:>18,d} parameters")
print(f"\nGPT-3 is about {gpt3_scale / gpt2_small:,.0f} times bigger than GPT-2 small, and {gpt3_scale / mlp:,.0f} times bigger than the little classifier")

a straight line                                   2 parameters
a small image classifier (MLP)              109,386 parameters
GPT-2 small shape                       123,532,032 parameters
GPT-3 shape                         174,563,733,504 parameters

GPT-3 is about 1,413 times bigger than GPT-2 small, and 1,595,851 times bigger than the little classifier

The estimates land on the published sizes: about 124 million for GPT-2 small and about 175 billion for the GPT-3 shape. A model of that size needs hundreds of gigabytes just to hold its weights in 16-bit numbers, which is why serving and quantising large models is a field of its own (chapter 99).