BUILD · TRAIN · GENERATE

Building Large Language Models
From Scratch

By the end of this course you will have built a working Transformer language model in Python, trained it on real text, and generated new text from it. The same architecture powers GPT-4 — scaled up by a factor of roughly 100,000.

Python 3.10+ PyTorch 2.x ~300 lines of code Runs on CPU (slow) or GPU (fast) No cloud required
01

How Language Models Work

Tokens, sequences, and the next-token prediction task
~25 min

A language model has one job: given a sequence of tokens, predict the probability distribution over what token comes next. That is the complete task definition. Every capability that emerges from large language models — reasoning, coding, translation, summarisation — arises from doing this task well on vast amounts of text.

Tokens: the atomic unit

Text is not fed to a language model as characters or words. It is broken into tokens — sub-word units that balance vocabulary size against sequence length. Modern LLMs use Byte-Pair Encoding (BPE), which starts with individual characters and repeatedly merges the most frequent adjacent pairs until a target vocabulary size is reached.

# BPE tokenisation example Text: "The borrower repaid" Tokens: ["The", " borrow", "er", " repaid"] IDs: [464, 11470, 263, 2071] # For our from-scratch model we'll use character-level tokenisation: # Each character is one token. Simple, educational, less efficient. Text: "hello" Tokens: ['h', 'e', 'l', 'l', 'o'] IDs: [8, 5, 12, 12, 15] # indices into vocabulary # Vocabulary: every unique character in the training text # Vocabulary size (V): typically 65–100 for character-level English text

The prediction task

Training a language model means learning to predict the next token given all previous tokens. Given the sequence "The loan was", the model should assign high probability to "approved", "denied", "processed" etc. and low probability to nonsense.

# The model learns P(next_token | all_previous_tokens) Input sequence: [T, h, e, " ", l, o, a, n, " ", w, a, s] Target sequence: [h, e, " ", l, o, a, n, " ", w, a, s, " "] # Every position is a training example: # Given [T] → predict h # Given [T, h] → predict e # Given [T, h, e] → predict " " # Given [T, h, e, " ", l, o, a, n] → predict " " # ... etc # This means one sequence of length T gives us T training examples!

The context window

The model can only see a fixed number of previous tokens — the context window (also called block size or sequence length). For our small model we will use 128 or 256 tokens. GPT-4 uses 128,000. The architecture is identical; the difference is scale.

Interactive: TokeniserDEMO

Type text to see it broken into character-level tokens and their integer IDs. This is exactly what your model will receive as input.

Embeddings: turning integers into vectors

Token IDs (integers) cannot be fed directly into a neural network. They are first converted to embedding vectors — dense lists of floating-point numbers that the model learns during training. Each token in the vocabulary gets its own embedding vector of size d_model (e.g. 128 or 512).

# Embedding table: shape [vocab_size, d_model] vocab_size = 65 # number of unique characters d_model = 128 # embedding dimension # Token ID 8 ('h') maps to a learned vector: embed('h') = [0.23, -0.41, 0.87, 0.12, ..., -0.55] # 128 numbers # For a sequence of T tokens: # Input shape: [batch_size, T] (integer token IDs) # After embed: [batch_size, T, d_model] (continuous vectors)
The core idea: Training a language model is just supervised learning where the label for every input position is the next token. The complexity is entirely in the architecture that maps input tokens to output probability distributions — and that architecture is the Transformer.

Understanding check

A training sequence "hello" of length 5 generates how many (input, target) pairs for next-token prediction?

Correct. Each of the 5 positions contributes one training example: [h]→e, [h,e]→l, [h,e,l]→l, [h,e,l,l]→o, [h,e,l,l,o]→(whatever follows). This is why language model training is data-efficient — a corpus of 1 million characters provides 1 million training examples.
02

The Attention Mechanism

Queries, keys, values, and scaled dot-product attention — the core innovation of the Transformer
~35 min · Core math

Attention is the mechanism that allows a language model to relate any token to any other token in the sequence, regardless of distance. Before attention, RNNs had to pass information through a chain of hidden states — information from early in a sequence degraded as it propagated forward. Attention solves this by allowing the model to directly "look at" any position when computing its representation.

The intuition: query, key, value

Attention is often described with a retrieval analogy. Imagine you are in a library (the sequence of tokens). You have a question — your Query (Q). Each book in the library has a spine with index information — the Key (K). You scan the keys, find the books most relevant to your query, and read their content — the Value (V). The output is a weighted mixture of values, where the weights come from how well each key matched your query.

# For each token position, we compute Q, K, V by projecting the embedding: Q = X @ W_Q # shape: [T, d_k] — what am I looking for? K = X @ W_K # shape: [T, d_k] — what do I contain? V = X @ W_V # shape: [T, d_v] — what do I give out? # W_Q, W_K, W_V are learned weight matrices # Attention scores: how much does each position attend to each other? scores = Q @ K.transpose(-2, -1) # shape: [T, T] # Scale by √d_k to prevent extremely large values (gradient vanishing) scores = scores / sqrt(d_k) # Softmax: convert scores to probabilities that sum to 1 weights = softmax(scores, dim=-1) # shape: [T, T] # Final output: weighted mixture of values output = weights @ V # shape: [T, d_v] # Compact form (the famous formula): Attention(Q, K, V) = softmax( Q·Kᵀ / √d_k ) · V

Causal (masked) attention

Language models are autoregressive — they generate tokens left to right. During training, position 5 should not be able to see tokens at positions 6, 7, 8... (that would be cheating — the answer is in the input). We enforce this with a causal mask: setting attention scores for future positions to -∞ before softmax, which makes their weights exactly 0.

# Causal mask: upper triangle is -inf, lower triangle (including diagonal) is 0 mask = [[ 0, -inf, -inf, -inf], # position 0 sees only itself [ 0, 0, -inf, -inf], # position 1 sees positions 0,1 [ 0, 0, 0, -inf], # position 2 sees positions 0,1,2 [ 0, 0, 0, 0]] # position 3 sees all scores = scores + mask # add mask before softmax weights = softmax(scores) # -inf → 0 after softmax (e^-inf = 0)
Interactive: Attention weight visualiserDEMO

Click a token to see which other tokens it attends to most strongly. Brighter cells = higher attention weight. The upper triangle is masked (future tokens cannot be seen).

Click a token above to see its attention pattern.

Multi-head attention

A single attention head learns one type of relationship between tokens. Multi-head attention runs h attention heads in parallel, each with different learned Q/K/V projections, then concatenates their outputs. This allows the model to simultaneously attend to different types of relationships — syntactic, semantic, positional.

# Multi-head attention with h heads: head_i = Attention(X @ W_Q_i, X @ W_K_i, X @ W_V_i) # for each head i combined = concat(head_1, head_2, ..., head_h) # shape: [T, h*d_v] output = combined @ W_O # project back to d_model # Typical settings: d_model = 512, n_heads = 8, d_k = d_v = d_model / n_heads = 64 # Each head works in 64-dimensional space; outputs concatenate to 512

Softmax: the probability converter

Softmax converts a vector of arbitrary real numbers into a probability distribution that sums to 1. It is used both in the attention mechanism and in the final output layer of the model.

softmax(xᵢ) = e^xᵢ / Σⱼ e^xⱼ # Example: raw attention scores for one query against 4 keys scores = [2.1, 0.8, -0.5, 1.4] exp = [e^2.1, e^0.8, e^-0.5, e^1.4] = [8.17, 2.23, 0.61, 4.06] sum_exp = 8.17 + 2.23 + 0.61 + 4.06 = 15.07 weights = [0.542, 0.148, 0.040, 0.269] # sum = 1.0 ✓ # Position 0 gets 54% of attention weight — strongest match
Interactive: Softmax explorerDEMO

Adjust raw attention scores to see how softmax converts them to probability weights. Notice how dominant scores suppress others exponentially.

Why attention is powerful: Every token can directly influence every other token in a single layer, with learned, content-dependent weights. The cost is O(T²) computation per layer — manageable for context windows up to ~100k tokens, but the reason very long context is expensive.
03

The Transformer Architecture

Positional encoding, feedforward layers, layer norm, and how the blocks stack
~30 min

The Transformer is a stack of identical blocks. Each block has two sub-layers: a multi-head attention layer (which relates tokens to each other) and a feedforward network (which transforms each token independently). Understanding these components and how they fit together is the complete architectural picture.

Positional encoding

Attention is permutation-invariant — if you shuffle the tokens, the attention outputs shuffle in the same way, but the model has no way to know tokens were in a different order. Positional encodings inject position information by adding a position-dependent vector to each token embedding.

# Learned positional embeddings (what we'll use — simpler): pos_embed = nn.Embedding(block_size, d_model) x = token_embed(tokens) + pos_embed(positions) # Original sinusoidal positional encoding (from "Attention is All You Need"): PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) # Different frequencies for different dimensions — unique vector per position # Advantage: can generalise to sequence lengths longer than training

The feedforward network (FFN)

After attention has allowed tokens to communicate, each token is independently processed by a small 2-layer neural network. This is where most of the model's "factual knowledge" is believed to be stored. The FFN expands the dimension (typically by 4×), applies a non-linearity, then contracts back.

FFN(x) = max(0, x @ W₁ + b₁) @ W₂ + b₂ # Or with GELU activation (smoother than ReLU, used by GPT-2+): FFN(x) = GELU(x @ W₁ + b₁) @ W₂ + b₂ # Dimensions: W₁: [d_model, 4*d_model] # expand — e.g. 512 → 2048 W₂: [4*d_model, d_model] # contract — e.g. 2048 → 512

Layer normalisation

Layer normalisation stabilises training by normalising the activations across the feature dimension for each token independently. Unlike batch normalisation (which normalises across the batch), layer norm works on each position's vector and is well-suited to variable-length sequences.

LayerNorm(x) = γ * (x - mean(x)) / (std(x) + ε) + β # γ (scale) and β (shift) are learned parameters # ε is a small constant (~1e-5) for numerical stability # Applied BEFORE each sub-layer in GPT-2+ (Pre-Norm formulation)

Residual connections

Each sub-layer wraps its computation in a residual (skip) connection: the input is added to the output. This creates a direct gradient pathway through the network, allowing training of very deep models without vanishing gradients.

# One Transformer block: x = x + Attention(LayerNorm(x)) # attention sub-layer x = x + FFN(LayerNorm(x)) # feedforward sub-layer # The '+' is the residual connection # Gradient can flow directly: ∂loss/∂x = 1 + ∂attention/∂x # The '1' term prevents vanishing gradients in deep networks

The complete GPT architecture

Full model: input → output

1. Token embedding: [B, T] integer IDs → [B, T, d_model] float vectors

2. Position embedding: Add position-dependent vectors → [B, T, d_model]

3. N × Transformer blocks: Each block: LayerNorm → MultiHeadAttention → Residual → LayerNorm → FFN → Residual

4. Final LayerNorm: Normalise the output of the last block

5. Output projection: [B, T, d_model] → [B, T, vocab_size] — one logit per token in vocabulary

6. Softmax (at inference): Logits → probabilities for next token

Interactive: Transformer block data flowDEMO

Visualise how tensor shapes change as data flows through the model. Adjust model size parameters.

128
4
4
128

Understanding check

Why does the feedforward network in a Transformer expand to 4×d_model before contracting back? What purpose does this serve?

Correct. The FFN is often described as "where the model stores its knowledge." The attention layer routes information between tokens, but the FFN transforms each token's representation in a high-dimensional space. Each neuron in the expanded layer can be thought of as storing a pattern that, when recognised, activates a particular response. The 4× expansion is empirically found to work well — it's not derived from theory.
04

Building in PyTorch

Writing the model code — every component from scratch
~40 min · Hands-on code

Now we build. Each component is a PyTorch nn.Module. We build them bottom-up: attention head → multi-head attention → transformer block → full model. Read each block carefully — every line corresponds to a concept from modules 1–3.

Setup and configuration

config.pypython
import torch
import torch.nn as nn
from torch.nn import functional as F

# ── Hyperparameters ─────────────────────────────────────
# Small model: trains in ~10 min on CPU, ~1 min on GPU
batch_size  = 32    # sequences processed in parallel
block_size  = 128   # context window (tokens the model can see)
d_model     = 128   # embedding dimension
n_heads     = 4     # attention heads
n_layers    = 4     # transformer blocks
dropout     = 0.1   # regularisation
learning_rate = 3e-4
max_iters   = 5000
eval_interval = 500

device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'Using device: {device}')

Single attention head

attention.py — single headpython
class Head(nn.Module):
    """One head of self-attention."""

    def __init__(self, head_size):
        super().__init__()
        # Projections for Q, K, V — no bias (common in transformers)
        self.key   = nn.Linear(d_model, head_size, bias=False)
        self.query = nn.Linear(d_model, head_size, bias=False)
        self.value = nn.Linear(d_model, head_size, bias=False)
        # Causal mask: register as buffer (not a parameter, but saved with model)
        self.register_buffer('tril',
            torch.tril(torch.ones(block_size, block_size)))
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        B, T, C = x.shape  # batch, time (seq len), channels (d_model)

        k = self.key(x)    # (B, T, head_size)
        q = self.query(x)  # (B, T, head_size)
        v = self.value(x)  # (B, T, head_size)

        # Attention scores: Q · Kᵀ / √head_size
        scale = k.shape[-1] ** -0.5         # 1/√head_size
        wei = q @ k.transpose(-2, -1) * scale  # (B, T, T)

        # Apply causal mask: future positions → -inf
        wei = wei.masked_fill(
            self.tril[:T, :T] == 0,
            float('-inf'))                   # (B, T, T)

        # Softmax: turn scores into probabilities
        wei = F.softmax(wei, dim=-1)         # (B, T, T)
        wei = self.dropout(wei)

        # Weighted aggregation of values
        out = wei @ v                          # (B, T, head_size)
        return out

Multi-head attention

attention.py — multi-headpython
class MultiHeadAttention(nn.Module):
    """Multiple attention heads running in parallel."""

    def __init__(self, n_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList(
            [Head(head_size) for _ in range(n_heads)])
        # Project concatenated head outputs back to d_model
        self.proj = nn.Linear(n_heads * head_size, d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        # Run each head, concatenate along the feature dimension
        out = torch.cat([h(x) for h in self.heads], dim=-1)
        out = self.dropout(self.proj(out))
        return out

Feedforward network and Transformer block

model.py — FFN and Blockpython
class FeedForward(nn.Module):
    """Position-wise feedforward: expand → non-linearity → contract."""

    def __init__(self, d_model):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),   # expand 4×
            nn.GELU(),                           # smooth non-linearity
            nn.Linear(4 * d_model, d_model),   # contract back
            nn.Dropout(dropout),
        )

    def forward(self, x):
        return self.net(x)


class Block(nn.Module):
    """One Transformer block: communication (attention) + computation (FFN)."""

    def __init__(self, d_model, n_heads):
        super().__init__()
        head_size = d_model // n_heads  # each head gets equal share
        self.attn = MultiHeadAttention(n_heads, head_size)
        self.ffn  = FeedForward(d_model)
        # Pre-norm: LayerNorm before each sub-layer (GPT-2 style)
        self.ln1  = nn.LayerNorm(d_model)
        self.ln2  = nn.LayerNorm(d_model)

    def forward(self, x):
        # Residual connection: x = x + sublayer(LayerNorm(x))
        x = x + self.attn(self.ln1(x))   # attention + residual
        x = x + self.ffn(self.ln2(x))    # feedforward + residual
        return x

The full GPT model

model.py — complete modelpython
class GPT(nn.Module):
    """Decoder-only Transformer (GPT architecture)."""

    def __init__(self, vocab_size):
        super().__init__()
        # 1. Token embedding table
        self.tok_embed = nn.Embedding(vocab_size, d_model)
        # 2. Positional embedding table (learned)
        self.pos_embed = nn.Embedding(block_size, d_model)
        # 3. Stack of Transformer blocks
        self.blocks = nn.Sequential(
            *[Block(d_model, n_heads) for _ in range(n_layers)])
        # 4. Final layer norm
        self.ln_f = nn.LayerNorm(d_model)
        # 5. Output projection: d_model → vocab_size (logits)
        self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
        # Weight tying: share token embedding and output projection weights
        # (reduces parameters, improves performance — used in GPT-2)
        self.tok_embed.weight = self.lm_head.weight
        self.apply(self._init_weights)

    def _init_weights(self, module):
        # Small initial weights → stable training
        if isinstance(module, nn.Linear):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                torch.nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def forward(self, idx, targets=None):
        B, T = idx.shape
        # 1. Token + position embeddings
        tok_emb = self.tok_embed(idx)          # (B, T, d_model)
        pos_emb = self.pos_embed(
            torch.arange(T, device=device))     # (T, d_model)
        x = tok_emb + pos_emb                   # (B, T, d_model)
        # 2. Transformer blocks
        x = self.blocks(x)                     # (B, T, d_model)
        # 3. Final norm + project to vocabulary
        x = self.ln_f(x)                       # (B, T, d_model)
        logits = self.lm_head(x)               # (B, T, vocab_size)
        # 4. Compute loss if targets provided (training mode)
        if targets is None:
            return logits, None
        # Cross-entropy loss over all positions
        B, T, V = logits.shape
        logits_2d  = logits.view(B*T, V)
        targets_1d = targets.view(B*T)
        loss = F.cross_entropy(logits_2d, targets_1d)
        return logits, loss
Interactive: Model parameter counterDEMO

See exactly how many parameters your model has and where they live. Adjust to understand how scale is achieved.

128
4
4
65
05

The Training Loop

Cross-entropy loss, backpropagation, AdamW, and what loss curves tell you
~30 min

Training is the process of adjusting model weights to minimise the loss — the gap between what the model predicts and what the correct next token actually is. This requires three components: a loss function, a method to compute gradients (backpropagation), and an optimiser to update weights.

Cross-entropy loss

The loss function for language models is cross-entropy — it measures how surprised the model is by the correct next token. If the model assigns high probability to the correct token, the loss is low. If it assigns low probability, the loss is high.

CrossEntropy(logits, target) = -log( softmax(logits)[target] ) # Example: vocabulary of 5 tokens, correct next token is index 2 logits = [1.2, -0.5, 3.1, 0.8, -1.2] # raw model outputs probs = softmax(logits) = [0.11, 0.02, 0.73, 0.07, 0.01] # after softmax loss = -log(0.73) = 0.31 # low loss — model was fairly confident # If the model was wrong (predicted token 0): loss = -log(0.11) = 2.21 # high loss — model surprised by the answer # Baseline: random model predicts each token equally loss_random = -log(1/vocab_size) = -log(1/65) = log(65) ≈ 4.17 # A trained model should achieve loss well below 4.17

Backpropagation

Backpropagation computes the gradient of the loss with respect to every parameter in the model — using the chain rule of calculus. PyTorch does this automatically via its autograd engine. You call loss.backward() and every parameter's .grad attribute is populated.

# Chain rule intuition: # loss depends on logits, which depend on W_lm_head, which depend on... # ∂loss/∂W = ∂loss/∂logits × ∂logits/∂W # PyTorch autograd does this automatically for every parameter. # You just need to define the forward pass correctly.

AdamW optimiser

AdamW is the standard optimiser for Transformers. It adapts the learning rate for each parameter based on the history of gradients, and applies weight decay (L2 regularisation) correctly — separately from the gradient step, which is why it's called AdamW (decoupled weight decay).

# Adam update rule (simplified): m = β₁ × m + (1-β₁) × gradient # momentum (running mean of gradients) v = β₂ × v + (1-β₂) × gradient² # adaptive (running mean of squared gradients) param = param - lr × m / (√v + ε) # update # Weight decay (AdamW): param = param × (1 - lr × weight_decay) # applied separately # Typical settings for LLMs: lr = 3e-4 # learning rate betas = (0.9, 0.95) weight_decay = 0.1

The training loop

train.py — data loading and training looppython
import torch

# ── Data preparation ─────────────────────────────────────
with open('input.txt', 'r') as f:
    text = f.read()

# Build character vocabulary
chars  = sorted(list(set(text)))
vocab_size = len(chars)
stoi = {ch:i for i,ch in enumerate(chars)}   # char → int
itos = {i:ch for i,ch in enumerate(chars)}   # int → char
encode = lambda s: [stoi[c] for c in s]
decode = lambda l: ''.join([itos[i] for i in l])

# Train/validation split
data = torch.tensor(encode(text), dtype=torch.long)
n    = int(0.9 * len(data))
train_data = data[:n]
val_data   = data[n:]

def get_batch(split):
    """Sample a batch of (inputs, targets) from the dataset."""
    data = train_data if split == 'train' else val_data
    # Random starting positions for each sequence in the batch
    ix = torch.randint(len(data) - block_size, (batch_size,))
    x  = torch.stack([data[i         : i+block_size  ] for i in ix])
    y  = torch.stack([data[i+1       : i+block_size+1] for i in ix])
    return x.to(device), y.to(device)

# ── Model and optimiser ──────────────────────────────────
model = GPT(vocab_size).to(device)
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)

print(f'Parameters: {sum(p.numel() for p in model.parameters()):,}')

# ── Training loop ────────────────────────────────────────
for step in range(max_iters):

    # Periodically evaluate on validation set
    if step % eval_interval == 0:
        model.eval()
        with torch.no_grad():
            xv, yv = get_batch('val')
            _, val_loss = model(xv, yv)
        model.train()
        print(f'step {step:5d}: val_loss = {val_loss.item():.4f}')

    # Forward pass
    xb, yb = get_batch('train')
    logits, loss = model(xb, yb)

    # Backward pass
    optimizer.zero_grad(set_to_none=True)  # clear old gradients
    loss.backward()                          # compute new gradients
    optimizer.step()                         # update weights

print('Training complete!')
Interactive: Training loss curve simulatorDEMO

See how different hyperparameters affect the training curve. Identify overfitting, underfitting, and good training runs.

3e-4
Medium
Medium
06

Generation Strategies

Temperature, top-k, top-p, and how to sample from a trained model
~20 min

After training, the model can generate text. Generation is autoregressive: sample one token, append it to the context, sample the next, repeat. The interesting choices are in how you sample from the output distribution — these choices dramatically affect output quality, creativity, and coherence.

The generation code

generate.pypython
def generate(model, idx, max_new_tokens,
              temperature=1.0, top_k=None):
    """
    Generate text autoregressively.
    idx: (B, T) tensor of starting tokens
    Returns: (B, T + max_new_tokens) tensor
    """
    model.eval()
    with torch.no_grad():
        for _ in range(max_new_tokens):
            # Crop context to block_size (can't exceed the window)
            idx_cond = idx[:, -block_size:]

            # Forward pass → logits for next token
            logits, _ = model(idx_cond)
            logits = logits[:, -1, :]  # take last position: (B, vocab_size)

            # ── Sampling strategies ────────────────────────

            # 1. Temperature: reshape the distribution
            logits = logits / temperature

            # 2. Top-k: keep only the k most likely tokens
            if top_k is not None:
                vals, _ = torch.topk(logits, top_k)
                logits[logits < vals[:, [-1]]] = float('-inf')

            # 3. Convert to probabilities
            probs = F.softmax(logits, dim=-1)

            # 4. Sample from the distribution
            idx_next = torch.multinomial(probs, num_samples=1)

            # 5. Append and continue
            idx = torch.cat((idx, idx_next), dim=1)

    return idx

# Usage: generate 500 characters starting from newline
context = torch.zeros((1, 1), dtype=torch.long, device=device)
output  = generate(model, context, max_new_tokens=500, temperature=0.8, top_k=40)
print(decode(output[0].tolist()))
Interactive: Generation strategy explorerDEMO

Adjust temperature and top-k to see how they reshape the probability distribution for the next token. This is exactly what happens at each step of generation.

1.0
All tokens

Sampling strategy guide

  • Temperature = 1.0: Sample from the model's raw distribution. Balanced creativity vs. coherence.
  • Temperature < 1.0 (e.g. 0.7): Sharpen the distribution — model picks more confidently from top tokens. More coherent, less creative. Use for factual or structured output.
  • Temperature > 1.0: Flatten the distribution — increases randomness. Useful for creative generation. Risk of incoherence.
  • Top-k = 40: Restrict sampling to only the 40 most probable tokens. Prevents low-probability garbage tokens from ever appearing.
  • Top-p (nucleus sampling) = 0.9: Restrict sampling to the smallest set of tokens whose cumulative probability exceeds 0.9. Adapts to the distribution — wider when the model is uncertain, narrower when confident.
  • Greedy (temperature → 0): Always pick the most probable token. Deterministic but often repetitive.
07

Complete Implementation

The full runnable file — copy it, get a dataset, and train your first LLM
RUNNABLE CODE

This is everything in one file. Copy it, save as gpt.py, get a text file (input.txt), and run it. On CPU it will produce readable output within 10–20 minutes. On a GPU, within 2–3 minutes.

Before running: Install dependencies with pip install torch. Get a training text file — the complete works of Shakespeare (download here, ~1MB) works perfectly. Save it as input.txt in the same directory.
gpt.py — complete implementation (~280 lines)python
"""
GPT from scratch — character-level language model
Based on: Vaswani et al. "Attention is All You Need" (2017)
          Karpathy "nanoGPT" (2022)

Usage:
  python gpt.py               # train on input.txt and generate
  python gpt.py --generate    # load saved model and generate only

Training corpus: put any .txt file as input.txt
Recommended: Tiny Shakespeare (~1MB)
  wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt
"""
import torch
import torch.nn as nn
from torch.nn import functional as F
import time, argparse

# ════════════════════════════════════════════════════════════
# HYPERPARAMETERS — adjust to your compute budget
# ════════════════════════════════════════════════════════════
batch_size    = 32     # sequences per batch
block_size    = 128    # context window (tokens)
d_model       = 128    # embedding dimension
n_heads       = 4      # attention heads per block
n_layers      = 4      # transformer blocks
dropout       = 0.1    # dropout rate
learning_rate = 3e-4   # AdamW learning rate
max_iters     = 5000   # training steps
eval_interval = 500    # evaluate every N steps
save_path     = 'gpt_model.pt'

device = 'cuda' if torch.cuda.is_available() else 'cpu'

# ════════════════════════════════════════════════════════════
# DATA
# ════════════════════════════════════════════════════════════
def load_data(path='input.txt'):
    with open(path, 'r', encoding='utf-8') as f:
        text = f.read()
    chars  = sorted(set(text))
    stoi   = {c:i for i,c in enumerate(chars)}
    itos   = {i:c for i,c in enumerate(chars)}
    encode = lambda s: [stoi[c] for c in s]
    decode = lambda l: ''.join(itos[i] for i in l)
    data   = torch.tensor(encode(text), dtype=torch.long)
    n      = int(0.9*len(data))
    return data[:n], data[n:], len(chars), encode, decode

def get_batch(data, bsz=batch_size, blk=block_size):
    ix = torch.randint(len(data)-blk, (bsz,))
    x  = torch.stack([data[i  :i+blk  ] for i in ix])
    y  = torch.stack([data[i+1:i+blk+1] for i in ix])
    return x.to(device), y.to(device)

# ════════════════════════════════════════════════════════════
# MODEL COMPONENTS
# ════════════════════════════════════════════════════════════
class Head(nn.Module):
    def __init__(self, head_size):
        super().__init__()
        self.key   = nn.Linear(d_model, head_size, bias=False)
        self.query = nn.Linear(d_model, head_size, bias=False)
        self.value = nn.Linear(d_model, head_size, bias=False)
        self.register_buffer('tril',
            torch.tril(torch.ones(block_size, block_size)))
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        B,T,C = x.shape
        k = self.key(x); q = self.query(x); v = self.value(x)
        wei = q @ k.transpose(-2,-1) * k.shape[-1]**-0.5
        wei = wei.masked_fill(self.tril[:T,:T]==0, float('-inf'))
        wei = self.dropout(F.softmax(wei, dim=-1))
        return wei @ v

class MultiHeadAttention(nn.Module):
    def __init__(self, n_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(n_heads)])
        self.proj  = nn.Linear(n_heads*head_size, d_model)
        self.drop  = nn.Dropout(dropout)
    def forward(self, x):
        return self.drop(self.proj(torch.cat([h(x) for h in self.heads], dim=-1)))

class FeedForward(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_model, 4*d_model), nn.GELU(),
            nn.Linear(4*d_model, d_model), nn.Dropout(dropout))
    def forward(self, x): return self.net(x)

class Block(nn.Module):
    def __init__(self):
        super().__init__()
        hs = d_model // n_heads
        self.attn = MultiHeadAttention(n_heads, hs)
        self.ffn  = FeedForward()
        self.ln1  = nn.LayerNorm(d_model)
        self.ln2  = nn.LayerNorm(d_model)
    def forward(self, x):
        x = x + self.attn(self.ln1(x))
        x = x + self.ffn(self.ln2(x))
        return x

class GPT(nn.Module):
    def __init__(self, vocab_size):
        super().__init__()
        self.tok_embed = nn.Embedding(vocab_size, d_model)
        self.pos_embed = nn.Embedding(block_size, d_model)
        self.blocks    = nn.Sequential(*[Block() for _ in range(n_layers)])
        self.ln_f      = nn.LayerNorm(d_model)
        self.lm_head   = nn.Linear(d_model, vocab_size, bias=False)
        self.tok_embed.weight = self.lm_head.weight  # weight tying
        self.apply(self._init)

    def _init(self, m):
        if isinstance(m, (nn.Linear, nn.Embedding)):
            nn.init.normal_(m.weight, 0, 0.02)
        if isinstance(m, nn.Linear) and m.bias is not None:
            nn.init.zeros_(m.bias)

    def forward(self, idx, targets=None):
        B,T = idx.shape
        x = (self.tok_embed(idx) +
             self.pos_embed(torch.arange(T, device=device)))
        x = self.ln_f(self.blocks(x))
        logits = self.lm_head(x)
        loss = None
        if targets is not None:
            B,T,V = logits.shape
            loss = F.cross_entropy(logits.view(B*T,V), targets.view(B*T))
        return logits, loss

    def generate(self, idx, max_new_tokens, temperature=1.0, top_k=40):
        self.eval()
        with torch.no_grad():
            for _ in range(max_new_tokens):
                logits, _ = self(idx[:, -block_size:])
                logits = logits[:, -1, :] / temperature
                if top_k is not None:
                    v, _ = torch.topk(logits, top_k)
                    logits[logits < v[:, [-1]]] = float('-inf')
                idx_next = torch.multinomial(F.softmax(logits, -1), 1)
                idx = torch.cat((idx, idx_next), dim=1)
        return idx

# ════════════════════════════════════════════════════════════
# MAIN
# ════════════════════════════════════════════════════════════
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--generate', action='store_true')
    args = parser.parse_args()

    train_data, val_data, vocab_size, encode, decode = load_data()
    model = GPT(vocab_size).to(device)
    print(f'Device: {device} | Vocab: {vocab_size} | '
          f'Params: {sum(p.numel() for p in model.parameters()):,}')

    if args.generate:
        model.load_state_dict(torch.load(save_path, map_location=device))
    else:
        # Train
        opt = torch.optim.AdamW(model.parameters(), lr=learning_rate,
                                weight_decay=0.1)
        t0 = time.time()
        for step in range(1, max_iters+1):
            xb, yb = get_batch(train_data)
            _, loss = model(xb, yb)
            opt.zero_grad(True); loss.backward(); opt.step()
            if step % eval_interval == 0:
                model.eval()
                with torch.no_grad():
                    xv, yv = get_batch(val_data)
                    _, vloss = model(xv, yv)
                model.train()
                elapsed = time.time() - t0
                print(f'step {step:5d} | train {loss.item():.4f} | '
                      f'val {vloss.item():.4f} | {elapsed:.0f}s')
        torch.save(model.state_dict(), save_path)
        print(f'Model saved to {save_path}')

    # Generate
    start = torch.zeros((1,1), dtype=torch.long, device=device)
    output = model.generate(start, max_new_tokens=500,
                             temperature=0.8, top_k=40)
    print('\n──── GENERATED TEXT ────\n')
    print(decode(output[0].tolist()))

Expected output and what to look for

What you should see as training progresses

  • Step 0: val_loss ≈ 4.17 (random model — log(65) for 65-char vocab)
  • Step 500: val_loss ≈ 2.8–3.2. Generated text has some English letters but mostly nonsense.
  • Step 2000: val_loss ≈ 1.8–2.2. Generated text has real words, some plausible word sequences.
  • Step 5000: val_loss ≈ 1.4–1.6. Generated text reads like Shakespearean dialogue — character names, complete sentences, punctuation. Not coherent but clearly language-like.

If val_loss stops improving but train_loss keeps falling: overfitting. Increase dropout or reduce model size.

If both losses are high after 2000 steps: learning rate may be too low or too high. Try 1e-3 or 1e-4.

Scaling up: the path to GPT-scale

The only differences between your model and GPT-3

The architecture is identical. The differences are purely in scale:

  • Your model: d_model=128, n_layers=4, n_heads=4, vocab=65, block=128 → ~800K parameters
  • GPT-2 small: d_model=768, n_layers=12, n_heads=12, vocab=50257, block=1024 → 117M parameters
  • GPT-3: d_model=12288, n_layers=96, n_heads=96, vocab=50257, block=2048 → 175B parameters
  • Training data: Your model: ~1MB text. GPT-3: 570GB of text (~500,000× more).
  • Compute: Your model: one laptop CPU. GPT-3: ~10,000 A100 GPUs for months.

The code you have would produce GPT-2 if you changed 4 numbers, got a 570GB dataset, and had ~$500,000 of GPU compute.

What you have built: A decoder-only Transformer language model — the same fundamental architecture as every major LLM in production today. The attention mechanism, the residual blocks, the layer norm, the causal masking, the cross-entropy loss, the autoregressive generation — it's all there, in ~280 lines. Scale is an engineering and resource problem; the architecture is what you now understand completely.

Complete concept reference

Token
Atomic unit of text. Character-level (65 vocab) or BPE sub-word (50k vocab). Integer IDs fed to the model.
Embedding
Learned dense vector representing a token. Shape: [vocab_size, d_model]. Converts int IDs to continuous vectors.
Query / Key / Value
Three learned projections of each token. Q = "what am I looking for?", K = "what do I contain?", V = "what do I give?"
Scaled dot-product attention
softmax(QKᵀ/√d_k)·V. The core operation. O(T²) per layer.
Causal mask
Upper-triangular -inf matrix applied before softmax. Prevents tokens from attending to future positions.
Multi-head attention
h parallel attention heads, concatenated and projected. Learns different relationship types simultaneously.
Positional encoding
Position-dependent vectors added to token embeddings. Gives the model information about token order.
Feedforward network
2-layer MLP per token, 4× expansion. Where the model's "factual knowledge" is believed to reside.
Layer normalisation
Normalises activations per token across features. Stabilises training. Applied before each sub-layer (Pre-Norm).
Residual connection
x = x + sublayer(x). Direct gradient pathway. Enables training of very deep networks.
Cross-entropy loss
-log(P(correct_token)). Low when model is confident and correct, high when wrong. Baseline: log(vocab_size).
Backpropagation
Chain rule applied to compute ∂loss/∂every_parameter. PyTorch autograd does this automatically.
AdamW
Adaptive optimiser with decoupled weight decay. Standard for Transformer training. lr=3e-4 is a good starting point.
Temperature
Divides logits before softmax. <1 = more deterministic. >1 = more random. 0.8 is a good default.
Top-k sampling
Restrict sampling to k most probable tokens. Prevents low-probability garbage. k=40 is a good default.
Autoregressive generation
Sample one token, append, repeat. The only way to generate with a decoder-only model.