By the end of this course you will have built a working Transformer language model in Python, trained it on real text, and generated new text from it. The same architecture powers GPT-4 — scaled up by a factor of roughly 100,000.
A language model has one job: given a sequence of tokens, predict the probability distribution over what token comes next. That is the complete task definition. Every capability that emerges from large language models — reasoning, coding, translation, summarisation — arises from doing this task well on vast amounts of text.
Text is not fed to a language model as characters or words. It is broken into tokens — sub-word units that balance vocabulary size against sequence length. Modern LLMs use Byte-Pair Encoding (BPE), which starts with individual characters and repeatedly merges the most frequent adjacent pairs until a target vocabulary size is reached.
Training a language model means learning to predict the next token given all previous tokens. Given the sequence "The loan was", the model should assign high probability to "approved", "denied", "processed" etc. and low probability to nonsense.
The model can only see a fixed number of previous tokens — the context window (also called block size or sequence length). For our small model we will use 128 or 256 tokens. GPT-4 uses 128,000. The architecture is identical; the difference is scale.
Type text to see it broken into character-level tokens and their integer IDs. This is exactly what your model will receive as input.
Token IDs (integers) cannot be fed directly into a neural network. They are first converted to embedding vectors — dense lists of floating-point numbers that the model learns during training. Each token in the vocabulary gets its own embedding vector of size d_model (e.g. 128 or 512).
A training sequence "hello" of length 5 generates how many (input, target) pairs for next-token prediction?
Attention is the mechanism that allows a language model to relate any token to any other token in the sequence, regardless of distance. Before attention, RNNs had to pass information through a chain of hidden states — information from early in a sequence degraded as it propagated forward. Attention solves this by allowing the model to directly "look at" any position when computing its representation.
Attention is often described with a retrieval analogy. Imagine you are in a library (the sequence of tokens). You have a question — your Query (Q). Each book in the library has a spine with index information — the Key (K). You scan the keys, find the books most relevant to your query, and read their content — the Value (V). The output is a weighted mixture of values, where the weights come from how well each key matched your query.
Language models are autoregressive — they generate tokens left to right. During training, position 5 should not be able to see tokens at positions 6, 7, 8... (that would be cheating — the answer is in the input). We enforce this with a causal mask: setting attention scores for future positions to -∞ before softmax, which makes their weights exactly 0.
Click a token to see which other tokens it attends to most strongly. Brighter cells = higher attention weight. The upper triangle is masked (future tokens cannot be seen).
A single attention head learns one type of relationship between tokens. Multi-head attention runs h attention heads in parallel, each with different learned Q/K/V projections, then concatenates their outputs. This allows the model to simultaneously attend to different types of relationships — syntactic, semantic, positional.
Softmax converts a vector of arbitrary real numbers into a probability distribution that sums to 1. It is used both in the attention mechanism and in the final output layer of the model.
Adjust raw attention scores to see how softmax converts them to probability weights. Notice how dominant scores suppress others exponentially.
The Transformer is a stack of identical blocks. Each block has two sub-layers: a multi-head attention layer (which relates tokens to each other) and a feedforward network (which transforms each token independently). Understanding these components and how they fit together is the complete architectural picture.
Attention is permutation-invariant — if you shuffle the tokens, the attention outputs shuffle in the same way, but the model has no way to know tokens were in a different order. Positional encodings inject position information by adding a position-dependent vector to each token embedding.
After attention has allowed tokens to communicate, each token is independently processed by a small 2-layer neural network. This is where most of the model's "factual knowledge" is believed to be stored. The FFN expands the dimension (typically by 4×), applies a non-linearity, then contracts back.
Layer normalisation stabilises training by normalising the activations across the feature dimension for each token independently. Unlike batch normalisation (which normalises across the batch), layer norm works on each position's vector and is well-suited to variable-length sequences.
Each sub-layer wraps its computation in a residual (skip) connection: the input is added to the output. This creates a direct gradient pathway through the network, allowing training of very deep models without vanishing gradients.
1. Token embedding: [B, T] integer IDs → [B, T, d_model] float vectors
2. Position embedding: Add position-dependent vectors → [B, T, d_model]
3. N × Transformer blocks: Each block: LayerNorm → MultiHeadAttention → Residual → LayerNorm → FFN → Residual
4. Final LayerNorm: Normalise the output of the last block
5. Output projection: [B, T, d_model] → [B, T, vocab_size] — one logit per token in vocabulary
6. Softmax (at inference): Logits → probabilities for next token
Visualise how tensor shapes change as data flows through the model. Adjust model size parameters.
Why does the feedforward network in a Transformer expand to 4×d_model before contracting back? What purpose does this serve?
Now we build. Each component is a PyTorch nn.Module. We build them bottom-up: attention head → multi-head attention → transformer block → full model. Read each block carefully — every line corresponds to a concept from modules 1–3.
import torch import torch.nn as nn from torch.nn import functional as F # ── Hyperparameters ───────────────────────────────────── # Small model: trains in ~10 min on CPU, ~1 min on GPU batch_size = 32 # sequences processed in parallel block_size = 128 # context window (tokens the model can see) d_model = 128 # embedding dimension n_heads = 4 # attention heads n_layers = 4 # transformer blocks dropout = 0.1 # regularisation learning_rate = 3e-4 max_iters = 5000 eval_interval = 500 device = 'cuda' if torch.cuda.is_available() else 'cpu' print(f'Using device: {device}')
class Head(nn.Module): """One head of self-attention.""" def __init__(self, head_size): super().__init__() # Projections for Q, K, V — no bias (common in transformers) self.key = nn.Linear(d_model, head_size, bias=False) self.query = nn.Linear(d_model, head_size, bias=False) self.value = nn.Linear(d_model, head_size, bias=False) # Causal mask: register as buffer (not a parameter, but saved with model) self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size))) self.dropout = nn.Dropout(dropout) def forward(self, x): B, T, C = x.shape # batch, time (seq len), channels (d_model) k = self.key(x) # (B, T, head_size) q = self.query(x) # (B, T, head_size) v = self.value(x) # (B, T, head_size) # Attention scores: Q · Kᵀ / √head_size scale = k.shape[-1] ** -0.5 # 1/√head_size wei = q @ k.transpose(-2, -1) * scale # (B, T, T) # Apply causal mask: future positions → -inf wei = wei.masked_fill( self.tril[:T, :T] == 0, float('-inf')) # (B, T, T) # Softmax: turn scores into probabilities wei = F.softmax(wei, dim=-1) # (B, T, T) wei = self.dropout(wei) # Weighted aggregation of values out = wei @ v # (B, T, head_size) return out
class MultiHeadAttention(nn.Module): """Multiple attention heads running in parallel.""" def __init__(self, n_heads, head_size): super().__init__() self.heads = nn.ModuleList( [Head(head_size) for _ in range(n_heads)]) # Project concatenated head outputs back to d_model self.proj = nn.Linear(n_heads * head_size, d_model) self.dropout = nn.Dropout(dropout) def forward(self, x): # Run each head, concatenate along the feature dimension out = torch.cat([h(x) for h in self.heads], dim=-1) out = self.dropout(self.proj(out)) return out
class FeedForward(nn.Module): """Position-wise feedforward: expand → non-linearity → contract.""" def __init__(self, d_model): super().__init__() self.net = nn.Sequential( nn.Linear(d_model, 4 * d_model), # expand 4× nn.GELU(), # smooth non-linearity nn.Linear(4 * d_model, d_model), # contract back nn.Dropout(dropout), ) def forward(self, x): return self.net(x) class Block(nn.Module): """One Transformer block: communication (attention) + computation (FFN).""" def __init__(self, d_model, n_heads): super().__init__() head_size = d_model // n_heads # each head gets equal share self.attn = MultiHeadAttention(n_heads, head_size) self.ffn = FeedForward(d_model) # Pre-norm: LayerNorm before each sub-layer (GPT-2 style) self.ln1 = nn.LayerNorm(d_model) self.ln2 = nn.LayerNorm(d_model) def forward(self, x): # Residual connection: x = x + sublayer(LayerNorm(x)) x = x + self.attn(self.ln1(x)) # attention + residual x = x + self.ffn(self.ln2(x)) # feedforward + residual return x
class GPT(nn.Module): """Decoder-only Transformer (GPT architecture).""" def __init__(self, vocab_size): super().__init__() # 1. Token embedding table self.tok_embed = nn.Embedding(vocab_size, d_model) # 2. Positional embedding table (learned) self.pos_embed = nn.Embedding(block_size, d_model) # 3. Stack of Transformer blocks self.blocks = nn.Sequential( *[Block(d_model, n_heads) for _ in range(n_layers)]) # 4. Final layer norm self.ln_f = nn.LayerNorm(d_model) # 5. Output projection: d_model → vocab_size (logits) self.lm_head = nn.Linear(d_model, vocab_size, bias=False) # Weight tying: share token embedding and output projection weights # (reduces parameters, improves performance — used in GPT-2) self.tok_embed.weight = self.lm_head.weight self.apply(self._init_weights) def _init_weights(self, module): # Small initial weights → stable training if isinstance(module, nn.Linear): torch.nn.init.normal_(module.weight, mean=0.0, std=0.02) if module.bias is not None: torch.nn.init.zeros_(module.bias) elif isinstance(module, nn.Embedding): torch.nn.init.normal_(module.weight, mean=0.0, std=0.02) def forward(self, idx, targets=None): B, T = idx.shape # 1. Token + position embeddings tok_emb = self.tok_embed(idx) # (B, T, d_model) pos_emb = self.pos_embed( torch.arange(T, device=device)) # (T, d_model) x = tok_emb + pos_emb # (B, T, d_model) # 2. Transformer blocks x = self.blocks(x) # (B, T, d_model) # 3. Final norm + project to vocabulary x = self.ln_f(x) # (B, T, d_model) logits = self.lm_head(x) # (B, T, vocab_size) # 4. Compute loss if targets provided (training mode) if targets is None: return logits, None # Cross-entropy loss over all positions B, T, V = logits.shape logits_2d = logits.view(B*T, V) targets_1d = targets.view(B*T) loss = F.cross_entropy(logits_2d, targets_1d) return logits, loss
See exactly how many parameters your model has and where they live. Adjust to understand how scale is achieved.
Training is the process of adjusting model weights to minimise the loss — the gap between what the model predicts and what the correct next token actually is. This requires three components: a loss function, a method to compute gradients (backpropagation), and an optimiser to update weights.
The loss function for language models is cross-entropy — it measures how surprised the model is by the correct next token. If the model assigns high probability to the correct token, the loss is low. If it assigns low probability, the loss is high.
Backpropagation computes the gradient of the loss with respect to every parameter in the model — using the chain rule of calculus. PyTorch does this automatically via its autograd engine. You call loss.backward() and every parameter's .grad attribute is populated.
AdamW is the standard optimiser for Transformers. It adapts the learning rate for each parameter based on the history of gradients, and applies weight decay (L2 regularisation) correctly — separately from the gradient step, which is why it's called AdamW (decoupled weight decay).
import torch # ── Data preparation ───────────────────────────────────── with open('input.txt', 'r') as f: text = f.read() # Build character vocabulary chars = sorted(list(set(text))) vocab_size = len(chars) stoi = {ch:i for i,ch in enumerate(chars)} # char → int itos = {i:ch for i,ch in enumerate(chars)} # int → char encode = lambda s: [stoi[c] for c in s] decode = lambda l: ''.join([itos[i] for i in l]) # Train/validation split data = torch.tensor(encode(text), dtype=torch.long) n = int(0.9 * len(data)) train_data = data[:n] val_data = data[n:] def get_batch(split): """Sample a batch of (inputs, targets) from the dataset.""" data = train_data if split == 'train' else val_data # Random starting positions for each sequence in the batch ix = torch.randint(len(data) - block_size, (batch_size,)) x = torch.stack([data[i : i+block_size ] for i in ix]) y = torch.stack([data[i+1 : i+block_size+1] for i in ix]) return x.to(device), y.to(device) # ── Model and optimiser ────────────────────────────────── model = GPT(vocab_size).to(device) optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate) print(f'Parameters: {sum(p.numel() for p in model.parameters()):,}') # ── Training loop ──────────────────────────────────────── for step in range(max_iters): # Periodically evaluate on validation set if step % eval_interval == 0: model.eval() with torch.no_grad(): xv, yv = get_batch('val') _, val_loss = model(xv, yv) model.train() print(f'step {step:5d}: val_loss = {val_loss.item():.4f}') # Forward pass xb, yb = get_batch('train') logits, loss = model(xb, yb) # Backward pass optimizer.zero_grad(set_to_none=True) # clear old gradients loss.backward() # compute new gradients optimizer.step() # update weights print('Training complete!')
See how different hyperparameters affect the training curve. Identify overfitting, underfitting, and good training runs.
After training, the model can generate text. Generation is autoregressive: sample one token, append it to the context, sample the next, repeat. The interesting choices are in how you sample from the output distribution — these choices dramatically affect output quality, creativity, and coherence.
def generate(model, idx, max_new_tokens, temperature=1.0, top_k=None): """ Generate text autoregressively. idx: (B, T) tensor of starting tokens Returns: (B, T + max_new_tokens) tensor """ model.eval() with torch.no_grad(): for _ in range(max_new_tokens): # Crop context to block_size (can't exceed the window) idx_cond = idx[:, -block_size:] # Forward pass → logits for next token logits, _ = model(idx_cond) logits = logits[:, -1, :] # take last position: (B, vocab_size) # ── Sampling strategies ──────────────────────── # 1. Temperature: reshape the distribution logits = logits / temperature # 2. Top-k: keep only the k most likely tokens if top_k is not None: vals, _ = torch.topk(logits, top_k) logits[logits < vals[:, [-1]]] = float('-inf') # 3. Convert to probabilities probs = F.softmax(logits, dim=-1) # 4. Sample from the distribution idx_next = torch.multinomial(probs, num_samples=1) # 5. Append and continue idx = torch.cat((idx, idx_next), dim=1) return idx # Usage: generate 500 characters starting from newline context = torch.zeros((1, 1), dtype=torch.long, device=device) output = generate(model, context, max_new_tokens=500, temperature=0.8, top_k=40) print(decode(output[0].tolist()))
Adjust temperature and top-k to see how they reshape the probability distribution for the next token. This is exactly what happens at each step of generation.
This is everything in one file. Copy it, save as gpt.py, get a text file (input.txt), and run it. On CPU it will produce readable output within 10–20 minutes. On a GPU, within 2–3 minutes.
pip install torch. Get a training text file — the complete works of Shakespeare (download here, ~1MB) works perfectly. Save it as input.txt in the same directory.
""" GPT from scratch — character-level language model Based on: Vaswani et al. "Attention is All You Need" (2017) Karpathy "nanoGPT" (2022) Usage: python gpt.py # train on input.txt and generate python gpt.py --generate # load saved model and generate only Training corpus: put any .txt file as input.txt Recommended: Tiny Shakespeare (~1MB) wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt """ import torch import torch.nn as nn from torch.nn import functional as F import time, argparse # ════════════════════════════════════════════════════════════ # HYPERPARAMETERS — adjust to your compute budget # ════════════════════════════════════════════════════════════ batch_size = 32 # sequences per batch block_size = 128 # context window (tokens) d_model = 128 # embedding dimension n_heads = 4 # attention heads per block n_layers = 4 # transformer blocks dropout = 0.1 # dropout rate learning_rate = 3e-4 # AdamW learning rate max_iters = 5000 # training steps eval_interval = 500 # evaluate every N steps save_path = 'gpt_model.pt' device = 'cuda' if torch.cuda.is_available() else 'cpu' # ════════════════════════════════════════════════════════════ # DATA # ════════════════════════════════════════════════════════════ def load_data(path='input.txt'): with open(path, 'r', encoding='utf-8') as f: text = f.read() chars = sorted(set(text)) stoi = {c:i for i,c in enumerate(chars)} itos = {i:c for i,c in enumerate(chars)} encode = lambda s: [stoi[c] for c in s] decode = lambda l: ''.join(itos[i] for i in l) data = torch.tensor(encode(text), dtype=torch.long) n = int(0.9*len(data)) return data[:n], data[n:], len(chars), encode, decode def get_batch(data, bsz=batch_size, blk=block_size): ix = torch.randint(len(data)-blk, (bsz,)) x = torch.stack([data[i :i+blk ] for i in ix]) y = torch.stack([data[i+1:i+blk+1] for i in ix]) return x.to(device), y.to(device) # ════════════════════════════════════════════════════════════ # MODEL COMPONENTS # ════════════════════════════════════════════════════════════ class Head(nn.Module): def __init__(self, head_size): super().__init__() self.key = nn.Linear(d_model, head_size, bias=False) self.query = nn.Linear(d_model, head_size, bias=False) self.value = nn.Linear(d_model, head_size, bias=False) self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size))) self.dropout = nn.Dropout(dropout) def forward(self, x): B,T,C = x.shape k = self.key(x); q = self.query(x); v = self.value(x) wei = q @ k.transpose(-2,-1) * k.shape[-1]**-0.5 wei = wei.masked_fill(self.tril[:T,:T]==0, float('-inf')) wei = self.dropout(F.softmax(wei, dim=-1)) return wei @ v class MultiHeadAttention(nn.Module): def __init__(self, n_heads, head_size): super().__init__() self.heads = nn.ModuleList([Head(head_size) for _ in range(n_heads)]) self.proj = nn.Linear(n_heads*head_size, d_model) self.drop = nn.Dropout(dropout) def forward(self, x): return self.drop(self.proj(torch.cat([h(x) for h in self.heads], dim=-1))) class FeedForward(nn.Module): def __init__(self): super().__init__() self.net = nn.Sequential( nn.Linear(d_model, 4*d_model), nn.GELU(), nn.Linear(4*d_model, d_model), nn.Dropout(dropout)) def forward(self, x): return self.net(x) class Block(nn.Module): def __init__(self): super().__init__() hs = d_model // n_heads self.attn = MultiHeadAttention(n_heads, hs) self.ffn = FeedForward() self.ln1 = nn.LayerNorm(d_model) self.ln2 = nn.LayerNorm(d_model) def forward(self, x): x = x + self.attn(self.ln1(x)) x = x + self.ffn(self.ln2(x)) return x class GPT(nn.Module): def __init__(self, vocab_size): super().__init__() self.tok_embed = nn.Embedding(vocab_size, d_model) self.pos_embed = nn.Embedding(block_size, d_model) self.blocks = nn.Sequential(*[Block() for _ in range(n_layers)]) self.ln_f = nn.LayerNorm(d_model) self.lm_head = nn.Linear(d_model, vocab_size, bias=False) self.tok_embed.weight = self.lm_head.weight # weight tying self.apply(self._init) def _init(self, m): if isinstance(m, (nn.Linear, nn.Embedding)): nn.init.normal_(m.weight, 0, 0.02) if isinstance(m, nn.Linear) and m.bias is not None: nn.init.zeros_(m.bias) def forward(self, idx, targets=None): B,T = idx.shape x = (self.tok_embed(idx) + self.pos_embed(torch.arange(T, device=device))) x = self.ln_f(self.blocks(x)) logits = self.lm_head(x) loss = None if targets is not None: B,T,V = logits.shape loss = F.cross_entropy(logits.view(B*T,V), targets.view(B*T)) return logits, loss def generate(self, idx, max_new_tokens, temperature=1.0, top_k=40): self.eval() with torch.no_grad(): for _ in range(max_new_tokens): logits, _ = self(idx[:, -block_size:]) logits = logits[:, -1, :] / temperature if top_k is not None: v, _ = torch.topk(logits, top_k) logits[logits < v[:, [-1]]] = float('-inf') idx_next = torch.multinomial(F.softmax(logits, -1), 1) idx = torch.cat((idx, idx_next), dim=1) return idx # ════════════════════════════════════════════════════════════ # MAIN # ════════════════════════════════════════════════════════════ if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('--generate', action='store_true') args = parser.parse_args() train_data, val_data, vocab_size, encode, decode = load_data() model = GPT(vocab_size).to(device) print(f'Device: {device} | Vocab: {vocab_size} | ' f'Params: {sum(p.numel() for p in model.parameters()):,}') if args.generate: model.load_state_dict(torch.load(save_path, map_location=device)) else: # Train opt = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=0.1) t0 = time.time() for step in range(1, max_iters+1): xb, yb = get_batch(train_data) _, loss = model(xb, yb) opt.zero_grad(True); loss.backward(); opt.step() if step % eval_interval == 0: model.eval() with torch.no_grad(): xv, yv = get_batch(val_data) _, vloss = model(xv, yv) model.train() elapsed = time.time() - t0 print(f'step {step:5d} | train {loss.item():.4f} | ' f'val {vloss.item():.4f} | {elapsed:.0f}s') torch.save(model.state_dict(), save_path) print(f'Model saved to {save_path}') # Generate start = torch.zeros((1,1), dtype=torch.long, device=device) output = model.generate(start, max_new_tokens=500, temperature=0.8, top_k=40) print('\n──── GENERATED TEXT ────\n') print(decode(output[0].tolist()))
If val_loss stops improving but train_loss keeps falling: overfitting. Increase dropout or reduce model size.
If both losses are high after 2000 steps: learning rate may be too low or too high. Try 1e-3 or 1e-4.
The architecture is identical. The differences are purely in scale:
The code you have would produce GPT-2 if you changed 4 numbers, got a 570GB dataset, and had ~$500,000 of GPU compute.