imara learning library ← all courses
AI Math · Foundation Models · self-paced

The math you need to read a Transformer

Eight rungs, from "what does ℝB×L×d mean" to reading a real survival-analysis loss. No prerequisite beyond arithmetic. By the end, the formulas in a modern foundation-model spec — dot products, softmax, cross-entropy, InfoNCE, hazards — read like sentences.

8
rungs, each depending on the last
5
interactive widgets — drag the math
0 prereq
beyond multiply, add, and logarithms
1 goal
read real research formulas unaided
Rung 01the alphabet

Reading the notation itself

Most formulas look scary only because the symbols are unfamiliar, not because the ideas are hard. Learn the alphabet first and half the intimidation disappears.

Shapes: what ℝB×L×d means

Neural networks pass around blocks of numbers. The notation just states the block's dimensions.

ze ∈ ℝB×L×d"z_e is a block of real numbers shaped Batch × Length × dimension"

Read it right to left as nesting: d numbers make one vector; L of those make one sequence; B of those make one batch. So ℝB×L×d is "B customers, each a sequence of L events, each event a d-number vector." The ∈ means "is a member of / has the type." That is the entire content of the symbol.

The linear layer: Wx + b

The single most common operation in any network. Take an input vector x, multiply by a learned matrix W, add a learned offset vector b.

u = W x + ba "linear map" or "projection" — reshape and reweight the input

A matrix multiply is just "compute several weighted sums of the inputs at once." Each output number is (row of W) · x + b. Every "head," "projection," and "encoder layer" is built from stacks of these. When you see Wλ LN(ht) + bλ, it is one linear layer applied to a normalised input — nothing more.

The summation and product signs

Σk ak = a1 + a2 + …     ∏k ak = a1 × a2 × …Σ adds a list up; ∏ multiplies a list together

The little k ≤ d or j=1 under the sign says which items to include. That's it. When you later meet S(d) = ∏k≤d(1 − λk), you already know it means "multiply (1−λk) together for every day k up to d."

A few more marks

SymbolReads asExample
xksubscript = "the k-th entry"uk = day-k logit
q(a)superscript = a label, not a powerquantiles for action-type a
[a ; b]concatenate — glue vectors end to end[h1;…;hF]
xᵀtranspose — flip a row/column (sets up a dot product)qᵀv
‖x‖norm — the length of vector xused in cosine, Rung 02
|S|size of set S (how many items)(1/|S|) Σ = average over S
You can now read

The shape line from your spec — ze = tok_out[:, 0] → reshape ∈ ℝB×L×d — says "take slot 0 of every event's token output, and arrange the result as Batch × Length × dimension." Pure bookkeeping.

Rung 02the core idea

Similarity: the dot product and cosine

This is the most important primitive in the entire field. Attention, contrastive learning, segmentation and lookalike scoring all reduce to one question — how similar are two vectors? — and the answer is always a dot product.

The dot product

u · v = uᵀv = u1v1 + u2v2 + … + udvdmultiply matching entries, then add them all up → one number

That single number is a similarity score. It is large and positive when the two vectors point the same way, near zero when they're unrelated, and negative when they point apart. Whenever a model asks "how relevant is A to B?", a dot product is doing the asking.

Worked number. u=[2,1], v=[1,3] → 2·1 + 1·3 = 5. Positive, so they broadly agree.

Cosine similarity — direction only

The raw dot product grows if you just make the vectors longer, which is often noise. Divide by both lengths to strip out magnitude and keep only direction:

cos(u,v) = (uᵀv) / (‖u‖ · ‖v‖)ranges from −1 (opposite) through 0 (unrelated) to +1 (same direction)

where ‖u‖ = √(u1² + u2² + …) is the vector's length. Drag the two vectors below and watch the dot product and cosine respond — this is worth building intuition for, because it is literally how customers get compared for segmentation.

Drag the arrowheads · dot product & cosine update live
Point them the same way → cosine near +1. At right angles → 0. Opposite → −1. Length changes the dot product but not the cosine.
You can now read

Your affinity formula score(u) = (1/|S|) Σv∈S (z̃uᵀz̃v)/(‖z̃u‖‖z̃v‖) is just "a customer's score = their average cosine similarity to everyone in set S." A lookalike score, built entirely from embedding geometry.

Rung 03turning scores into meaning

The squashers: logits, sigmoid, softmax

Models compute raw scores, then squash them into probabilities. Three functions do all the squashing. Knowing which does what unlocks most loss functions.

Logit → the raw score

A logit is just a score before it's been turned into a probability — any real number, from −∞ to +∞. "The model outputs logits" means "numbers not yet squashed." The u in u = WλLN(ht)+bλ is a logit.

Sigmoid — one number → one probability

σ(x) = 1 / (1 + e−x)any real number in → a probability between 0 and 1 out

Big positive → near 1, big negative → near 0, zero → exactly 0.5. Use it for a single yes/no probability. σ(0)=0.50, σ(2)=0.88, σ(−2)=0.12. Drag the logit:

Sigmoid · slide the logit, read the probability
σ(x) = 0.500
This is exactly λk = σ(uk) in a hazard head: a raw day-k score becomes a day-k probability.

Softmax — a list of numbers → a distribution

When there are several competing options, softmax turns their logits into probabilities that sum to 1:

softmax(z)i = ezi/τ / Σj ezj/τexponentiate each score, then normalise so they add to 1

The temperature τ controls sharpness: small τ makes the winner dominate (decisive), large τ flattens everything (uncertain). This is the same normalisation used inside attention (with τ = √dk) and, as you'll see next, hidden inside InfoNCE. Adjust the logits and temperature:

Softmax · three logits + temperature → probabilities
Rung 04how models learn

Cross-entropy: the workhorse loss

A loss is one number saying "how wrong is the model right now?" Training nudges the weights to shrink it. For "pick the right category," that number is cross-entropy.

CE = − log( ptrue )negative log of the probability the model gave the CORRECT answer

The logic is beautifully simple. If the model was confident and right (ptrue ≈ 1), then −log(1) = 0 — no penalty. If it was confident and wrong (ptrue ≈ 0), then −log(0) = ∞ — enormous penalty. So it rewards confident correctness and punishes confident mistakes, hard. Drag the predicted probability of the true class:

Cross-entropy · confidence in the true answer vs the penalty
loss = −log(p) = 0.105
Notice the asymmetry: going from 0.9 → 0.99 barely helps, but 0.1 → 0.01 is punished savagely. Confidence in a wrong answer is the worst thing a model can do.

Worked numbers. Sure and right, p=0.9 → loss 0.105. Unsure, p=0.5 → 0.69. Confidently wrong, p=0.1 → 2.30.

You can now read

The multi-field training loss Ltotal = Σf wf · CEf + λsurv · LNLL reads: "add up the cross-entropy for each field f (weighted by importance wf), then add the survival loss (weighted by λsurv)." One model, many objectives, each on its own dial. You'll meet LNLL at Rung 07.

Rung 05learning without labels

Contrastive learning & InfoNCE

InfoNCE looks like the scariest formula in your spec. It is not — it's softmax and cross-entropy wearing a costume. Once you see the disguise, it's a multiple-choice quiz.

LInfoNCE = − log   [ es(q, v⁺) ] / [ es(q, v⁺) + Σj es(q, v⁻j) ]make the right match score higher than the K wrong ones

Read the fraction: v⁺ is the right answer (the true counterparty), v⁻j are K decoys (wrong answers sampled at random), and s(q,v) is a similarity score — a scaled dot product qᵀv/τ from Rungs 02–03. The fraction is exactly softmax: "the right answer's score over all scores" = the probability the model assigns to the correct match. Wrapping it in −log(…) is exactly cross-entropy from Rung 04.

So the whole objective means: pull true pairs together, push mismatches apart. That's contrastive learning — and it's how you learn useful representations with no hand-labels, just "these two things genuinely co-occurred, those don't."

Two tricks in the same spec

C = hash(counterparty_id) mod 50,000 — millions of counterparties won't fit in an embedding table, so hashing maps each id into one of 50,000 buckets (rare collisions, acceptable). And Pneg(c) ∝ fcα, α=0.75 — pick decoys roughly by popularity, softened by the 0.75 power so common counterparties don't dominate. Both are standard, borrowed straight from word2vec. Your own merchant/market vocab will hit the same wall at scale — hashing is the answer.

Rung 06numbers into tokens

Quantile bucketing & argmax

The trick that lets a discrete model handle money without shredding it into digits. You met it in the PRAGMA course as percentile buckets; here is the formal version.

b(x | a) = argmaxj { qj(a) ≤ x },   j = 1, …, 32"which of the 32 buckets does amount x fall into?"

Unpack it. The qj(a) are 32 quantile boundaries learned from the data for action-type a — the amounts that split all transactions into 32 equal-frequency bins. argmaxj means "the index j that maximises the condition," and the condition { qj ≤ x } is true for every boundary still below x. So the whole thing returns the highest bucket whose boundary x has passed — i.e., x's bucket number.

That's the entire meaning of the np.searchsorted call in the training project. argmax / argmin in general returns the index that maximises/minimises something, not the value — a very common operator worth memorising.

Why bother: a bucket index is a single token that preserves ordering and magnitude (bucket 30 > bucket 5 means bigger), unlike text tokenisation which would turn 14.99 into meaningless fragments. Zero usually gets its own bucket 0.

Rung 07the genuinely new one

Survival analysis: modelling when, not just whether

This is the hardest rung and the one with no equivalent in the PRAGMA course. It answers a different question: not "will this loan default?" but "on which day will it resolve, and what's the chance it's still healthy at day 60?" It's how you predict timing under censored data.

Hazard: the building block

λk = σ(uk) = P(T = k | T ≥ k)chance the loan resolves on day k, GIVEN it was still open entering day k

Read the bar | as "given." The hazard λk is not the raw chance of resolving on day k — it's the conditional chance given survival so far. Each day gets its own hazard, produced by a sigmoid (Rung 03) on a day-k logit.

Survival: chain the daily odds together

S(d) = ∏k≤d (1 − λk)     pDPD30 = S(60)probability of surviving every day up to d — the reported number is S at day 60

Surviving to day d means surviving day 1 and day 2 and … day d. "And" for independent probabilities means multiply — hence the product ∏ of the per-day survival odds (1−λk). The reported risk number is simply the survival probability at a chosen horizon. Build the curve — set a baseline daily hazard and an early-days spike, and watch S(60) emerge:

Survival curve · per-day hazards compound into S(d)
S(60) = psurvive-to-60 = —
a tiny daily hazard still compounds over 60 days
Push the daily hazard up a little and watch S(60) fall a lot — that compounding is the whole point of the product.

The likelihood, and censoring

LNLL = − [ Σk=1T log(1 − λk) + δ · log λT ]negative log-likelihood of the observed survival pattern

Two parts. Σ log(1−λk) is the log-probability of surviving every day the loan was actually open — summing logs is the same as multiplying the probabilities (that's just log(a·b)=log a+log b, turning the survival product into a sum). The second term, δ · log λT, only fires when δ = 1, i.e. the event actually happened at day T.

Censoring — the subtle, important bit

δ = 0 marks a censored loan — one still open when your data ended, so you never saw it resolve. It contributes its at-risk days (the survival sum) but no event term. This is "censoring by omission, never imputation": you don't guess what would have happened, you just count what you observed. Getting this right is what separates a real survival model from a naive classifier that would wrongly treat "not yet defaulted" as "repaid."

Rung 08it all comes together

Reading the architecture, assembled

Every symbol now has a meaning. Here are the architecture formulas from a real spec, read end to end — no new math, just the primitives from Rungs 01–07 combined.

The profile encoder

tok = [USR, p1, …, pP]    za = ProfileEnc(tok)[:, 0]prepend a [USR] slot, encode, keep slot 0 as the profile vector

Glue a special [USR] token in front of the profile fields (Rung 01 concat), run the bidirectional encoder, and read out position 0 — the [USR] slot, which has aggregated the whole profile into one vector. The key-padding mask you saw is just Rung 03's "set missing slots to −∞ so softmax ignores them."

Event pooling — an MLP, not attention

ze = W2 φ( W1[h1; … ; hF] ),   φ = GELUconcatenate the fields, push through a 2-layer MLP

Concatenate the field vectors (Rung 01), then apply two linear layers (Rung 01) with a GELU between them. GELU is just a smooth "squasher" that lets stacked linear layers model curves instead of collapsing into one flat map. The design note is a real engineering call: with only ~8 fields per event, full attention isn't worth its cost across billions of events, so a cheap MLP pools them instead.

The history encoder — one vector per customer

zu = HistEnc( [USR], ze₁, …, zeₙ )contextualise profile + all events → the customer embedding

Feed the profile vector and every event vector into the final encoder; its output is the single per-customer vector you cluster for segments or score with a head. Same three-encoder shape as the PRAGMA course — you've now read it in full mathematical detail.

The whole toolkit, in one breath

Shapes and linear maps (01) build the layers. Dot products (02) measure similarity. Squashers (03) turn scores into probabilities. Cross-entropy (04) and InfoNCE (05) are the losses that teach it. Bucketing (06) gets numbers in; survival (07) gets timing out. Assembled (08), they're a foundation model. Every research formula you'll meet is a recombination of these seven ideas.

Where to go next