Eight rungs, from "what does ℝB×L×d mean" to reading a real survival-analysis loss. No prerequisite beyond arithmetic. By the end, the formulas in a modern foundation-model spec — dot products, softmax, cross-entropy, InfoNCE, hazards — read like sentences.
Most formulas look scary only because the symbols are unfamiliar, not because the ideas are hard. Learn the alphabet first and half the intimidation disappears.
Neural networks pass around blocks of numbers. The notation just states the block's dimensions.
Read it right to left as nesting: d numbers make one vector; L of those make one sequence; B of those make one batch. So ℝB×L×d is "B customers, each a sequence of L events, each event a d-number vector." The ∈ means "is a member of / has the type." That is the entire content of the symbol.
The single most common operation in any network. Take an input vector x, multiply by a learned matrix W, add a learned offset vector b.
A matrix multiply is just "compute several weighted sums of the inputs at once." Each output number is (row of W) · x + b. Every "head," "projection," and "encoder layer" is built from stacks of these. When you see Wλ LN(ht) + bλ, it is one linear layer applied to a normalised input — nothing more.
The little k ≤ d or j=1 under the sign says which items to include. That's it. When you later meet S(d) = ∏k≤d(1 − λk), you already know it means "multiply (1−λk) together for every day k up to d."
| Symbol | Reads as | Example |
|---|---|---|
| xk | subscript = "the k-th entry" | uk = day-k logit |
| q(a) | superscript = a label, not a power | quantiles for action-type a |
| [a ; b] | concatenate — glue vectors end to end | [h1;…;hF] |
| xᵀ | transpose — flip a row/column (sets up a dot product) | qᵀv |
| ‖x‖ | norm — the length of vector x | used in cosine, Rung 02 |
| |S| | size of set S (how many items) | (1/|S|) Σ = average over S |
The shape line from your spec — ze = tok_out[:, 0] → reshape ∈ ℝB×L×d — says "take slot 0 of every event's token output, and arrange the result as Batch × Length × dimension." Pure bookkeeping.
This is the most important primitive in the entire field. Attention, contrastive learning, segmentation and lookalike scoring all reduce to one question — how similar are two vectors? — and the answer is always a dot product.
That single number is a similarity score. It is large and positive when the two vectors point the same way, near zero when they're unrelated, and negative when they point apart. Whenever a model asks "how relevant is A to B?", a dot product is doing the asking.
Worked number. u=[2,1], v=[1,3] → 2·1 + 1·3 = 5. Positive, so they broadly agree.
The raw dot product grows if you just make the vectors longer, which is often noise. Divide by both lengths to strip out magnitude and keep only direction:
where ‖u‖ = √(u1² + u2² + …) is the vector's length. Drag the two vectors below and watch the dot product and cosine respond — this is worth building intuition for, because it is literally how customers get compared for segmentation.
Your affinity formula score(u) = (1/|S|) Σv∈S (z̃uᵀz̃v)/(‖z̃u‖‖z̃v‖) is just "a customer's score = their average cosine similarity to everyone in set S." A lookalike score, built entirely from embedding geometry.
Models compute raw scores, then squash them into probabilities. Three functions do all the squashing. Knowing which does what unlocks most loss functions.
A logit is just a score before it's been turned into a probability — any real number, from −∞ to +∞. "The model outputs logits" means "numbers not yet squashed." The u in u = WλLN(ht)+bλ is a logit.
Big positive → near 1, big negative → near 0, zero → exactly 0.5. Use it for a single yes/no probability. σ(0)=0.50, σ(2)=0.88, σ(−2)=0.12. Drag the logit:
When there are several competing options, softmax turns their logits into probabilities that sum to 1:
The temperature τ controls sharpness: small τ makes the winner dominate (decisive), large τ flattens everything (uncertain). This is the same normalisation used inside attention (with τ = √dk) and, as you'll see next, hidden inside InfoNCE. Adjust the logits and temperature:
A loss is one number saying "how wrong is the model right now?" Training nudges the weights to shrink it. For "pick the right category," that number is cross-entropy.
The logic is beautifully simple. If the model was confident and right (ptrue ≈ 1), then −log(1) = 0 — no penalty. If it was confident and wrong (ptrue ≈ 0), then −log(0) = ∞ — enormous penalty. So it rewards confident correctness and punishes confident mistakes, hard. Drag the predicted probability of the true class:
Worked numbers. Sure and right, p=0.9 → loss 0.105. Unsure, p=0.5 → 0.69. Confidently wrong, p=0.1 → 2.30.
The multi-field training loss Ltotal = Σf wf · CEf + λsurv · LNLL reads: "add up the cross-entropy for each field f (weighted by importance wf), then add the survival loss (weighted by λsurv)." One model, many objectives, each on its own dial. You'll meet LNLL at Rung 07.
InfoNCE looks like the scariest formula in your spec. It is not — it's softmax and cross-entropy wearing a costume. Once you see the disguise, it's a multiple-choice quiz.
Read the fraction: v⁺ is the right answer (the true counterparty), v⁻j are K decoys (wrong answers sampled at random), and s(q,v) is a similarity score — a scaled dot product qᵀv/τ from Rungs 02–03. The fraction is exactly softmax: "the right answer's score over all scores" = the probability the model assigns to the correct match. Wrapping it in −log(…) is exactly cross-entropy from Rung 04.
So the whole objective means: pull true pairs together, push mismatches apart. That's contrastive learning — and it's how you learn useful representations with no hand-labels, just "these two things genuinely co-occurred, those don't."
C = hash(counterparty_id) mod 50,000 — millions of counterparties won't fit in an embedding table, so hashing maps each id into one of 50,000 buckets (rare collisions, acceptable). And Pneg(c) ∝ fcα, α=0.75 — pick decoys roughly by popularity, softened by the 0.75 power so common counterparties don't dominate. Both are standard, borrowed straight from word2vec. Your own merchant/market vocab will hit the same wall at scale — hashing is the answer.
The trick that lets a discrete model handle money without shredding it into digits. You met it in the PRAGMA course as percentile buckets; here is the formal version.
Unpack it. The qj(a) are 32 quantile boundaries learned from the data for action-type a — the amounts that split all transactions into 32 equal-frequency bins. argmaxj means "the index j that maximises the condition," and the condition { qj ≤ x } is true for every boundary still below x. So the whole thing returns the highest bucket whose boundary x has passed — i.e., x's bucket number.
That's the entire meaning of the np.searchsorted call in the training project. argmax / argmin in general returns the index that maximises/minimises something, not the value — a very common operator worth memorising.
Why bother: a bucket index is a single token that preserves ordering and magnitude (bucket 30 > bucket 5 means bigger), unlike text tokenisation which would turn 14.99 into meaningless fragments. Zero usually gets its own bucket 0.
This is the hardest rung and the one with no equivalent in the PRAGMA course. It answers a different question: not "will this loan default?" but "on which day will it resolve, and what's the chance it's still healthy at day 60?" It's how you predict timing under censored data.
Read the bar | as "given." The hazard λk is not the raw chance of resolving on day k — it's the conditional chance given survival so far. Each day gets its own hazard, produced by a sigmoid (Rung 03) on a day-k logit.
Surviving to day d means surviving day 1 and day 2 and … day d. "And" for independent probabilities means multiply — hence the product ∏ of the per-day survival odds (1−λk). The reported risk number is simply the survival probability at a chosen horizon. Build the curve — set a baseline daily hazard and an early-days spike, and watch S(60) emerge:
Two parts. Σ log(1−λk) is the log-probability of surviving every day the loan was actually open — summing logs is the same as multiplying the probabilities (that's just log(a·b)=log a+log b, turning the survival product into a sum). The second term, δ · log λT, only fires when δ = 1, i.e. the event actually happened at day T.
δ = 0 marks a censored loan — one still open when your data ended, so you never saw it resolve. It contributes its at-risk days (the survival sum) but no event term. This is "censoring by omission, never imputation": you don't guess what would have happened, you just count what you observed. Getting this right is what separates a real survival model from a naive classifier that would wrongly treat "not yet defaulted" as "repaid."
Every symbol now has a meaning. Here are the architecture formulas from a real spec, read end to end — no new math, just the primitives from Rungs 01–07 combined.
Glue a special [USR] token in front of the profile fields (Rung 01 concat), run the bidirectional encoder, and read out position 0 — the [USR] slot, which has aggregated the whole profile into one vector. The key-padding mask you saw is just Rung 03's "set missing slots to −∞ so softmax ignores them."
Concatenate the field vectors (Rung 01), then apply two linear layers (Rung 01) with a GELU between them. GELU is just a smooth "squasher" that lets stacked linear layers model curves instead of collapsing into one flat map. The design note is a real engineering call: with only ~8 fields per event, full attention isn't worth its cost across billions of events, so a cheap MLP pools them instead.
Feed the profile vector and every event vector into the final encoder; its output is the single per-customer vector you cluster for segments or score with a head. Same three-encoder shape as the PRAGMA course — you've now read it in full mathematical detail.
Shapes and linear maps (01) build the layers. Dot products (02) measure similarity. Squashers (03) turn scores into probabilities. Cross-entropy (04) and InfoNCE (05) are the losses that teach it. Bucketing (06) gets numbers in; survival (07) gets timing out. Assembled (08), they're a foundation model. Every research formula you'll meet is a recombination of these seven ideas.