Revolut trained a single model on raw banking events and beat their task-specific pipelines on credit, fraud, and more. This week distils what PRAGMA actually is, whether it makes sense for a platform like ours, and how to build a scaled-down version whose first job is customer segmentation.
All lifts are relative to Revolut's own baselines; absolute numbers were withheld as commercially sensitive, and the paper's worked examples use synthetic data.
PRAGMA is a family of encoder-style foundation models for banking event sequences. It reads a customer's ordered history of transactions, app taps, trades and messages, plus their profile state, and turns all of it into one dense vector that describes who that customer is, behaviourally.
The word foundation model is doing the heavy lifting. The idea, borrowed from language and vision, is to pre-train one large model on a broad corpus once, then adapt that same model cheaply to many narrow tasks. PRAGMA is the first credible version of that for consumer banking histories, and at up to a billion parameters it is the largest published encoder backbone of its kind.
Before PRAGMA, Revolut looked like most of the industry: a separate machine-learning pipeline for every question. One model for fraud, another for credit, another for churn, each with its own hand-crafted features, its own bespoke queries, its own maintenance burden. Every new question meant starting the feature-engineering grind again, and none of the models shared what they had learned about a customer.
If sequences of banking events carry transferable structure the way sentences do, then one pre-trained backbone should be able to feed all those tasks — replacing a fleet of hand-tuned models with a shared representation plus a small head per task.
This is the most common misread, so it is worth stating plainly. PRAGMA is encoder-only and bidirectional, in the BERT lineage, not the GPT one. It does not generate text or chat. It is trained by masking parts of a customer's history and learning to reconstruct them, which forces it to build a rich internal picture of normal behaviour. The output you use is an embedding — a fixed-length vector — not a sentence.
The team deliberately chose an encoder because the goal is discriminative: score this customer, classify this event, group these users. Generation would be the wrong tool.
A single pre-trained PRAGMA was evaluated across six tasks. The headline is not just that it works, but the size of the lifts on the hardest, rarest-signal problems:
| Downstream task | Headline metric | Relative lift |
|---|---|---|
| Credit scoring | PR-AUC | +130.2% |
| Communication engagement | PR-AUC | +79.4% |
| External fraud | Recall | +64.7% |
| Product recommendation | mAP | +40.5% |
| Recurrent transactions | F1 | +5.8% |
| Lifetime value | ROC-AUC | +2.6% |
Notice the pattern: the biggest wins are where the positive class is rare and hard to feature-engineer for (defaults, re-engagement, fraud). Where a task is already easy to model with a few good features (lifetime value, recurring subscriptions), the lift is small. That shape matters a lot for deciding where a model like this earns its keep — a point we return to on Day 4.
PRAGMA's real product is a reusable customer embedding learned from raw events with no labels. Every downstream task, including the segmentation we build this week, is just something you attach on top of that embedding.
To understand how to build our own, we need to see the two ideas that make PRAGMA work: a tokenisation scheme designed for structured events rather than text, and a three-encoder architecture that keeps who a customer is separate from what they did.
The obvious shortcut — serialise each transaction into a string and hand it to a normal transformer — fails for two concrete reasons. First, every field name and delimiter explodes into several subword tokens, so a long history becomes impossibly long. Second, and worse, numeric amounts get shredded into digit fragments that throw away magnitude and ordering. A model that can't tell that 14.99 and 1,499.00 differ in scale is useless for finance.
PRAGMA instead breaks every field into three parts: a semantic type (the key), a value, and a temporal coordinate. A card payment becomes a small structured object, not a sentence:
Each value type is handled the way it deserves:
Keys need only ~60 tokens of vocabulary; values need ~28k. Compact, and every token means something.
Timing is a first-class signal, so PRAGMA encodes it two ways. A smooth logarithmic transform of the gap since the previous event, 8·ln(1 + t/8), compresses years-long gaps while keeping seconds-level precision for recent activity. Separately, each timestamp is decomposed into calendar cycles — hour of day, day of week, day of month — so the model can learn payday rhythms and weekend patterns.
The architecture's key design choice is to separate slow-moving identity from fast-moving behaviour, then fuse them:
Reads the static or slow context: account tenure, plan, service region, balance quantile, plus "life-long" milestones like the date of the first top-up. Produces a single [USR] vector.
Encodes each event independently into an [EVT] vector, then adds that event's calendar features.
Takes the [USR] vector and the sequence of [EVT] vectors, adds time-to-last-event via rotary position encoding, and contextualises the whole thing into a final record-level embedding.
That final [USR] output of the History Encoder is the vector you cluster, probe, or fine-tune. The two-branch design pays off unevenly and informatively: an ablation showed the profile branch alone adds +31.8% PR-AUC on credit scoring (static context like tenure is very telling for default risk) but slightly hurts re-engagement prediction, which is driven almost entirely by recent event patterns. The model degrades gracefully when a branch doesn't matter.
There are no labels in pre-training. PRAGMA hides parts of each history and learns to fill them back in, using three flavours of masking at once: individual tokens (15%), whole events (10%), and all values of a chosen key (10%). Reconstructing a whole missing event, or guessing an amount given only its context, is what forces the model to learn how banking behaviour actually hangs together.
PRAGMA comes in three sizes — 10M, 100M and 1B parameters. The 10M model converged in about 2 days on 16 H100s; the larger two took roughly two weeks. Sequence packing and dynamic batching gave a 2–5× throughput win by never wasting compute on padding. Keep those numbers in mind for Day 4: the small model is not a supercomputer project.
A pre-trained backbone is inert until you attach something to it. PRAGMA supports two adaptation modes, and understanding the difference between them is the whole game — because the cheaper one is exactly what customer segmentation needs.
Freeze the backbone entirely. Push each customer's history through it once, take the final embedding, and train a tiny model on top — a logistic regression for a labelled task, or, as we'll do, a clustering algorithm for segmentation. Fitting takes minutes. This is the fast lane: use it to check whether a task is even learnable before you invest in anything heavier.
When you need the last few points of accuracy, update a small fraction of the weights — 2–4% — with Low-Rank Adaptation, leaving the shared backbone mostly frozen. PRAGMA's authors found LoRA consistently matched or beat training a model from scratch, while using a fraction of the compute and avoiding catastrophic forgetting. That result is the practical justification for the whole foundation-model approach: pre-train once, adapt many times, cheaply.
| Embedding probe | LoRA fine-tune | |
|---|---|---|
| Backbone weights | Frozen | ~2–4% updated |
| Needs labels? | Only if the head does | Yes |
| Time to fit | Minutes | Hours to days |
| Best for | Screening, segmentation | Squeezing out top accuracy |
Segmentation is unsupervised. It needs the frozen embedding and nothing else — no default outcomes, no fraud labels, no waiting for loans to mature. That makes it the cheapest possible first use case for a behavioural backbone, and the fastest way to prove the approach internally before committing to labelled tasks.
PRAGMA processes each customer's history in isolation. That is fine for most tasks but breaks on inherently relational problems. On anti-money-laundering — where the signal lives in networks of connected accounts, not any single history — PRAGMA scored 47% worse than the network-aware production baseline. The lesson: a per-customer embedding is powerful for per-customer questions, and the wrong tool when the answer lives between customers. Segmentation is squarely a per-customer question, so we're on safe ground.
The exciting part of the paper is the architecture. The sobering part is what it takes to reproduce, and that gap is exactly where we need clear eyes. You cannot download PRAGMA — it is proprietary. What the paper gives you is a recipe, and the hardest ingredient is not the model.
Revolut pre-trained on 26M customers across 111 countries. The transformer itself is standard, reproducible, and by now well understood. The thing most institutions genuinely cannot copy is a proprietary, multi-source behavioural corpus at that scale. If someone tells you the barrier is GPUs, they have it backwards — the barrier is data.
Our position is different from Revolut's in a way that favours this approach for our specific segment. We serve thin-file and Do-Not-Bank customers — precisely the people for whom hand-crafted, bureau-driven features are weakest. When there's little formal credit history, a representation learned from raw behaviour (repayments, savings deposits, USSD and app events, transaction cadence) is disproportionately valuable, because behaviour is the signal that exists when a bureau file doesn't.
As multi-market credit infrastructure sitting between capital providers and distribution platforms, we see exactly the shape of data PRAGMA is designed for: multi-source event streams per customer across markets. That is the raw material. What we lack is Revolut's volume and their dedicated research org.
You do not choose between "build a 1B-parameter foundation model" and "do nothing". There's a ladder, and the smart move is to climb only as far as the data justifies:
Train a 5–20M parameter PRAGMA-style encoder on your own events with masked modelling. Extract embeddings. Cluster them. This is a weeks-not-quarters project and needs no labels. It is Day 5 of this course.
Once repayment outcomes have matured, attach a linear probe or LoRA head for credit scoring on the same frozen backbone. The credit lift is where the paper's returns are largest.
Only once pooled multi-market data is large and clean does a bigger backbone pay off. Model size follows data size, never the reverse.
The paper's own scaling results show the small model is already competitive for behavioural tasks like lifetime value and recurring transactions — the extra parameters mostly bought gains on credit scoring. For a first segmentation model, small is not a compromise; it is the correct starting point. Reach for scale only when a specific task's accuracy demonstrably plateaus.
Time to make it concrete. We'll build a scaled-down encoder, pre-train it with masked modelling on event data, extract one embedding per customer, and cluster those embeddings into behavioural segments. The code below runs; the demo lets you feel what clustering embeddings is doing before you read it.
Traditional segmentation buckets customers by demographics or a handful of RFM-style features — income band, age, recency/frequency/monetary. For thin-file customers those inputs are thin too. Clustering a behavioural embedding instead groups people by how they actually transact, save and engage, which is both richer and available from day one of a customer's life. The segments come out behavioural and actionable rather than demographic and static.
events → key-value-time tokens → small encoder → masked-modelling pre-training → freeze → extract [USR] embedding per customer → cluster → profile & name segments → act.
Below is a 2D projection of synthetic customer embeddings — think of it as what you'd see after running the encoder and reducing its output to two axes for the eye. Move the slider to choose how many segments, run k-means, and interpret each segment. This is exactly the last step of the pipeline, made visible.
The per-segment stats are synthetic and illustrative. "Interpret" calls your library's Claude proxy (/api/claude) to turn cluster statistics into a persona and actions; if the proxy isn't running it falls back to a built-in heuristic so the demo still works offline.
Everything starts with putting your data in PRAGMA's shape. Each customer becomes a profile record plus a time-ordered list of events, and each field is a (key, value, time) triple. In production you'd learn percentile buckets from data; here we keep it readable.
# One customer = profile fields + an ordered list of events.
# Every field is a (key, value) pair; events also carry a timestamp.
from dataclasses import dataclass, field
import numpy as np
# --- vocabulary: keys are cheap, values are bucketised -----------------
KEYS = ["type","direction","amount","currency","mcc",
"plan","region","tenure"] # ~dozens in practice
N_AMOUNT_BUCKETS = 16 # percentile bins for money
def amount_bucket(x, edges):
# edges learned from the training set via np.quantile; bucket 0 = zero
if x == 0: return 0
return 1 + int(np.searchsorted(edges, x))
@dataclass
class Event:
kv: dict # {"type":"card_payment","amount":24,...}
t_log: float # 8*ln(1+dt/8): seconds since previous event
@dataclass
class Customer:
profile: dict # {"plan":"basic","region":"TZ","tenure":7}
events: list = field(default_factory=list)
def log_time(dt_seconds):
return 8.0 * np.log1p(dt_seconds / 8.0) # PRAGMA's soft-log time
We collapse PRAGMA's three encoders into a compact two-stage model: encode each event, then contextualise the sequence with the profile prepended as a [USR] token. The [USR] output of the sequence encoder is our customer embedding. This is a real, trainable model — just small.
import torch, torch.nn as nn
class PragmaMini(nn.Module):
def __init__(self, vocab, d=128, heads=4, ev_layers=2, hist_layers=2):
super().__init__()
self.tok = nn.Embedding(vocab, d, padding_idx=0)
self.usr = nn.Parameter(torch.randn(1,1,d) * .02) # [USR] token
self.time_proj = nn.Linear(1, d) # log-time -> vector
def block(n):
layer = nn.TransformerEncoderLayer(d, heads, d*4,
dropout=.1, batch_first=True, activation="gelu")
return nn.TransformerEncoder(layer, n)
self.event_enc = block(ev_layers) # tokens within one event
self.hist_enc = block(hist_layers) # across the event sequence
self.mlm_head = nn.Linear(d, vocab) # reconstruct masked tokens
def encode_events(self, ev_tokens):
# ev_tokens: (B, n_events, n_fields) -> one vector per event
B,E,F = ev_tokens.shape
x = self.tok(ev_tokens.view(B*E, F)) # (B*E, F, d)
h = self.event_enc(x)
return h.mean(1).view(B, E, -1) # (B, E, d)
def forward(self, ev_tokens, t_log, profile_vec):
ev = self.encode_events(ev_tokens) # (B, E, d)
ev = ev + self.time_proj(t_log.unsqueeze(-1)) # inject timing
usr = self.usr.expand(ev.size(0), -1, -1) + profile_vec[:,None,:]
seq = torch.cat([usr, ev], dim=1) # [USR] + events
z = self.hist_enc(seq) # (B, 1+E, d)
return z[:, 0], z # z[:,0] = customer embedding
No labels. We randomly mask event tokens and train the model to reconstruct them. This is where the encoder learns what normal behaviour looks like — the entire source of the embedding's value.
import torch, torch.nn.functional as F
MASK_ID = 1 # reserved token id for [MASK]
def mask_tokens(tokens, p=.15):
labels = tokens.clone()
keep = (torch.rand_like(tokens.float()) < p) & (tokens > 1)
labels[~keep] = -100 # ignore unmasked positions in the loss
masked = tokens.clone()
masked[keep] = MASK_ID
return masked, labels
def pretrain(model, loader, epochs=10, lr=3e-4):
opt = torch.optim.AdamW(model.parameters(), lr=lr)
for _ in range(epochs):
for ev, t_log, prof in loader:
masked, labels = mask_tokens(ev)
_, z = model(masked, t_log, prof) # z: (B,1+E,d)
logits = model.mlm_head(z[:, 1:]) # predict event tokens
loss = F.cross_entropy(
logits.reshape(-1, logits.size(-1)),
labels.reshape(-1), ignore_index=-100,
label_smoothing=.1) # as in the paper
opt.zero_grad(); loss.backward(); opt.step()
return model
Now the embedding probe idea from Day 3, in its unsupervised form. Freeze the backbone, run every customer through it once, and cluster the resulting [USR] vectors. K-means gives clean, roughly-equal-sized segments; HDBSCAN is the alternative when you want it to find irregular groups and flag outliers on its own.
import torch, numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
@torch.no_grad()
def embed_all(model, loader):
model.eval()
out = []
for ev, t_log, prof in loader:
z_usr, _ = model(ev, t_log, prof)
out.append(z_usr.cpu().numpy())
return np.concatenate(out) # (N, d) customer embeddings
def segment(embeddings, k_range=range(3,9)):
X = StandardScaler().fit_transform(embeddings) # pre-norm model -> scale
best = None
for k in k_range: # pick k by silhouette
km = KMeans(k, n_init="auto", random_state=0).fit(X)
s = silhouette_score(X, km.labels_)
if best is None or s > best[1]: best = (km, s, k)
km, score, k = best
print(f"chose k={k} (silhouette {score:.3f})")
return km.labels_, km.cluster_centers_
A cluster id is not a segment until you can describe it. For each cluster, summarise the real behaviour of its members — transaction cadence, savings rate, channel mix, repayment reliability, tenure — then give it a name a product manager can use. That profiling table is what turns "cluster 3" into "steady savers, light borrowers" and then into a decision.
| Segment (example) | Behavioural signature | Action it unlocks |
|---|---|---|
| Steady savers, thin credit | Regular deposits, few loans, low volatility | Pre-qualify a first small credit line |
| High-frequency, low-balance | Many small txns, little buffer | Liquidity nudges, short-tenor products |
| Dormant re-activators | Long gap then a burst of activity | Targeted re-engagement, watch for fraud |
| Reliable repayers | On-time repayment cadence, growing volume | Credit-line increases, cross-sell |
The same frozen backbone that produced these segments is the one you later attach a credit-scoring head to. Build it once for segmentation, reuse it for everything per-customer. Guardrail: before any segment informs a credit decision, check what each market's rules permit — a learned embedding can silently encode protected attributes, and inclusion is only inclusion if the model can be governed.
Day 5's code teaches the ideas at toy scale — one file, padded batches, a few hundred synthetic customers. This appendix is the same architecture built for the real thing: multiple markets under one backbone, RAM-safe streaming over sharded data, masked-modelling pre-training with checkpointing, and the frozen-embedding handoff that feeds your segments. It runs end to end on synthetic data out of the box, so you can see it work before pointing it at your own events.
Same model, hardened for scale. The differences are all about surviving real data volume and connecting to how you'll actually serve it:
| Concern | Day 5 (teaching) | Appendix (production shape) |
|---|---|---|
| Data volume | Fits in memory | Streamed from sharded Parquet — never loads the base into RAM |
| Markets | One | All markets, one shared backbone, market as a profile field |
| Batching | Simple padding | Shards grouped by event-count → near-uniform batches, little padding waste |
| Masking | Token-level only | All three: token, whole-event, and key-level |
| Training | Bare loop | Mixed precision, warmup + cosine decay, grad clip, checkpoints, DDP-ready |
| Output | Embeddings in memory | Extraction job → one vector per customer, on disk, ready to push on-prem |
| File | Role |
|---|---|
| config.py | All hyperparameters + S/M size presets |
| prepare_data.py | Raw long tables → vocabulary (learned numeric buckets) → sharded records |
| tokeniser.py | Key–value–time tokenisation; single shared embedding vocab |
| dataset.py | Streaming shard reader + collate with the three masking strategies |
| model.py | Profile + Event + History encoders, MLM head — outputs the customer embedding |
| pretrain.py | Masked-modelling training loop |
| embed.py | Freeze backbone → one vector per customer (the handoff) |
| make_synth.py | Synthetic 2-market data so you can run it all before touching real data |
| README.md | Architecture decisions, compute sizing, run instructions, governance |
The split you'd use to keep serving cheap — expensive model trained rarely, light scoring run often — is baked into which file does what:
prepare_data.py → pretrain.py. Runs occasionally to train the shared model. This is the only part that needs serious compute.
embed.py freezes the backbone and emits one vector per customer — computed in batch, pushed on-prem.
Cluster the vectors for segments, or attach small supervised heads. Milliseconds per customer, no GPU. This is where your segments and scores actually get produced.
# prove the pipeline runs on synthetic data before touching real events
pip install torch pandas pyarrow numpy scikit-learn
python make_synth.py # writes raw/events + raw/profile
python prepare_data.py --size S # vocab + sharded records
python pretrain.py --size S --batch 32 # trains the backbone
python embed.py --ckpt checkpoints/backbone_S_final.pt # -> vectors
# then cluster embeddings/*.parquet — that's your segments
Full pre-training on tens of millions of customers is a multi-GPU, multi-day job — the reference 10M-parameter model took ~2 days on 16 H100s. Start at size S on a 1–2M-customer sample to validate, then scale to M on rented GPUs. But you pay that rarely; extraction and heads stay on CPU.
Log which backbone version and embedding date produced each score, decide per market what a learned embedding may inform for credit, and keep this off relational tasks (AML rings) — it reads each customer in isolation. The README expands on all three.
Pairs with Day 5 — the download is the same pipeline you just read, ready to run.