imara learning library ← all courses
Foundation Models · Credit & Risk · 1 week

PRAGMA and the case for a behavioural foundation model

Revolut trained a single model on raw banking events and beat their task-specific pipelines on credit, fraud, and more. This week distils what PRAGMA actually is, whether it makes sense for a platform like ours, and how to build a scaled-down version whose first job is customer segmentation.

26M
user records in pre-training, across 111 countries
24B
events · 207 B tokens of behaviour
+130%
PR-AUC lift on credit scoring vs the task-specific model
1 → 6
one backbone, six downstream tasks

All lifts are relative to Revolut's own baselines; absolute numbers were withheld as commercially sensitive, and the paper's worked examples use synthetic data.

Day 01~1 hr

What PRAGMA actually is

PRAGMA is a family of encoder-style foundation models for banking event sequences. It reads a customer's ordered history of transactions, app taps, trades and messages, plus their profile state, and turns all of it into one dense vector that describes who that customer is, behaviourally.

The word foundation model is doing the heavy lifting. The idea, borrowed from language and vision, is to pre-train one large model on a broad corpus once, then adapt that same model cheaply to many narrow tasks. PRAGMA is the first credible version of that for consumer banking histories, and at up to a billion parameters it is the largest published encoder backbone of its kind.

The problem it replaces

Before PRAGMA, Revolut looked like most of the industry: a separate machine-learning pipeline for every question. One model for fraud, another for credit, another for churn, each with its own hand-crafted features, its own bespoke queries, its own maintenance burden. Every new question meant starting the feature-engineering grind again, and none of the models shared what they had learned about a customer.

The core bet

If sequences of banking events carry transferable structure the way sentences do, then one pre-trained backbone should be able to feed all those tasks — replacing a fleet of hand-tuned models with a shared representation plus a small head per task.

It is not an LLM

This is the most common misread, so it is worth stating plainly. PRAGMA is encoder-only and bidirectional, in the BERT lineage, not the GPT one. It does not generate text or chat. It is trained by masking parts of a customer's history and learning to reconstruct them, which forces it to build a rich internal picture of normal behaviour. The output you use is an embedding — a fixed-length vector — not a sentence.

The team deliberately chose an encoder because the goal is discriminative: score this customer, classify this event, group these users. Generation would be the wrong tool.

What the one backbone can do

A single pre-trained PRAGMA was evaluated across six tasks. The headline is not just that it works, but the size of the lifts on the hardest, rarest-signal problems:

Downstream taskHeadline metricRelative lift
Credit scoringPR-AUC+130.2%
Communication engagementPR-AUC+79.4%
External fraudRecall+64.7%
Product recommendationmAP+40.5%
Recurrent transactionsF1+5.8%
Lifetime valueROC-AUC+2.6%

Notice the pattern: the biggest wins are where the positive class is rare and hard to feature-engineer for (defaults, re-engagement, fraud). Where a task is already easy to model with a few good features (lifetime value, recurring subscriptions), the lift is small. That shape matters a lot for deciding where a model like this earns its keep — a point we return to on Day 4.

Read this if you read nothing else today

PRAGMA's real product is a reusable customer embedding learned from raw events with no labels. Every downstream task, including the segmentation we build this week, is just something you attach on top of that embedding.

Day 02~1.5 hr

How it works: tokenisation and architecture

To understand how to build our own, we need to see the two ideas that make PRAGMA work: a tokenisation scheme designed for structured events rather than text, and a three-encoder architecture that keeps who a customer is separate from what they did.

Why you can't just feed events to an LLM

The obvious shortcut — serialise each transaction into a string and hand it to a normal transformer — fails for two concrete reasons. First, every field name and delimiter explodes into several subword tokens, so a long history becomes impossibly long. Second, and worse, numeric amounts get shredded into digit fragments that throw away magnitude and ordering. A model that can't tell that 14.99 and 1,499.00 differ in scale is useless for finance.

Key–value–time tokenisation

PRAGMA instead breaks every field into three parts: a semantic type (the key), a value, and a temporal coordinate. A card payment becomes a small structured object, not a sentence:

Type
card_payment
Direction
out
Amount
14.99 → bucket 24
Currency
gbp
MCC
6012
@time
log-secs 18.95

Each value type is handled the way it deserves:

Keys need only ~60 tokens of vocabulary; values need ~28k. Compact, and every token means something.

Time, encoded twice

Timing is a first-class signal, so PRAGMA encodes it two ways. A smooth logarithmic transform of the gap since the previous event, 8·ln(1 + t/8), compresses years-long gaps while keeping seconds-level precision for recent activity. Separately, each timestamp is decomposed into calendar cycles — hour of day, day of week, day of month — so the model can learn payday rhythms and weekend patterns.

Three encoders, two branches

The architecture's key design choice is to separate slow-moving identity from fast-moving behaviour, then fuse them:

Profile State Encoder

Reads the static or slow context: account tenure, plan, service region, balance quantile, plus "life-long" milestones like the date of the first top-up. Produces a single [USR] vector.

Event Encoder

Encodes each event independently into an [EVT] vector, then adds that event's calendar features.

History Encoder

Takes the [USR] vector and the sequence of [EVT] vectors, adds time-to-last-event via rotary position encoding, and contextualises the whole thing into a final record-level embedding.

That final [USR] output of the History Encoder is the vector you cluster, probe, or fine-tune. The two-branch design pays off unevenly and informatively: an ablation showed the profile branch alone adds +31.8% PR-AUC on credit scoring (static context like tenure is very telling for default risk) but slightly hurts re-engagement prediction, which is driven almost entirely by recent event patterns. The model degrades gracefully when a branch doesn't matter.

Pre-training: masked modelling

There are no labels in pre-training. PRAGMA hides parts of each history and learns to fill them back in, using three flavours of masking at once: individual tokens (15%), whole events (10%), and all values of a chosen key (10%). Reconstructing a whole missing event, or guessing an amount given only its context, is what forces the model to learn how banking behaviour actually hangs together.

The scale that made it work

PRAGMA comes in three sizes — 10M, 100M and 1B parameters. The 10M model converged in about 2 days on 16 H100s; the larger two took roughly two weeks. Sequence packing and dynamic batching gave a 2–5× throughput win by never wasting compute on padding. Keep those numbers in mind for Day 4: the small model is not a supercomputer project.

Day 03~1 hr

Adaptation: how you get value out

A pre-trained backbone is inert until you attach something to it. PRAGMA supports two adaptation modes, and understanding the difference between them is the whole game — because the cheaper one is exactly what customer segmentation needs.

Mode 1 — Embedding probe (frozen)

Freeze the backbone entirely. Push each customer's history through it once, take the final embedding, and train a tiny model on top — a logistic regression for a labelled task, or, as we'll do, a clustering algorithm for segmentation. Fitting takes minutes. This is the fast lane: use it to check whether a task is even learnable before you invest in anything heavier.

Mode 2 — LoRA fine-tuning

When you need the last few points of accuracy, update a small fraction of the weights — 2–4% — with Low-Rank Adaptation, leaving the shared backbone mostly frozen. PRAGMA's authors found LoRA consistently matched or beat training a model from scratch, while using a fraction of the compute and avoiding catastrophic forgetting. That result is the practical justification for the whole foundation-model approach: pre-train once, adapt many times, cheaply.

 Embedding probeLoRA fine-tune
Backbone weightsFrozen~2–4% updated
Needs labels?Only if the head doesYes
Time to fitMinutesHours to days
Best forScreening, segmentationSqueezing out top accuracy
Why this matters for us

Segmentation is unsupervised. It needs the frozen embedding and nothing else — no default outcomes, no fraud labels, no waiting for loans to mature. That makes it the cheapest possible first use case for a behavioural backbone, and the fastest way to prove the approach internally before committing to labelled tasks.

One honest limitation to carry forward

PRAGMA processes each customer's history in isolation. That is fine for most tasks but breaks on inherently relational problems. On anti-money-laundering — where the signal lives in networks of connected accounts, not any single history — PRAGMA scored 47% worse than the network-aware production baseline. The lesson: a per-customer embedding is powerful for per-customer questions, and the wrong tool when the answer lives between customers. Segmentation is squarely a per-customer question, so we're on safe ground.

Day 04~1.5 hr

Should _able build one? The honest version

The exciting part of the paper is the architecture. The sobering part is what it takes to reproduce, and that gap is exactly where we need clear eyes. You cannot download PRAGMA — it is proprietary. What the paper gives you is a recipe, and the hardest ingredient is not the model.

The moat is the corpus, not the code

Revolut pre-trained on 26M customers across 111 countries. The transformer itself is standard, reproducible, and by now well understood. The thing most institutions genuinely cannot copy is a proprietary, multi-source behavioural corpus at that scale. If someone tells you the barrier is GPUs, they have it backwards — the barrier is data.

Where this actually helps us

Our position is different from Revolut's in a way that favours this approach for our specific segment. We serve thin-file and Do-Not-Bank customers — precisely the people for whom hand-crafted, bureau-driven features are weakest. When there's little formal credit history, a representation learned from raw behaviour (repayments, savings deposits, USSD and app events, transaction cadence) is disproportionately valuable, because behaviour is the signal that exists when a bureau file doesn't.

What we have to work with

As multi-market credit infrastructure sitting between capital providers and distribution platforms, we see exactly the shape of data PRAGMA is designed for: multi-source event streams per customer across markets. That is the raw material. What we lack is Revolut's volume and their dedicated research org.

The realistic ladder

You do not choose between "build a 1B-parameter foundation model" and "do nothing". There's a ladder, and the smart move is to climb only as far as the data justifies:

Rung 1 — Embedding probe on a small backbone

Train a 5–20M parameter PRAGMA-style encoder on your own events with masked modelling. Extract embeddings. Cluster them. This is a weeks-not-quarters project and needs no labels. It is Day 5 of this course.

Rung 2 — Add supervised heads where labels exist

Once repayment outcomes have matured, attach a linear probe or LoRA head for credit scoring on the same frozen backbone. The credit lift is where the paper's returns are largest.

Rung 3 — Scale the corpus, then the model

Only once pooled multi-market data is large and clean does a bigger backbone pay off. Model size follows data size, never the reverse.

Rule of thumb on scale

The paper's own scaling results show the small model is already competitive for behavioural tasks like lifetime value and recurring transactions — the extra parameters mostly bought gains on credit scoring. For a first segmentation model, small is not a compromise; it is the correct starting point. Reach for scale only when a specific task's accuracy demonstrably plateaus.

The four questions to answer before you commit

  1. Volume. How many customers have enough events to be worth embedding? (Users with zero events are simply dropped; very long histories are truncated to recent activity.)
  2. Sources. Can we actually join transactions, repayments, savings and app/USSD events per customer across markets, with consistent identifiers?
  3. Governance. What can a learned embedding legally inform, per market, especially for credit decisions on protected populations?
  4. First target. Is there a per-customer (not relational) question where a shared representation beats what we have today? Segmentation is the safest yes.
Day 05~2 hr

Build it: customer segmentation on a PRAGMA-style backbone

Time to make it concrete. We'll build a scaled-down encoder, pre-train it with masked modelling on event data, extract one embedding per customer, and cluster those embeddings into behavioural segments. The code below runs; the demo lets you feel what clustering embeddings is doing before you read it.

Why segmentation, and why embeddings

Traditional segmentation buckets customers by demographics or a handful of RFM-style features — income band, age, recency/frequency/monetary. For thin-file customers those inputs are thin too. Clustering a behavioural embedding instead groups people by how they actually transact, save and engage, which is both richer and available from day one of a customer's life. The segments come out behavioural and actionable rather than demographic and static.

The whole pipeline in one line

events → key-value-time tokens → small encoder → masked-modelling pre-training → freeze → extract [USR] embedding per customer → cluster → profile & name segments → act.

See it first: cluster the embeddings

Below is a 2D projection of synthetic customer embeddings — think of it as what you'd see after running the encoder and reducing its output to two axes for the eye. Move the slider to choose how many segments, run k-means, and interpret each segment. This is exactly the last step of the pipeline, made visible.

Segmentation playground · k-means on customer embeddings
4
Each point is one customer. Axes are a 2D projection of the embedding — real ones have hundreds of dimensions.

The per-segment stats are synthetic and illustrative. "Interpret" calls your library's Claude proxy (/api/claude) to turn cluster statistics into a persona and actions; if the proxy isn't running it falls back to a built-in heuristic so the demo still works offline.

Step 1 — Represent events as key–value–time

Everything starts with putting your data in PRAGMA's shape. Each customer becomes a profile record plus a time-ordered list of events, and each field is a (key, value, time) triple. In production you'd learn percentile buckets from data; here we keep it readable.

1_tokenise.py
# One customer = profile fields + an ordered list of events.
# Every field is a (key, value) pair; events also carry a timestamp.
from dataclasses import dataclass, field
import numpy as np

# --- vocabulary: keys are cheap, values are bucketised -----------------
KEYS  = ["type","direction","amount","currency","mcc",
         "plan","region","tenure"]           # ~dozens in practice
N_AMOUNT_BUCKETS = 16                             # percentile bins for money

def amount_bucket(x, edges):
    # edges learned from the training set via np.quantile; bucket 0 = zero
    if x == 0: return 0
    return 1 + int(np.searchsorted(edges, x))

@dataclass
class Event:
    kv:   dict                 # {"type":"card_payment","amount":24,...}
    t_log: float               # 8*ln(1+dt/8): seconds since previous event

@dataclass
class Customer:
    profile: dict              # {"plan":"basic","region":"TZ","tenure":7}
    events:  list = field(default_factory=list)

def log_time(dt_seconds):
    return 8.0 * np.log1p(dt_seconds / 8.0)     # PRAGMA's soft-log time

Step 2 — A minimal PRAGMA-style encoder

We collapse PRAGMA's three encoders into a compact two-stage model: encode each event, then contextualise the sequence with the profile prepended as a [USR] token. The [USR] output of the sequence encoder is our customer embedding. This is a real, trainable model — just small.

2_encoder.py
import torch, torch.nn as nn

class PragmaMini(nn.Module):
    def __init__(self, vocab, d=128, heads=4, ev_layers=2, hist_layers=2):
        super().__init__()
        self.tok = nn.Embedding(vocab, d, padding_idx=0)
        self.usr = nn.Parameter(torch.randn(1,1,d) * .02)   # [USR] token
        self.time_proj = nn.Linear(1, d)                    # log-time -> vector

        def block(n):
            layer = nn.TransformerEncoderLayer(d, heads, d*4,
                        dropout=.1, batch_first=True, activation="gelu")
            return nn.TransformerEncoder(layer, n)

        self.event_enc = block(ev_layers)     # tokens within one event
        self.hist_enc  = block(hist_layers)   # across the event sequence
        self.mlm_head  = nn.Linear(d, vocab)  # reconstruct masked tokens

    def encode_events(self, ev_tokens):
        # ev_tokens: (B, n_events, n_fields) -> one vector per event
        B,E,F = ev_tokens.shape
        x = self.tok(ev_tokens.view(B*E, F))          # (B*E, F, d)
        h = self.event_enc(x)
        return h.mean(1).view(B, E, -1)               # (B, E, d)

    def forward(self, ev_tokens, t_log, profile_vec):
        ev = self.encode_events(ev_tokens)            # (B, E, d)
        ev = ev + self.time_proj(t_log.unsqueeze(-1))  # inject timing
        usr = self.usr.expand(ev.size(0), -1, -1) + profile_vec[:,None,:]
        seq = torch.cat([usr, ev], dim=1)            # [USR] + events
        z = self.hist_enc(seq)                        # (B, 1+E, d)
        return z[:, 0], z                          # z[:,0] = customer embedding

Step 3 — Pre-train with masked modelling

No labels. We randomly mask event tokens and train the model to reconstruct them. This is where the encoder learns what normal behaviour looks like — the entire source of the embedding's value.

3_pretrain.py
import torch, torch.nn.functional as F

MASK_ID = 1                      # reserved token id for [MASK]

def mask_tokens(tokens, p=.15):
    labels = tokens.clone()
    keep   = (torch.rand_like(tokens.float()) < p) & (tokens > 1)
    labels[~keep] = -100            # ignore unmasked positions in the loss
    masked = tokens.clone()
    masked[keep] = MASK_ID
    return masked, labels

def pretrain(model, loader, epochs=10, lr=3e-4):
    opt = torch.optim.AdamW(model.parameters(), lr=lr)
    for _ in range(epochs):
        for ev, t_log, prof in loader:
            masked, labels = mask_tokens(ev)
            _, z = model(masked, t_log, prof)         # z: (B,1+E,d)
            logits = model.mlm_head(z[:, 1:])         # predict event tokens
            loss = F.cross_entropy(
                logits.reshape(-1, logits.size(-1)),
                labels.reshape(-1), ignore_index=-100,
                label_smoothing=.1)                    # as in the paper
            opt.zero_grad(); loss.backward(); opt.step()
    return model

Step 4 — Freeze, extract embeddings, cluster

Now the embedding probe idea from Day 3, in its unsupervised form. Freeze the backbone, run every customer through it once, and cluster the resulting [USR] vectors. K-means gives clean, roughly-equal-sized segments; HDBSCAN is the alternative when you want it to find irregular groups and flag outliers on its own.

4_segment.py
import torch, numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score

@torch.no_grad()
def embed_all(model, loader):
    model.eval()
    out = []
    for ev, t_log, prof in loader:
        z_usr, _ = model(ev, t_log, prof)
        out.append(z_usr.cpu().numpy())
    return np.concatenate(out)                    # (N, d) customer embeddings

def segment(embeddings, k_range=range(3,9)):
    X = StandardScaler().fit_transform(embeddings)   # pre-norm model -> scale
    best = None
    for k in k_range:                              # pick k by silhouette
        km = KMeans(k, n_init="auto", random_state=0).fit(X)
        s  = silhouette_score(X, km.labels_)
        if best is None or s > best[1]: best = (km, s, k)
    km, score, k = best
    print(f"chose k={k} (silhouette {score:.3f})")
    return km.labels_, km.cluster_centers_

Step 5 — Profile, name, and act

A cluster id is not a segment until you can describe it. For each cluster, summarise the real behaviour of its members — transaction cadence, savings rate, channel mix, repayment reliability, tenure — then give it a name a product manager can use. That profiling table is what turns "cluster 3" into "steady savers, light borrowers" and then into a decision.

Segment (example)Behavioural signatureAction it unlocks
Steady savers, thin creditRegular deposits, few loans, low volatilityPre-qualify a first small credit line
High-frequency, low-balanceMany small txns, little bufferLiquidity nudges, short-tenor products
Dormant re-activatorsLong gap then a burst of activityTargeted re-engagement, watch for fraud
Reliable repayersOn-time repayment cadence, growing volumeCredit-line increases, cross-sell
The payoff, and the guardrail

The same frozen backbone that produced these segments is the one you later attach a credit-scoring head to. Build it once for segmentation, reuse it for everything per-customer. Guardrail: before any segment informs a credit decision, check what each market's rules permit — a learned embedding can silently encode protected attributes, and inclusion is only inclusion if the model can be governed.

Where to go next

Appendixdownload + guide

From course to production

Day 5's code teaches the ideas at toy scale — one file, padded batches, a few hundred synthetic customers. This appendix is the same architecture built for the real thing: multiple markets under one backbone, RAM-safe streaming over sharded data, masked-modelling pre-training with checkpointing, and the frozen-embedding handoff that feeds your segments. It runs end to end on synthetic data out of the box, so you can see it work before pointing it at your own events.

pragma_train.zip
9 files — a runnable PyTorch pipeline: data prep, tokeniser, streaming dataset, the three-encoder model, pre-training loop, and embedding extraction. Plus a synthetic-data generator and a full README.
↓ Download the project

What changed from the Day 5 teaching code

Same model, hardened for scale. The differences are all about surviving real data volume and connecting to how you'll actually serve it:

ConcernDay 5 (teaching)Appendix (production shape)
Data volumeFits in memoryStreamed from sharded Parquet — never loads the base into RAM
MarketsOneAll markets, one shared backbone, market as a profile field
BatchingSimple paddingShards grouped by event-count → near-uniform batches, little padding waste
MaskingToken-level onlyAll three: token, whole-event, and key-level
TrainingBare loopMixed precision, warmup + cosine decay, grad clip, checkpoints, DDP-ready
OutputEmbeddings in memoryExtraction job → one vector per customer, on disk, ready to push on-prem

What's in the box

FileRole
config.pyAll hyperparameters + S/M size presets
prepare_data.pyRaw long tables → vocabulary (learned numeric buckets) → sharded records
tokeniser.pyKey–value–time tokenisation; single shared embedding vocab
dataset.pyStreaming shard reader + collate with the three masking strategies
model.pyProfile + Event + History encoders, MLM head — outputs the customer embedding
pretrain.pyMasked-modelling training loop
embed.pyFreeze backbone → one vector per customer (the handoff)
make_synth.pySynthetic 2-market data so you can run it all before touching real data
README.mdArchitecture decisions, compute sizing, run instructions, governance

How the files map to the two-tier design

The split you'd use to keep serving cheap — expensive model trained rarely, light scoring run often — is baked into which file does what:

Tier 1 — the expensive backbone (GPU, cloud, rare)

prepare_data.py → pretrain.py. Runs occasionally to train the shared model. This is the only part that needs serious compute.

The handoff

embed.py freezes the backbone and emits one vector per customer — computed in batch, pushed on-prem.

Tier 2 — cheap heads (CPU, on-prem, often)

Cluster the vectors for segments, or attach small supervised heads. Milliseconds per customer, no GPU. This is where your segments and scores actually get produced.

Quick start

terminal
# prove the pipeline runs on synthetic data before touching real events
pip install torch pandas pyarrow numpy scikit-learn
python make_synth.py                     # writes raw/events + raw/profile
python prepare_data.py --size S          # vocab + sharded records
python pretrain.py --size S --batch 32    # trains the backbone
python embed.py --ckpt checkpoints/backbone_S_final.pt   # -> vectors
# then cluster embeddings/*.parquet — that's your segments
Be honest about scale

Full pre-training on tens of millions of customers is a multi-GPU, multi-day job — the reference 10M-parameter model took ~2 days on 16 H100s. Start at size S on a 1–2M-customer sample to validate, then scale to M on rented GPUs. But you pay that rarely; extraction and heads stay on CPU.

Before it touches a lending decision

Log which backbone version and embedding date produced each score, decide per market what a learned embedding may inform for credit, and keep this off relational tasks (AML rings) — it reads each customer in isolation. The README expands on all three.

Pairs with Day 5 — the download is the same pipeline you just read, ready to run.