Visa trained one transformer on billions of card transactions — and it beat their production fraud systems by 111% while also serving as a general embedding provider that lifted recommendations by 104%. This week distils how TREASURE works, why it's built so differently from PRAGMA, and what it teaches for our own transaction models.
TREASURE — TRansformer Engine As Scalable Universal transaction Representation Encoder — is Visa Research's foundation model for payment data. The thesis is the one you know from PRAGMA: instead of a fleet of task-specific models, train one transformer over a card's transaction history and reuse it everywhere.
Visa processes over 300 billion transactions a year — about $15 trillion — across 4 billion+ cards, 150 million+ merchants and 200+ countries, and runs 100+ production ML models on that data. TREASURE is the attempt to put a single foundation model underneath those models. It was trained on a sample of ~6 billion transactions from 30 million cardholders over 26 months.
Every transaction has two sides. There's what the cardholder did (bought coffee for $3.14 at a merchant in Nairobi), and there's what the payment network said back (approved, or declined with a response code). Prior transaction models only learned the cardholder side. TREASURE's signature move is to learn both — cardholder behaviour and payment-network signals — in one model. Those network signals (decline codes, system flags) are powerful, real-time evidence for spotting fraud that behaviour alone misses.
PRAGMA is encoder-only and bidirectional (BERT-style, trained by filling in masked blanks). TREASURE is decoder-only and autoregressive (GPT-style, trained to predict the next transaction). Two valid designs for the same problem — we'll see why Visa chose generation over reconstruction, and the trade-offs, throughout the week.
TREASURE was validated doing two very different things with the same weights:
That dual role — a security engine and a representation service from one backbone — is the whole pitch of a transaction foundation model, and it's exactly the Tier-1/Tier-2 pattern from the PRAGMA course.
Raw transactions arrive as an interleaved log — every card's activity mixed together. The first step is to group them by card and sort by time. Once you do, a crucial structure appears, and it drives TREASURE's whole input design.
Group one card's transactions chronologically and you notice two kinds of fields: some never change across the card's life (its product type, issuing country, account age band) and some change every transaction (amount, merchant, city, time). TREASURE calls the first static and the second dynamic, and processes them with separate input sub-modules — more efficient than treating everything as changing. Toggle the view to see it:
Within each attribute vector, two data types are treated differently:
The numerical representation and all the categorical embeddings are then concatenated and fused through a couple more layers into one vector per transaction — the input the transformer actually consumes.
Each training sample is one card's ordered transactions. For deployment, TREASURE caps the history at the 512 most recent transactions — which still covers 2+ years for most cardholders — giving a fixed, predictable compute budget for real-time scoring. The same "keep the recent window" idea as PRAGMA's truncation.
At its core TREASURE is a small decoder-only transformer — the same family as a language model, but over transactions instead of words. What makes it a payment model is what sits on top: two prediction heads that look in two different directions in time.
The per-transaction vectors (plus the single static vector) go into a Transformer decoder with causal masked self-attention — each transaction can attend only to itself and earlier ones, never the future, exactly like next-word prediction. Two neat details:
For each transaction in the sequence, the output module makes two separate predictions:
Step through a short history and watch both heads fire at each transaction:
One more refinement. For numerical outputs like the next amount, TREASURE doesn't predict a single value — it predicts a whole log-normal distribution, estimating a mean and spread (μ, σ) in log scale. That captures the genuine uncertainty in "how much will they spend next" rather than pretending to know exactly, and it fits the long-tailed nature of money. A point estimate is just a sample from it.
Here's the problem that would sink a naive build. To predict the next merchant, the model must choose among 150 million+ possibilities. Standard training can't do that — and the fix is the most important engineering idea in the paper.
A language model predicts the next word from a vocabulary under a million. To do that it computes one logit per possible answer, then a softmax over all of them. For merchants that means 150M+ logits for every prediction, every time step, every sample — the GPU memory simply isn't there. Watch the wall you hit:
TREASURE splits categorical attributes by cardinality at a threshold of 1,024: small ones (country, category) use ordinary cross-entropy; big ones (merchant, city) use the InfoNCE contrastive loss. The idea: instead of scoring the true merchant against all 150M, score it against the true one plus a handful of random negatives, and push the true one's score above them.
The real trick is one word: shared. Rather than draw fresh negatives for every time step of every sample, TREASURE draws one pool of ~1,024 negatives and reuses it across the whole batch. That single choice keeps forward-pass memory under 3 GB and the backward pass under 6 GB, where the naive per-sample approach runs out of memory past 64 negatives.
def high_cardinality_loss(hidden, embedding, label_pos, n_neg):
# hidden: (batch, seq, dim) model output for the attribute
# embedding: (n_category, dim) the attribute's embedding table
pos = embedding[label_pos] # the true answers
# ONE shared pool of negatives for the whole batch:
neg_idx = torch.randint(embedding.shape[0], (n_neg,))
neg = embedding[neg_idx]
dot_pos = torch.einsum("ijk,ljk->ilj", hidden, pos)
dot_neg = torch.einsum("ijk,lk->ilj", hidden, neg)
allv = torch.cat([dot_pos, dot_neg], dim=1)
idx = torch.arange(hidden.shape[0])
num = dot_pos[idx, idx] # score of the truth
den = torch.logsumexp(allv, dim=1) # vs truth + shared negs
return num - den # InfoNCETREASURE predicts ~18 attributes at once (16 next-transaction + 2 network signals), so it juggles many losses. It treats the abnormal-behaviour flag — the fraud signal — as the primary task and scales every other loss relative to it, so the critical security objective dominates the gradient while the auxiliary tasks still help the model generalise. In their tests this beat both naively summing the losses and weighting them equally.
The payoff shows up in three places: raw performance, the quality of the embeddings, and a set of design lessons that transfer directly to a stack like ours.
| Finding | Result |
|---|---|
| Abnormal / fraud detection (standalone) | +111% vs production |
| Recommendation (as embedding provider) | +104% vs supervised embeddings |
| Transformer vs GRU / LSTM backbone | clearly better (RI 2.12 vs ~1.4) |
| Multi-purpose vs single-purpose models | multi-purpose wins on every task |
| Shared vs independent negatives (merchant Prec@1) | 0.142 vs 0.060 |
When Visa projected the learned merchant embeddings into 2D, structure emerged with no labels: merchants clustered by geography (European countries together; the US and Canada together, with the UK sitting between them and Europe) and by business category (drug stores next to grocery stores; fast-food next to full-service restaurants). The model recovered real market structure purely from transaction patterns — and a downstream team can use those vectors without ever seeing raw transactions, a real privacy advantage. Explore it:
A finding worth carrying: TREASURE's performance kept climbing with more training data (they extrapolate gains out to trillion-transaction scale), but gains from a bigger model flattened out. The lesson for anyone building one of these — and it echoes PRAGMA — is that the corpus is the lever, not raw parameter count.
| PRAGMA (Revolut) | TREASURE (Visa) | |
|---|---|---|
| Architecture | Encoder-only, bidirectional | Decoder-only, autoregressive |
| Pre-training task | Masked modelling (fill blanks) | Predict next transaction + current signals |
| Attribute design | Profile state + events branches | Static + dynamic sub-modules |
| Numbers | Percentile buckets (tokens) | Log-scale, predicted as log-normal (μ,σ) |
| Signature idea | Life-long profile events | Payment-network signals (approve/decline) |
| High cardinality | BPE / value vocabulary | InfoNCE + shared negative sampling |
| Both share | transformer over transaction sequences · reusable embeddings · per-entity in isolation · data scaling wins | |
Four transferable lessons: (1) a decoder-only, autoregressive design is a viable alternative to PRAGMA's encoder — it naturally forecasts "what next"; (2) model the outcome signals, not just behaviour — our analogue of Visa's decline codes is repayment outcomes, decline reasons, collections events; (3) InfoNCE with shared negatives is the standard tool when a field has millions of values (merchants, agents, billers across our markets); and (4) spend on data, not parameters. Both limitations also match: TREASURE, like PRAGMA, reads each card in isolation and flags graph/cross-entity modelling (shared merchants, networks) as future work — the same relational gap we noted for AML.
With PRAGMA and TREASURE side by side, you now have both major design templates for a transaction foundation model — encoder vs decoder, masked vs autoregressive, buckets vs distributions — and the engineering (high-cardinality losses, shared negatives, outcome-signal heads) to build one at scale for the payment and credit data we actually have.