imara learning library ← all courses
Foundation Models · Credit & Risk · 1 week

TREASURE: Visa's payment foundation model

Visa trained one transformer on billions of card transactions — and it beat their production fraud systems by 111% while also serving as a general embedding provider that lifted recommendations by 104%. This week distils how TREASURE works, why it's built so differently from PRAGMA, and what it teaches for our own transaction models.

300B+
transactions Visa processes a year ($15T)
+111%
fraud / abnormal-behaviour detection vs production
+104%
recommendation lift as an embedding provider
1 → many
one backbone replacing task-specific models
Module 01the idea

What TREASURE is

TREASURE — TRansformer Engine As Scalable Universal transaction Representation Encoder — is Visa Research's foundation model for payment data. The thesis is the one you know from PRAGMA: instead of a fleet of task-specific models, train one transformer over a card's transaction history and reuse it everywhere.

The scale it was built for

Visa processes over 300 billion transactions a year — about $15 trillion — across 4 billion+ cards, 150 million+ merchants and 200+ countries, and runs 100+ production ML models on that data. TREASURE is the attempt to put a single foundation model underneath those models. It was trained on a sample of ~6 billion transactions from 30 million cardholders over 26 months.

The one idea that makes it different: it models the network, not just you

Every transaction has two sides. There's what the cardholder did (bought coffee for $3.14 at a merchant in Nairobi), and there's what the payment network said back (approved, or declined with a response code). Prior transaction models only learned the cardholder side. TREASURE's signature move is to learn both — cardholder behaviour and payment-network signals — in one model. Those network signals (decline codes, system flags) are powerful, real-time evidence for spotting fraud that behaviour alone misses.

Encoder or decoder? The big fork vs PRAGMA

PRAGMA is encoder-only and bidirectional (BERT-style, trained by filling in masked blanks). TREASURE is decoder-only and autoregressive (GPT-style, trained to predict the next transaction). Two valid designs for the same problem — we'll see why Visa chose generation over reconstruction, and the trade-offs, throughout the week.

Two jobs, one model

TREASURE was validated doing two very different things with the same weights:

That dual role — a security engine and a representation service from one backbone — is the whole pitch of a transaction foundation model, and it's exactly the Tier-1/Tier-2 pattern from the PRAGMA course.

Module 02getting data in

The data & the input module

Raw transactions arrive as an interleaved log — every card's activity mixed together. The first step is to group them by card and sort by time. Once you do, a crucial structure appears, and it drives TREASURE's whole input design.

Static vs dynamic attributes — the core distinction

Group one card's transactions chronologically and you notice two kinds of fields: some never change across the card's life (its product type, issuing country, account age band) and some change every transaction (amount, merchant, city, time). TREASURE calls the first static and the second dynamic, and processes them with separate input sub-modules — more efficient than treating everything as changing. Toggle the view to see it:

Raw log → grouped by card → static vs dynamic
In the grouped view, the gold columns are static (constant for the card) and the blue columns are dynamic (change each transaction). Visa's sample had 5 static and 16 dynamic attributes per transaction.

Numbers and categories, handled separately

Within each attribute vector, two data types are treated differently:

The numerical representation and all the categorical embeddings are then concatenated and fused through a couple more layers into one vector per transaction — the input the transformer actually consumes.

A card = a sequence, capped at 512

Each training sample is one card's ordered transactions. For deployment, TREASURE caps the history at the 512 most recent transactions — which still covers 2+ years for most cardholders — giving a fixed, predictable compute budget for real-time scoring. The same "keep the recent window" idea as PRAGMA's truncation.

Module 03how it predicts

The architecture: a decoder with two heads

At its core TREASURE is a small decoder-only transformer — the same family as a language model, but over transactions instead of words. What makes it a payment model is what sits on top: two prediction heads that look in two different directions in time.

The backbone

The per-transaction vectors (plus the single static vector) go into a Transformer decoder with causal masked self-attention — each transaction can attend only to itself and earlier ones, never the future, exactly like next-word prediction. Two neat details:

The two heads — the clever part

For each transaction in the sequence, the output module makes two separate predictions:

at step i:   predict the signals of transaction i   and   the attributes of transaction i+1one head looks at "now", the other forecasts "next"

Step through a short history and watch both heads fire at each transaction:

The two heads · step through a card's sequence
The same transformer output at each step drives both heads. Training on both tasks together is what lets one model serve fraud detection (signals) and recommendation/forecasting (next attributes).

Predicting a number as a distribution

One more refinement. For numerical outputs like the next amount, TREASURE doesn't predict a single value — it predicts a whole log-normal distribution, estimating a mean and spread (μ, σ) in log scale. That captures the genuine uncertainty in "how much will they spend next" rather than pretending to know exactly, and it fits the long-tailed nature of money. A point estimate is just a sample from it.

Module 04the engineering win

The 150-million-class problem

Here's the problem that would sink a naive build. To predict the next merchant, the model must choose among 150 million+ possibilities. Standard training can't do that — and the fix is the most important engineering idea in the paper.

Why ordinary cross-entropy breaks

A language model predicts the next word from a vocabulary under a million. To do that it computes one logit per possible answer, then a softmax over all of them. For merchants that means 150M+ logits for every prediction, every time step, every sample — the GPU memory simply isn't there. Watch the wall you hit:

Full softmax vs InfoNCE · memory as cardinality grows
Full softmax grows with the whole vocabulary (red) and blows past GPU memory. InfoNCE computes only the true answer plus ~1,024 sampled negatives (gold) — flat, no matter how many merchants exist.

The fix: InfoNCE with shared negatives

TREASURE splits categorical attributes by cardinality at a threshold of 1,024: small ones (country, category) use ordinary cross-entropy; big ones (merchant, city) use the InfoNCE contrastive loss. The idea: instead of scoring the true merchant against all 150M, score it against the true one plus a handful of random negatives, and push the true one's score above them.

Lhcat = − log [ eh·E[y] / Σi∈I eh·E[i] ]push the true merchant's score above a set I of itself + sampled negatives

The real trick is one word: shared. Rather than draw fresh negatives for every time step of every sample, TREASURE draws one pool of ~1,024 negatives and reuses it across the whole batch. That single choice keeps forward-pass memory under 3 GB and the backward pass under 6 GB, where the naive per-sample approach runs out of memory past 64 negatives.

high-cardinality loss (adapted from the paper, CC BY 4.0)
def high_cardinality_loss(hidden, embedding, label_pos, n_neg):
    # hidden:    (batch, seq, dim)   model output for the attribute
    # embedding: (n_category, dim)   the attribute's embedding table
    pos = embedding[label_pos]                 # the true answers
    # ONE shared pool of negatives for the whole batch:
    neg_idx = torch.randint(embedding.shape[0], (n_neg,))
    neg = embedding[neg_idx]
    dot_pos = torch.einsum("ijk,ljk->ilj", hidden, pos)
    dot_neg = torch.einsum("ijk,lk->ilj", hidden, neg)
    allv = torch.cat([dot_pos, dot_neg], dim=1)
    idx  = torch.arange(hidden.shape[0])
    num  = dot_pos[idx, idx]                    # score of the truth
    den  = torch.logsumexp(allv, dim=1)        # vs truth + shared negs
    return num - den                          # InfoNCE

Keeping the main job in charge

TREASURE predicts ~18 attributes at once (16 next-transaction + 2 network signals), so it juggles many losses. It treats the abnormal-behaviour flag — the fraud signal — as the primary task and scales every other loss relative to it, so the critical security objective dominates the gradient while the auxiliary tasks still help the model generalise. In their tests this beat both naively summing the losses and weighting them equally.

Module 05what it proved

Results, the embeddings & TREASURE vs PRAGMA

The payoff shows up in three places: raw performance, the quality of the embeddings, and a set of design lessons that transfer directly to a stack like ours.

The headline numbers

FindingResult
Abnormal / fraud detection (standalone)+111% vs production
Recommendation (as embedding provider)+104% vs supervised embeddings
Transformer vs GRU / LSTM backboneclearly better (RI 2.12 vs ~1.4)
Multi-purpose vs single-purpose modelsmulti-purpose wins on every task
Shared vs independent negatives (merchant Prec@1)0.142 vs 0.060

The embeddings learned the world

When Visa projected the learned merchant embeddings into 2D, structure emerged with no labels: merchants clustered by geography (European countries together; the US and Canada together, with the UK sitting between them and Europe) and by business category (drug stores next to grocery stores; fast-food next to full-service restaurants). The model recovered real market structure purely from transaction patterns — and a downstream team can use those vectors without ever seeing raw transactions, a real privacy advantage. Explore it:

Merchant embeddings · colour by geography or category
Each dot is a merchant, placed by its learned embedding. The same vectors cluster sensibly whether you colour by where the merchant is or what it sells — evidence the embedding captured both at once.

Scaling: data matters more than model size

A finding worth carrying: TREASURE's performance kept climbing with more training data (they extrapolate gains out to trillion-transaction scale), but gains from a bigger model flattened out. The lesson for anyone building one of these — and it echoes PRAGMA — is that the corpus is the lever, not raw parameter count.

TREASURE vs PRAGMA — two roads to the same destination

 PRAGMA (Revolut)TREASURE (Visa)
ArchitectureEncoder-only, bidirectionalDecoder-only, autoregressive
Pre-training taskMasked modelling (fill blanks)Predict next transaction + current signals
Attribute designProfile state + events branchesStatic + dynamic sub-modules
NumbersPercentile buckets (tokens)Log-scale, predicted as log-normal (μ,σ)
Signature ideaLife-long profile eventsPayment-network signals (approve/decline)
High cardinalityBPE / value vocabularyInfoNCE + shared negative sampling
Both sharetransformer over transaction sequences · reusable embeddings · per-entity in isolation · data scaling wins
What to take for our stack

Four transferable lessons: (1) a decoder-only, autoregressive design is a viable alternative to PRAGMA's encoder — it naturally forecasts "what next"; (2) model the outcome signals, not just behaviour — our analogue of Visa's decline codes is repayment outcomes, decline reasons, collections events; (3) InfoNCE with shared negatives is the standard tool when a field has millions of values (merchants, agents, billers across our markets); and (4) spend on data, not parameters. Both limitations also match: TREASURE, like PRAGMA, reads each card in isolation and flags graph/cross-entity modelling (shared merchants, networks) as future work — the same relational gap we noted for AML.

You've seen the second blueprint

With PRAGMA and TREASURE side by side, you now have both major design templates for a transaction foundation model — encoder vs decoder, masked vs autoregressive, buckets vs distributions — and the engineering (high-cardinality losses, shared negatives, outcome-signal heads) to build one at scale for the payment and credit data we actually have.