The TREASURE course explained the model. This companion is the build: how to stand up a TREASURE-style transaction foundation model on our own stack — lakehouse to training to a real-time scorer in the decisioning engine — with the data residency, latency, monitoring and rollout realities of running it for live lending.
A transaction foundation model isn't one deployable thing — it's a backbone that feeds two very different production surfaces. Get that split right and everything downstream follows; get it wrong and you'll try to force one serving pattern onto two incompatible workloads.
TREASURE validated both; you deploy whichever each use case needs. Toggle them:
Mapped onto the existing stack: the backbone is trained off the lakehouse (Delta/Iceberg), exported to ONNX (the format the platform already serves), and loaded by the decisioning engine for live scoring, writing risk signals to the MongoDB scoring store. Embeddings land in a feature store that the credit, fraud and recommendation models read. Nothing here is exotic — it's the same shape as the current ONNX/PMML scoring path, with a bigger, smarter model behind it.
This maps cleanly onto the data-science / software-engineering split you advocate: data science owns the model lifecycle (Tier 1 — training, the embedding artifact, model monitoring) and software engineering owns the production runtime (Tier 2 — the ONNX scorer in the decisioning engine, system monitoring). Keep "system monitoring vs model monitoring" as the explicit seam, exactly as in the current scoring engine.
The model is only as good as the sequences you feed it. The job is to turn raw platform events into per-customer, time-ordered transaction histories with a clean static/dynamic split — and to do it both in batch (for training and embeddings) and in near-real-time (for live scoring).
Pull transaction, repayment and channel events from Delta/Iceberg, join to customer/card keys, and group by customer, sorted by time — the raw-log → grouped-by-card step from the TREASURE course. Orchestrate with Airflow.
Static: product, market, channel, tenure band. Dynamic per transaction: amount (log-scaled), merchant/agent, MCC, city, time-delta — plus the outcome signals that are your analogue of Visa's decline codes (approval, decline reason, repayment status).
Map categoricals to index tables (with an OOV bucket for new merchants/agents), cap each history at the recent 512, and write sharded Parquet partitioned by market — the data-residency boundary doubles as a natural shard key.
When you build a training sequence for a decision at time T, every feature must be what was known before T — never a value computed with hindsight. Outcome signals (repayment status, decline reason) are especially dangerous: they're known only after the transaction, so they're model outputs, never inputs at the same step — exactly as TREASURE treats network signals. A leak here inflates offline metrics and collapses in production.
For real-time scoring you can't re-query the lakehouse per transaction. Maintain each customer's recent-512 sequence as a rolling state updated by Flink off the Kafka event stream, so the current history is always a cache read away at authorisation time. New transaction in → append to the rolling sequence → ready for the next score.
Where a market forbids data leaving the country, the whole pipeline — lakehouse, training, serving — runs on the in-country stack (the self-deployed lake platform), and only the model artifact moves, not the data. Partitioning sequences by market from the start makes this a deployment choice, not a re-architecture.
Tier 1. This runs rarely, on GPU, and ends in one portable artifact your runtime can serve. The TREASURE course covered the architecture; here's what productionising the training run actually involves.
The platform already serves models via ONNX/PMML, so the backbone (and the frozen heads) export to ONNX and drop into the existing serving path — no new runtime to build. This is the clean seam between Tier 1 (Python/PyTorch, data science) and Tier 2 (the JVM decisioning engine, software engineering).
import torch
model.eval() # frozen backbone + heads
# example inputs: static vec + dynamic sequence (capped at 512)
dummy = (static_vec, dynamic_seq, seq_mask)
torch.onnx.export(
model, dummy, "treasure_v1.onnx",
input_names=["static", "dynamic", "mask"],
output_names=["risk_score", "embedding"],
dynamic_axes={"dynamic": {1: "seq_len"}}, # variable history length
opset_version=17)
# ship treasure_v1.onnx to the decisioning engine + the batch embedderTwo artifacts come out of the same model: a risk_score output for the live scorer, and an embedding output for the batch embedding service. Version them together (treasure_v1, v2…) and log which version produced every score — essential for the governance in Module 5.
The live scorer runs inside the authorisation path — a lending or payment decision is waiting on it — so latency is a hard constraint, not a nice-to-have. This is where TREASURE's design choices (fixed 512 window, small model, cached sequence) pay off.
Every millisecond here sits between a customer tapping "pay" and getting an answer. The scoring component has to fit inside the overall authorisation deadline with room to spare. Build the budget:
How it stays fast: the sequence is already cached (Flink keeps it warm, Module 2), the window is fixed at 512 so the transformer's cost is bounded and predictable, the model is small and ONNX-optimised, and it runs co-located with the decisioning engine so there's no network hop. Batch where you can, but a single live score must be lean.
Embeddings don't need per-transaction freshness. Recompute them in batch (nightly, via Airflow) for active customers, write them to a feature store, and let the credit, fraud and recommendation models read them. For a customer whose behaviour just shifted sharply, trigger an off-cycle refresh — the same cheap-signal / expensive-backbone cadence trick from the PRAGMA and voice courses: recompute light live features constantly, refresh the heavy embedding only when it's worth it.
Anything on the synchronous scoring path must have a hard timeout and a safe fallback — if the model is slow or unavailable, the decisioning engine falls back to the existing rule/scorecard path rather than hanging a live transaction. A foundation model that occasionally times out must never be able to block an authorisation.
A model that influences credit and payment decisions can't be shipped on launch day confidence. It earns production through staged exposure, continuous monitoring, and governance you can defend to a regulator — the focus of the current "scale and foundations" roadmap year.
TREASURE's own authors describe a multi-stage path — extensive offline eval, parallel validation, regulatory review — before production. Mirror it; never jump to live decisions:
| System monitoring (SWE) | Model monitoring (DS) | |
|---|---|---|
| Watches | Latency, timeouts, errors, throughput, memory | Score distributions, drift, feature health, accuracy vs outcomes |
| Alarms on | p99 latency breach, fallback rate spike | Input drift, calibration decay, performance drop |
| Owner | Production runtime team | Model lifecycle team |
Keep these as separate dashboards and on-call paths. A latency regression and a model-drift event are different failures with different fixes and different owners — conflating them is how incidents get misrouted at 2am.
From the two-tier split and where it lands in our stack, through the point-in-time data pipeline, the ONNX hand-off, the millisecond scoring path, and the staged, monitored, governed rollout — you can take TREASURE from a paper to a live transaction scorer that fits the architecture, the latency, the data-residency rules, and the ownership boundaries we already run on.