imara learning library ← all courses
Foundation Models · Production · companion

TREASURE in production: from paper to a live scorer

The TREASURE course explained the model. This companion is the build: how to stand up a TREASURE-style transaction foundation model on our own stack — lakehouse to training to a real-time scorer in the decisioning engine — with the data residency, latency, monitoring and rollout realities of running it for live lending.

2
serving modes: live scorer & embedding provider
512
recent-transaction window, fixed compute budget
ONNX
export format, matching our model serving
4
rollout stages before it touches a decision
Module 01the shape of it

The deployment plan

A transaction foundation model isn't one deployable thing — it's a backbone that feeds two very different production surfaces. Get that split right and everything downstream follows; get it wrong and you'll try to force one serving pattern onto two incompatible workloads.

The Tier-1 / Tier-2 split (same as PRAGMA)

Two serving modes, one backbone

TREASURE validated both; you deploy whichever each use case needs. Toggle them:

Serving modes
The live scorer runs synchronously inside the authorisation path (tight latency). The embedding provider runs mostly in batch and writes vectors to a feature store the rest of the platform reads.

Where it sits in our architecture

Mapped onto the existing stack: the backbone is trained off the lakehouse (Delta/Iceberg), exported to ONNX (the format the platform already serves), and loaded by the decisioning engine for live scoring, writing risk signals to the MongoDB scoring store. Embeddings land in a feature store that the credit, fraud and recommendation models read. Nothing here is exotic — it's the same shape as the current ONNX/PMML scoring path, with a bigger, smarter model behind it.

The ownership boundary you already drew

This maps cleanly onto the data-science / software-engineering split you advocate: data science owns the model lifecycle (Tier 1 — training, the embedding artifact, model monitoring) and software engineering owns the production runtime (Tier 2 — the ONNX scorer in the decisioning engine, system monitoring). Keep "system monitoring vs model monitoring" as the explicit seam, exactly as in the current scoring engine.

Module 02feeding the model

The data pipeline

The model is only as good as the sequences you feed it. The job is to turn raw platform events into per-customer, time-ordered transaction histories with a clean static/dynamic split — and to do it both in batch (for training and embeddings) and in near-real-time (for live scoring).

Batch: building training sequences

Assemble from the lakehouse

Pull transaction, repayment and channel events from Delta/Iceberg, join to customer/card keys, and group by customer, sorted by time — the raw-log → grouped-by-card step from the TREASURE course. Orchestrate with Airflow.

Split static vs dynamic

Static: product, market, channel, tenure band. Dynamic per transaction: amount (log-scaled), merchant/agent, MCC, city, time-delta — plus the outcome signals that are your analogue of Visa's decline codes (approval, decline reason, repayment status).

Encode & shard

Map categoricals to index tables (with an OOV bucket for new merchants/agents), cap each history at the recent 512, and write sharded Parquet partitioned by market — the data-residency boundary doubles as a natural shard key.

Point-in-time correctness — the silent killer

When you build a training sequence for a decision at time T, every feature must be what was known before T — never a value computed with hindsight. Outcome signals (repayment status, decline reason) are especially dangerous: they're known only after the transaction, so they're model outputs, never inputs at the same step — exactly as TREASURE treats network signals. A leak here inflates offline metrics and collapses in production.

Streaming: keeping histories fresh for live scoring

For real-time scoring you can't re-query the lakehouse per transaction. Maintain each customer's recent-512 sequence as a rolling state updated by Flink off the Kafka event stream, so the current history is always a cache read away at authorisation time. New transaction in → append to the rolling sequence → ready for the next score.

In-country data residency

Where a market forbids data leaving the country, the whole pipeline — lakehouse, training, serving — runs on the in-country stack (the self-deployed lake platform), and only the model artifact moves, not the data. Partitioning sequences by market from the start makes this a deployment choice, not a re-architecture.

Module 03the backbone artifact

Training & exporting the model

Tier 1. This runs rarely, on GPU, and ends in one portable artifact your runtime can serve. The TREASURE course covered the architecture; here's what productionising the training run actually involves.

The run

Export to ONNX — the handoff to the runtime

The platform already serves models via ONNX/PMML, so the backbone (and the frozen heads) export to ONNX and drop into the existing serving path — no new runtime to build. This is the clean seam between Tier 1 (Python/PyTorch, data science) and Tier 2 (the JVM decisioning engine, software engineering).

export.py — backbone → ONNX artifact
import torch
model.eval()                                # frozen backbone + heads
# example inputs: static vec + dynamic sequence (capped at 512)
dummy = (static_vec, dynamic_seq, seq_mask)
torch.onnx.export(
    model, dummy, "treasure_v1.onnx",
    input_names=["static", "dynamic", "mask"],
    output_names=["risk_score", "embedding"],
    dynamic_axes={"dynamic": {1: "seq_len"}},   # variable history length
    opset_version=17)
# ship treasure_v1.onnx to the decisioning engine + the batch embedder

Two artifacts come out of the same model: a risk_score output for the live scorer, and an embedding output for the batch embedding service. Version them together (treasure_v1, v2…) and log which version produced every score — essential for the governance in Module 5.

Module 04scoring in milliseconds

Real-time serving

The live scorer runs inside the authorisation path — a lending or payment decision is waiting on it — so latency is a hard constraint, not a nice-to-have. This is where TREASURE's design choices (fixed 512 window, small model, cached sequence) pay off.

The latency budget

Every millisecond here sits between a customer tapping "pay" and getting an answer. The scoring component has to fit inside the overall authorisation deadline with room to spare. Build the budget:

Scoring latency · keep it under the deadline

How it stays fast: the sequence is already cached (Flink keeps it warm, Module 2), the window is fixed at 512 so the transformer's cost is bounded and predictable, the model is small and ONNX-optimised, and it runs co-located with the decisioning engine so there's no network hop. Batch where you can, but a single live score must be lean.

Serving the embeddings (the other mode)

Embeddings don't need per-transaction freshness. Recompute them in batch (nightly, via Airflow) for active customers, write them to a feature store, and let the credit, fraud and recommendation models read them. For a customer whose behaviour just shifted sharply, trigger an off-cycle refresh — the same cheap-signal / expensive-backbone cadence trick from the PRAGMA and voice courses: recompute light live features constantly, refresh the heavy embedding only when it's worth it.

The authorisation-path discipline

Anything on the synchronous scoring path must have a hard timeout and a safe fallback — if the model is slow or unavailable, the decisioning engine falls back to the existing rule/scorecard path rather than hanging a live transaction. A foundation model that occasionally times out must never be able to block an authorisation.

Module 05running it safely

Operate & roll out

A model that influences credit and payment decisions can't be shipped on launch day confidence. It earns production through staged exposure, continuous monitoring, and governance you can defend to a regulator — the focus of the current "scale and foundations" roadmap year.

Rollout: earn the decision in stages

TREASURE's own authors describe a multi-stage path — extensive offline eval, parallel validation, regulatory review — before production. Mirror it; never jump to live decisions:

Staged rollout · advance only when the stage passes
Each stage gates the next. Shadow mode scores live traffic but acts on nothing; canary lets it decide a small slice; full only after both prove out against the incumbent.

Two kinds of monitoring — your explicit seam

 System monitoring (SWE)Model monitoring (DS)
WatchesLatency, timeouts, errors, throughput, memoryScore distributions, drift, feature health, accuracy vs outcomes
Alarms onp99 latency breach, fallback rate spikeInput drift, calibration decay, performance drop
OwnerProduction runtime teamModel lifecycle team

Keep these as separate dashboards and on-call paths. A latency regression and a model-drift event are different failures with different fixes and different owners — conflating them is how incidents get misrouted at 2am.

Drift, retraining & governance

The production checklist

You've got the build, not just the idea

From the two-tier split and where it lands in our stack, through the point-in-time data pipeline, the ONNX hand-off, the millisecond scoring path, and the staged, monitored, governed rollout — you can take TREASURE from a paper to a live transaction scorer that fits the architecture, the latency, the data-residency rules, and the ownership boundaries we already run on.