Prerequisites & Foundations

The Math Behind
the Programme

You don't need to be a mathematician to lead AI product development — but six concepts will give you genuine intuition for what models actually do, why they fail, and what the numbers mean. Each module takes 15–25 minutes.

1

Probability & Uncertainty

What it means for something to be "likely" — and why LLMs are fundamentally probabilistic
Most relevant: Session 2

Every AI model is, at its core, a machine for making predictions under uncertainty. To use one well, you need a working understanding of what probability actually is — not the formula, but the concept.

What is probability?

Probability is a number between 0 and 1 that describes how likely something is. A probability of 0 means impossible. A probability of 1 means certain. A probability of 0.7 means "if you ran this situation 100 times, you'd expect this outcome about 70 times."

Crucially, probability is about your state of knowledge, not about some fixed property of the world. Two people with different information about the same borrower might assign different probabilities to default — and both can be rational.

When an LLM generates a response, it doesn't "know" the answer and retrieve it. At every position in the output, it assigns a probability to every possible next word (token), then samples from those probabilities. That's why the same prompt can produce different outputs each time — and why asking a model to "be certain" doesn't make it so.

P(default) = 0.12 ← 12% chance this borrower defaults P(no default) = 0.88 ← these must sum to 1.0 P(A and B) = P(A) × P(B) ← if A and B are independent P(A or B) = P(A) + P(B) - P(A and B)
Fintech example

A credit model outputs P(default) = 0.08 for a borrower. This doesn't mean "this borrower will definitely repay" — it means "based on all the patterns in our training data, borrowers with this profile default about 8% of the time." Out of 1,000 such borrowers, roughly 80 will default. The model cannot tell you which 80.

Conditional probability: the most important concept in AI

Conditional probability is the probability of something happening given that something else is already true. Written as P(A | B) — "probability of A, given B."

Bayes' intuition (no formula needed)

Before seeing a mobile money statement, you might estimate P(default) = 0.15 for a new applicant. Then you see the statement shows consistent salary deposits for 24 months and a growing savings balance. You update: P(default | 24 months regular income) = 0.04.

That updating process — starting with a prior belief, seeing evidence, arriving at an updated belief — is what Bayesian thinking is. Every machine learning model is doing a version of this, just with millions of examples instead of one.

Interactive: Credit decision threshold DEMO

Drag the threshold to see how changing the cutoff affects approved loans, expected defaults, and expected good loans. This is the core tradeoff in every credit AI system.

15%
The key insight for Session 2: When an LLM says something confidently, that confidence is a function of how peaked the probability distribution over possible next tokens is — not a measure of factual correctness. A model can be highly confident and completely wrong. Confidence ≠ accuracy.

Quick check

A credit model outputs a default probability of 0.22 for an applicant. Your approval threshold is 0.20 (decline anyone above this). What happens?

Correct. The threshold is a hard cutoff — if P(default) > 0.20, the application is declined. This is a business decision, not a model decision. You could choose a threshold of 0.25 and approve this person; you'd approve more loans overall and have a slightly higher default rate. The threshold is the lever that balances revenue against risk.
2

Distributions

Understanding the shape of data — and why it matters for model behaviour
Most relevant: Sessions 2, 3, 8

A distribution describes how often different values occur. Understanding distributions is essential for understanding what a model learned, what data it expects, and why it degrades when the world changes.

The normal distribution

The bell curve. Values cluster around a mean (average), and get rarer as you move further away. Described by two numbers: the mean (where the centre is) and the standard deviation (how spread out the bell is). About 68% of values fall within 1 standard deviation of the mean; 95% within 2.

Many real-world measurements approximate a normal distribution: loan amounts, income levels, transaction counts. Machine learning models often assume their training data looks roughly normal — if your production data has a different shape, performance degrades.

Interactive: Distribution shape explorer DEMO

Adjust mean and spread to see how a distribution changes. Think of this as "monthly income distribution of loan applicants."

35k
10k

Drift: when distributions change

A model trained on data from one distribution will struggle when production data comes from a different distribution. This is called distribution shift or drift. It's the single most common cause of AI model degradation in the field.

Interactive: Training vs. production distribution drift DEMO

Blue = training data distribution. Orange = what production data looks like now. When they diverge, the model is operating outside what it learned. Slide to simulate an economic shock.

None
Fintech example — PSI (Population Stability Index)

PSI measures how much a distribution has shifted from a reference baseline. In credit: if the income distribution of your applicants today looks materially different from when you trained your model, PSI will be high. A PSI under 0.1 means stable. 0.1–0.2 means monitor. Above 0.2 means the model likely needs retraining. PSI is just a formula that quantifies the area between two distributions — you don't need to compute it by hand, but knowing what it measures tells you when to trust the alert.

Temperature: controlling an LLM's distribution

When an LLM generates the next token, it produces a probability distribution over the entire vocabulary. Temperature is a parameter that reshapes that distribution before sampling.

Interactive: LLM temperature effect DEMO

The model is choosing the next word after "The applicant's income is..." The bars show the probability assigned to each possible next token. Temperature reshapes the distribution.

1.0
The key insight for Sessions 2 & 8: When your training data distribution and your production data distribution diverge, the model hasn't "broken" — it's doing exactly what it learned to do, but the world has changed. Drift detection is just measuring that divergence quantitatively before it causes a business problem.
3

Vectors & Similarity

How AI systems represent meaning as numbers — the foundation of embeddings and semantic search
Most relevant: Sessions 2, 5

An embedding is a list of numbers that represents the meaning of something — a word, a sentence, a loan application, a customer profile. The reason AI systems can find semantically similar documents (not just keyword matches) is that similar meanings produce similar number-lists, and "similar number-lists" is something we can compute precisely.

What is a vector?

A vector is simply a list of numbers. In 2D: [3, 4]. In 3D: [1, 0, 7]. Embedding models produce vectors with hundreds or thousands of dimensions — but the intuition is the same as 2D. Each dimension captures some aspect of meaning.

Think of it this way: if you described a loan applicant using just two numbers — "income stability" (0–10) and "repayment history" (0–10) — you could plot them on a graph. Applicants with similar profiles would cluster together. That's all an embedding is doing, but with 1,536 dimensions instead of 2.

Interactive: 2D semantic space DEMO

Each dot represents a concept embedded in a simplified 2D space. Similar concepts cluster together. Click a concept to see which others are closest to it (highest similarity).

Click any concept to see its nearest neighbours.

Cosine similarity: measuring how close two vectors are

If two vectors point in the same direction, they are similar. If they point in perpendicular directions, they are unrelated. If they point in opposite directions, they are antonyms. Cosine similarity measures the angle between two vectors — returning a number from -1 (opposite) to 1 (identical).

cosine_similarity(A, B) = (A · B) / (|A| × |B|) In plain English: divide the dot product of the two vectors by the product of their lengths. Result between -1 and 1. similarity("defaulted on loan") vs ("missed payment") → ~0.82 very similar similarity("missed payment") vs ("salary received") → ~0.11 unrelated similarity("creditworthy") vs ("high risk") → ~-0.34 somewhat opposite
Fintech example — why this matters for RAG

When you build a RAG system over your lending policy documents, a loan officer types: "What happens if a borrower misses two payments?" The system embeds that query and finds the policy chunks whose embeddings are closest — which will include the "payment arrears" section, the "restructuring policy" section, and the "collections procedure" — even if none of those contain the exact words "misses two payments." That's semantic search: finding by meaning, not by keyword.

Interactive: Cosine similarity (2D) DEMO

Drag the sliders to set two 2D vectors. Watch the angle between them and the similarity score update.

30°
80°
The key insight for Session 2 & 5: Embeddings let you turn the fuzzy concept of "semantic similarity" into a precise number. This is what makes RAG work: retrieving not the documents that match keywords, but the documents that match meaning. The math is just a way to compute "how close are these two things in meaning-space."
4

Model Evaluation Metrics

Precision, recall, AUC — what they actually mean and which one to care about
Most relevant: Sessions 3, 4, 8

The single most important skill in AI product leadership is knowing how to ask "how good is this model?" in a way that maps to business outcomes — not just technical metrics. Different metrics measure different things, and optimising for the wrong one produces models that look good in presentations and fail in production.

The confusion matrix

Every binary prediction (default / not-default, fraud / legitimate, approve / decline) produces four possible outcomes:

Model predicted
Repaid (Negative) Defaulted (Positive)
Actual Repaid True Negative (TN)
Correctly declined a good borrower? Wait — no. Correctly identified a repayer.
False Positive (FP)
Predicted default, but they repaid. A missed opportunity.
Defaulted False Negative (FN)
Predicted repayment, but they defaulted. A bad loan.
True Positive (TP)
Correctly predicted default. Avoided a bad loan.

The four metrics you'll encounter

Accuracy = (TP + TN) / total. The percentage of all predictions that were correct. Misleading when classes are imbalanced — a model that says "everyone repays" is 92% accurate if 92% of borrowers repay.

Precision = TP / (TP + FP). Of all the defaults the model predicted, what fraction actually defaulted? High precision = when the model flags a default, it's usually right.

Recall = TP / (TP + FN). Of all the actual defaults, what fraction did the model catch? High recall = the model misses few actual defaults.

AUC (Area Under the ROC Curve) = a single number (0–1) measuring the model's ability to distinguish between the two classes regardless of threshold. 0.5 = no better than random. 0.9 = excellent. This is the standard summary metric for credit models.

Interactive: The precision-recall tradeoff DEMO

Move the decision threshold to see how precision and recall change. Notice that improving one comes at the cost of the other. Which matters more depends on your business — bad loans vs. missed good customers.

30%
Fintech example — which metric actually matters

A colleague shows you a model with 94% accuracy. Before celebrating, ask: what is the base rate of default in your portfolio? If it's 5%, a model that approves everyone has 95% accuracy. Your colleague's model may be worse than doing nothing. The right question is: what is the AUC? And then: at the threshold you'd actually use in production, what is the precision and recall, and what do those translate to in KES of bad debt avoided vs. good loans lost?

Disaggregated evaluation: checking for bias

A model with good overall AUC can still be biased if it performs significantly worse on a subgroup. This requires computing the same metrics separately for different groups.

Aggregate evaluation (misleading): Overall AUC = 0.87 ✓ Disaggregated evaluation (reveals the problem): AUC — salaried customers: 0.91 AUC — informal sector customers: 0.71 ← material disparity AUC — urban customers: 0.89 AUC — rural customers: 0.76 ← material disparity
The key insight for Sessions 3 & 8: Always ask "accurate on average, for whom?" A model that performs well on salaried urban customers and poorly on rural informal workers is not an accurate model — it's an accurate model for a subset that is not representative of your full market.
5

Statistical Thinking

Sample sizes, confidence intervals, and why small samples lie to you
Most relevant: Sessions 3, 4, 8

Statistical thinking is the discipline of asking "how much data do I need before I can trust this result?" and "how wide is the range of outcomes that are consistent with what I've observed?" These questions come up constantly in AI product development: evaluating model performance, interpreting A/B tests, deciding whether a model is degrading.

The law of large numbers

The average of many observations tends to the true mean as sample size increases. With 10 coin flips, you might see 8 heads — that doesn't mean the coin is biased. With 10,000 flips, if you see 8,000 heads, the coin is almost certainly biased.

In AI evaluation: measuring model accuracy on 20 test examples tells you almost nothing. Measuring on 1,000 examples gives you a reliable estimate. The larger the sample, the narrower the range of uncertainty around your estimate.

Confidence intervals

A confidence interval (CI) is a range that, with high probability, contains the true value you're trying to measure. "The model's accuracy is 87% ± 3%" (at 95% confidence) means: if you ran this evaluation many times, 95% of the intervals produced would contain the true accuracy.

The width of a CI depends on sample size. Smaller sample = wider interval = more uncertainty. An evaluation on 50 examples might give you "accuracy = 84%, 95% CI: 72%–92%". That range is so wide that 84% is almost meaningless — the true accuracy could be anywhere in that range.

Interactive: How sample size affects confidence DEMO

You've observed that your model correctly classifies X% of test cases. How wide is your confidence interval? Slide to change sample size and see how certainty changes.

85%
100

Correlation vs. causation

Machine learning models find correlations in data. They cannot, on their own, tell you whether those correlations reflect causation. This distinction matters enormously in credit:

Fintech example — correlation trap

A model discovers that applicants who apply for loans on Sunday mornings have 40% lower default rates. This is a correlation. The causal story might be: people with stable employment and regular weekend routines apply on Sunday mornings; it's the employment stability that predicts repayment, not the time of application. If you deploy a policy that gives preferential rates to Sunday morning applicants, you're acting on a spurious correlation that will break as soon as application patterns change.

The key insight for Session 3: When a data scientist shows you a model's performance on a validation set, the first question is always: how large is that set? An 89% accuracy claim on 50 examples has a confidence interval so wide as to be nearly meaningless. You should not be confident in model performance estimates until you have at least 300–500 examples in your evaluation set, and ideally 1,000+.
6

Cost & Scale Math

The arithmetic of AI product economics — inference costs, break-even, and unit economics
Most relevant: Sessions 4, 9

AI product economics requires building a simple cost model before you commit to a technical approach. The math involved is straightforward arithmetic — but the failure to do it at all, before deployment, is one of the most common causes of "we built the wrong thing" discoveries at scale.

Token costs: how LLM pricing works

Tokens and pricing

LLM providers charge per token. A token is roughly 3–4 characters or 0.75 words in English — African languages typically use more tokens per word because of morphological richness. Frontier models charge separately for input tokens (your prompt) and output tokens (the model's response). Input is usually cheaper.

The cost of a single API call = (input tokens × input price) + (output tokens × output price).

Example: Mobile money statement extraction Input: ~3,500 tokens (system prompt + statement text) Output: ~400 tokens (structured JSON result) At GPT-4o pricing (~$2.50/M input, ~$10/M output): Input cost = 3,500 / 1,000,000 × $2.50 = $0.00875 Output cost = 400 / 1,000,000 × $10 = $0.00400 Cost/call = $0.01275 ≈ $0.013 Monthly at different volumes: 10,000 calls/month → $130/month 100,000 calls/month → $1,300/month 1,000,000 calls/mo → $13,000/month
Interactive: AI inference cost calculator DEMO

Estimate monthly AI costs for your use case. Adjust parameters to see how volume and prompt length drive costs.

10,000
3,500
400

Break-even and unit economics

Unit economics formula for an AI lending feature: Revenue per application (AI-attributable): = approval_rate_lift × avg_loan_revenue = 0.12 × $8.50 = $1.02 Cost per application: = inference_cost + human_review_share + infra_amortised = $0.013 + $0.045 + $0.012 = $0.070 Contribution margin per application = $1.02 - $0.07 = $0.95 Fixed investment to recover: = engineering_build + evaluation_dataset + ongoing_model_ops = $45,000 + $8,000 + $3,000/month Break-even volume (ignoring ongoing ops): = $53,000 / $0.95 ≈ 55,800 applications
Interactive: Break-even calculator DEMO

Adjust revenue lift and cost per application to find your break-even point and payback period.

$1.00
7¢
$53k
30,000
The key insight for Sessions 4 & 9: AI cost scales linearly with volume (each call costs roughly the same). Revenue from AI features often scales super-linearly (better models unlock new customers, not just cheaper processing of existing ones). The economic case for AI in lending is usually not cost reduction — it's market expansion. Model the revenue side at least as carefully as the cost side.
Summary reference

Quick-reference: all concepts in one place

ConceptWhat it isWhy it matters for AI product leadersKey session
ProbabilityA number 0–1 describing how likely something isLLM outputs are probability distributions, not facts. Confidence ≠ correctness.S2
Conditional probabilityP(A | B) — likelihood of A given B is already trueHow models update beliefs given evidence; foundation of all credit scoringS2, S3
DistributionHow often different values occur in a datasetModels learn the training distribution. When production data shifts, performance degrades.S2, S8
Mean & std deviationCentre and spread of a distributionDescribes your data; alerts you when it changes (drift monitoring)S3, S8
TemperatureA parameter that sharpens or flattens an LLM's output distributionLow temp = deterministic. High temp = creative but inconsistent. Use low temp in production.S2
VectorA list of numbers representing a point in multi-dimensional spaceEmbeddings are vectors. Meaning is distance in vector space.S2, S5
Cosine similarityMeasures the angle between two vectors (0=unrelated, 1=identical)How semantic search finds relevant documents. The math behind RAG retrieval.S2, S5
Confusion matrixTP/TN/FP/FN breakdown of a classifier's errorsReveals what kind of mistakes a model makes. Essential for evaluating credit models.S3, S4
Precision vs. recallPrecision: are flagged positives right? Recall: are all positives caught?Choosing which to optimise is a business decision, not a technical oneS3, S4
AUCSummary metric for a classifier (0.5=random, 1.0=perfect)Standard credit model quality metric. Should always be supplemented with disaggregated eval.S4, S8
Confidence intervalRange that likely contains the true value of a metricTells you how much to trust an evaluation result. Small samples = wide intervals = unreliable.S3, S4
Distribution shift / driftWhen production input distributions diverge from training distributionsThe most common cause of model degradation. PSI quantifies it.S8
Unit economicsRevenue and cost attributable to a single unit of the productThe economic viability of an AI feature. Must be modelled before committing to architecture.S4, S9
Token costLLM API charge per token processedThe marginal cost of AI inference. Scales linearly with volume.S4