WEEK 02

AI Mental Models and Systems Thinking

For Product Teams
LLMs · Embeddings · Failure modes · Systems thinking
Reading time~60–90 min
FormatRead · Apply · Deliver
ContextAfrican fintech examples

In this session

You do not need to understand the mathematics of transformer architecture to lead AI product development effectively. But you do need accurate mental models of what LLMs are, how they behave, why they fail, and how to communicate about them with both technical and non-technical stakeholders. Inaccurate mental models cause specific, expensive problems. Product managers who believe LLMs are 'databases with natural language interfaces' build systems that fail on edge cases because they expect retrieval precision from a generative system. CTOs who believe LLMs are 'just autocomplete' underestimate their capability and overestimate their reliability. Engineers who believe fine-tuning is 'always the answer to poor performance' spend months on the wrong solution. This session builds the mental models you need to make good decisions about AI systems without requiring mathematical depth. It is a conceptual foundation, not a technical tutorial.

What a large language model actually is

A large language model is a statistical model of language — specifically, of the probability distribution over what token should come next given the tokens that preceded it. Training a model means adjusting billions of numerical parameters (weights) so that the model's predictions match the patterns in the training corpus. That is it. There is no database, no reasoning engine, no fact-checker, no memory system, and no understanding in the human sense of the word. This framing — LLM as next-token predictor — is uncomfortable because it seems to trivialise what these systems can do. A next-token predictor should not be able to write code, reason about multi-step problems, or produce coherent long-form analysis. But it can, and understanding why requires a conceptual step. The training corpora for frontier LLMs contain vast quantities of human-generated text that demonstrates reasoning, problem-solving, and explanation. By learning to predict what comes next in this text, the model learns the patterns of reasoning itself. It is not reasoning in the way a human reasons — it has no causal model of the world — but it has learned a very accurate statistical approximation of what reasoning looks like in text. For many practical tasks, this approximation is sufficient. Tokens and context windows. LLMs do not process words — they process tokens, which are character sequences of variable length (typically 3–4 characters on average for English). 'Nairobi' is one token; 'creditworthiness' might be two or three. This matters for African languages, which are often morphologically richer than English and therefore require more tokens per word, which has cost and context-window implications. The context window is the maximum amount of text the model can 'see' at once during inference. In mid-2025, frontier models have context windows of 100,000–200,000 tokens (roughly 75,000–150,000 words). This sounds large, but there is an important nuance: performance degrades on information that appears in the middle of a very long context window — a phenomenon called 'lost in the middle'. For document processing tasks, the implication is that critically important information should be positioned at the beginning or end of the prompt, not buried in the middle. Temperature and sampling. LLM outputs are probabilistic. At each token position, the model assigns a probability to every possible next token and samples from that distribution. Temperature is a parameter that controls how peaked or flat that distribution is. High temperature (e.g. 1.5) produces more varied, creative outputs by flattening the distribution. Low temperature (e.g. 0.1) produces more deterministic, conservative outputs by sharpening the peak. For production AI systems where consistency and accuracy matter — loan decision explanations, document extraction, fraud detection — low temperature is almost always correct. Temperature is a common misconfiguration in production systems built by teams who tested at default settings.

Embeddings and semantic search

Alongside generative models, the other category of AI system you will work with most frequently is embedding models. An embedding model converts a piece of text into a dense numerical vector — a list of hundreds or thousands of floating-point numbers — that encodes the semantic meaning of the text. Texts with similar meanings produce similar vectors. This makes possible a family of operations that are fundamental to production AI systems: Semantic search. Instead of searching for exact keyword matches, embed both the query and the documents in the same vector space and find the documents whose vectors are closest to the query vector. 'Find loans where the borrower mentioned job loss' will surface documents containing 'made redundant', 'business closed down', 'lost my work' — exact keyword search would miss all of these. Retrieval-augmented generation (RAG). The core pattern for grounding LLM outputs in factual, up-to-date, or private information. Embed a knowledge base (policy documents, product documentation, historical data), embed the user's query, retrieve the most semantically relevant chunks, and include them in the LLM prompt as context. The LLM then generates its response grounded in the retrieved information rather than relying on what it learned during training. We will cover RAG architecture in depth in Session 5. Classification and clustering. Embed a set of items (loan applications, customer complaints, transaction descriptions) and use the resulting vectors as features for downstream classification or clustering tasks. This is often dramatically more effective than manual feature engineering, because embeddings encode semantic relationships that hand-crafted features miss. The key design decision with embeddings is choosing the right embedding model for the language and domain. Frontier embedding models (e.g. text-embedding-3-large from OpenAI, or embedding models from Cohere) are trained predominantly on English text and perform well on English. Performance on African languages is variable — often acceptable for major languages like Swahili and Amharic, but potentially poor for lower-resource languages. This is a data strategy problem as much as a model selection problem.

Why AI systems fail: a taxonomy of failure modes

The single most important mental model shift for product leaders transitioning to AI-native development is moving from deterministic thinking to probabilistic thinking. Traditional software either works or it does not — a function either returns the correct value or it does not. AI systems exist on a spectrum of correctness, and they can be 'almost right' in ways that are just as dangerous as 'completely wrong'. Here is a taxonomy of the failure modes you will encounter in production: Hallucination. The model generates plausible-sounding but factually incorrect content. This is not a bug — it is an emergent property of next-token prediction. The model produces the token sequence that is statistically likely given the context; if the correct information is not in its training data or its context window, it will produce a confident-sounding guess. Hallucination rates vary significantly by task: factual recall about obscure or recent topics hallucinates at high rates; reasoning about well-structured problems in the prompt hallucinates at lower rates. The mitigation is architecture (grounding the model with retrieved facts via RAG) not prompting (asking the model to 'only say things it's certain about' is largely ineffective). Context blindness. Models have limited ability to track long-range dependencies within a context window, even when the context window is technically large enough to contain the relevant information. A model asked to summarise a 50-page document may produce a summary that ignores key facts mentioned in the middle sections. In financial document processing, this can cause material errors. Distribution shift. A model (or a fine-tuned model) trained on historical data degrades when the distribution of inputs changes — new product types, new user demographics, regulatory changes that alter application patterns, economic events that change repayment behaviour. This is a fundamental challenge in African market contexts where economic and market conditions change more rapidly than in mature markets. Detecting and managing distribution shift is a core MLOps responsibility (Session 8). Prompt sensitivity. LLM outputs are surprisingly sensitive to small changes in prompt wording. Changing 'classify this transaction as fraud or legitimate' to 'is this transaction fraudulent?' can produce different outputs on identical inputs, because the model associates different response patterns with different phrasings. This creates testing and reproducibility challenges that traditional software development does not have. The mitigation is a rigorous prompting discipline (treating prompts as code, version-controlling them, testing changes systematically) rather than treating prompts as informal instructions. Sycophancy. Models trained with reinforcement learning from human feedback (RLHF) — which includes all frontier models — tend to agree with users rather than correct them. If a user says 'This borrower seems creditworthy to me — what do you think?' the model will tend toward agreement even if the evidence in the context suggests otherwise. In high-stakes applications like credit decisioning, this is a serious risk. Mitigation: design prompts that ask for structured analysis before asking for a conclusion, and always separate the information-gathering prompt from the decision-support prompt. Emergent capability failures. LLMs have surprising capabilities that only emerge at scale. They also have surprising capability gaps that only become apparent in production. A model that handles 98% of a task perfectly may fail catastrophically on the remaining 2% in ways that no amount of pre-production testing surfaced. This is not a reason to avoid production deployment — it is a reason to design systems with failure detection, graceful degradation, and human escalation paths from day one.

Systems thinking for AI products

AI products are not isolated components — they are embedded in sociotechnical systems that include users, operators, regulators, data pipelines, and feedback mechanisms. Systems thinking is the discipline of reasoning about these interconnections rather than treating the AI model as the unit of analysis. Two systems thinking concepts are particularly important for AI product leaders: Feedback loops. AI systems that make decisions or recommendations that affect user behaviour create feedback loops. A credit scoring model that systematically denies credit to a certain demographic will, over time, reduce the volume of data the system has about that demographic — which can cause the model's performance on that demographic to degrade further. A recommendation system that surfaces certain content will change user behaviour in ways that alter the training data for the next model version. Understanding the feedback loops in your system before deployment is essential for predicting long-term behaviour. In African fintech specifically, there is a critical feedback loop around thin-file customers. A model trained on data from customers who have already received and repaid loans cannot directly generalise to customers with no loan history. If the model is used as a hard gate (deny any thin-file applicant), it perpetuates the thin-file problem. A thoughtfully designed system would include an explicit exploration strategy — granting small loans to some thin-file applicants specifically to generate data — rather than treating the model's output as definitive. Sociotechnical alignment. An AI system's behaviour in production is determined by more than the model. It is determined by how outputs are presented to users or decision-makers (framing effects), by what actions are available downstream (if the only option is approve/decline, a nuanced probability output becomes a hard threshold), and by the incentives of the operators (loan officers whose performance is measured by throughput will use AI pre-screening differently from loan officers measured by portfolio quality). Designing for sociotechnical alignment means thinking about all of these factors, not just the model accuracy metric.

Communicating AI to stakeholders

A recurring challenge for technical leaders in AI product development is translating between the probabilistic, nuanced world of AI systems and the binary, confident language that business stakeholders often prefer. There are three communication principles worth internalising: Precision without jargon. 'Our model has a 91% AUC on the validation set' is meaningless to a CFO. 'Our model correctly identifies 87% of loans that will default, at the cost of declining 12% of loans that would have repaid' is actionable. Lead with business implications, not model metrics. Uncertainty as a feature, not a weakness. Business stakeholders often interpret AI uncertainty as a failure. 'The model is only 80% confident on this application' sounds weaker than '100% of our decisions are deterministic.' The reframe: the model's uncertainty estimate is valuable information that a rule-based system cannot provide. A model that says 'I am 60% confident this borrower will repay' is being more useful than a rule that says 'approved' — if you design the system to act on that uncertainty information (manual review, smaller initial limit, more frequent check-ins). What the system cannot do. The most important communication obligation for an AI product leader is to clearly define the limits of the system before deployment. What types of inputs will it handle poorly? What decisions should always go to a human? What does 'low confidence' mean operationally? These questions should be answered in writing, reviewed by the team, and communicated to operations and compliance before go-live. The alternative — letting people discover the limits during an incident — is more expensive and more damaging to trust.

Self-check questions

Before moving to the next session, make sure you can answer these without re-reading. If you cannot, go back to the relevant section.

Session deliverable

AI System Mental Model Map

Map out the AI component of your product (real or hypothetical) — what it knows, what it assumes, where it is likely to fail, and what the downstream consequences of those failures are.

Show worked example ↓
Mobile money statement income extractor
Show worked example ↓
PDFs of mobile money statements (M-Pesa, Airtel Money, MTN MoMo) uploaded by loan applicants. Variable format quality — some are clean digital PDFs, some are photos of printed statements. Expected completeness: 3 months minimum, though many applicants submit only 1 month.
Show worked example ↓
Structured JSON: estimated monthly income (point estimate + confidence interval), income source classification (salary/business/remittance/mixed), recurrence score (0-100), and anomaly flags (unusual large credits, round-number patterns suggesting fabrication).
Show worked example ↓
1. Format diversity failure: WhatsApp screenshot statements and handwritten printouts cause OCR errors that cascade into incorrect extraction — worst for rural applicants who cannot access clean digital statements. 2. Distribution shift: a new mobile money provider enters the market; the extractor has no training examples for their statement format and silently produces wrong outputs. 3. Context blindness: a 6-month statement with complex multi-source income gets the early months ignored in the summary.
Show worked example ↓
If the system systematically under-estimates income for informal business owners (because transaction patterns are less regular than salary deposits), those applicants get declined. Declined applicants produce no repayment outcome data. The model never learns that they were creditworthy. This perpetuates a bias against informal sector borrowers — who are a large fraction of the TAM.
Show worked example ↓
The system should not be the final decision on applications where: (a) confidence interval is wide (>±40% of point estimate), (b) statement covers less than 2 months, (c) income source is classified as 'mixed' with business component >50%. In these cases, route to human review with the extraction as supporting context, not as a recommendation.
← Week 01 Week 03 →