WEEK 04

Model Selection, Cost Analysis

And Build-versus-Buy
Model evaluation · TCO modelling · Fine-tuning vs RAG · Infrastructure constraints
Reading time~60–90 min
FormatRead · Apply · Deliver
ContextAfrican fintech examples

In this session

Choosing a model is not a technical decision — it is a business decision with technical constraints. The question is never 'which model performs best on the benchmark?' It is 'which model delivers the best outcome for this specific task, at acceptable cost, within our infrastructure constraints, and with acceptable dependency risk?' This session gives you a structured framework for making that decision. We will cover how to evaluate models for specific tasks rather than general capability, how to model the true cost of different approaches, and how to think about the build/fine-tune/buy decision with honesty about what each path actually requires.

Why benchmarks mislead and how to evaluate models for your task

Model benchmarks — MMLU, HumanEval, GSM8K, and the dozens of others published by model providers and academic labs — measure performance on standardised test sets. They are useful for understanding a model's general capability envelope. They are almost useless for predicting performance on your specific task. The gap between benchmark performance and task-specific performance exists for several reasons. Benchmark tasks are designed to be well-specified, clean, and unambiguous. Production tasks are messy: inputs have formatting issues, edge cases are common, the correct output is often context-dependent. Benchmarks measure what a model can do; production performance measures what a model does on the actual distribution of your inputs. The only way to evaluate model performance for your task is to run the models you are considering on a representative sample of your actual data. This requires an evaluation dataset: a set of (input, expected output) pairs drawn from the real distribution of your task. Building this evaluation dataset is not optional — it is the foundation of every model selection decision, model version comparison, and production performance monitoring. A practical approach for building an evaluation dataset when you have limited labelled data: start with 100–200 examples where you are highly confident of the correct output (the 'easy' cases), supplement with 50–100 examples that represent known edge cases (the challenging inputs), and include 50 examples where the correct output requires judgment (the 'hard' cases). This gives you a 200–400 example evaluation set that, while imperfect, is far more predictive than any public benchmark. For tasks where human judgment is the ground truth (does this explanation sound natural to a customer? does this document summary correctly capture the key terms?), you need human evaluators who are representative of your actual use context. For fintech AI in African markets, this means evaluators who speak the relevant languages and understand the local financial context — not a crowd-sourced labelling platform with primarily English-speaking evaluators.

The model landscape: frontier, open-source, and fine-tuned

The model options available in mid-2025 fall into three broad categories, each with distinct tradeoffs: Frontier API models (GPT-4o, Claude Sonnet/Opus, Gemini Pro/Ultra, Command R+) offer the highest general capability with minimal setup effort. You call an API and get state-of-the-art outputs. The tradeoffs are cost (API pricing at scale can be significant), latency (round-trip to a US or EU data centre adds 200–800ms, which matters for real-time user interactions), data privacy (your inputs leave your infrastructure), and dependency risk (a provider pricing change or terms-of-service change can materially affect your unit economics). Open-source models (Llama 3.x, Mistral, Qwen, Phi-3, Gemma) can be self-hosted, which eliminates the data privacy concern and reduces marginal cost at scale (you pay for compute, not per-token pricing). The tradeoffs are infrastructure complexity (you need to provision, manage, and scale GPU inference infrastructure), a capability gap relative to frontier models on complex tasks (though this gap has been closing rapidly), and the engineering cost of maintaining the deployment. Fine-tuned models are base models (usually open-source, sometimes frontier models that support fine-tuning) that have been adapted to a specific task or domain using your own data. Fine-tuning can dramatically improve performance on a specific, well-defined task, and can produce a smaller model that outperforms a larger general model on that task (better performance at lower inference cost). The requirements are a fine-tuning dataset of sufficient quality and volume (typically 1,000–50,000 examples depending on the task), GPU compute for training (manageable cost for small models on cloud platforms), and an MLOps process for managing model versions. For African market AI specifically, there is a fourth category worth naming: African language specialists. Models from Lelapa AI (Afrikaans, isiZulu, isiXhosa), ToumAI (multiple African languages including Arabic dialects, Swahili, Somali), and Masakhane (research models, multiple languages) fill a capability gap that neither frontier nor standard open-source models address well. For any product where language quality in African languages is important, evaluating these options against frontier models on your specific language tasks is worth the effort.

Total cost of ownership modelling

The 'cost' of a model is not the price per token. Total cost of ownership (TCO) for an AI feature includes several components that are frequently underestimated: Inference cost is the most visible component — API pricing or compute cost for self-hosted models. For API pricing, calculate: (average tokens per request × cost per 1M tokens × expected request volume per month). For a mobile money statement extraction task, a single request might consume 4,000 input tokens and generate 500 output tokens. At GPT-4o pricing of approximately $2.50/M input and $10/M output tokens, that is approximately $0.015 per extraction. At 50,000 extractions per month, that is $750/month — manageable. At 500,000, it is $7,500. Scale the numbers to your expected volume. Evaluation and iteration cost is the engineering time to build your evaluation dataset, run comparative evaluations when you consider changing models, and iterate on prompts or fine-tuning. For a two-engineer team, plan for 2–4 weeks of effort per significant model change. This is a fixed cost that many teams amortise poorly — they incur it repeatedly because they did not invest in a proper evaluation framework the first time. Integration and maintenance cost covers the ongoing engineering effort to maintain the integration, handle API changes, manage model version deprecation, and update prompts as the model changes. Frontier model providers regularly update and deprecate model versions. If your production system is tightly coupled to a specific model version (as it often should be for reproducibility), each deprecation triggers a testing and migration effort. Latency cost is the business cost of slow responses — reduced user satisfaction, lower completion rates for multi-step flows. For a USSD-based loan application in Africa, a 3-second response is frustrating; a 10-second response causes session abandonment. Latency is not just a user experience concern — it translates directly to conversion rate and application completion metrics. Fine-tuning investment, if applicable, includes the one-time cost of building the training dataset, running the fine-tuning job (GPU hours), evaluating the fine-tuned model, and the ongoing cost of re-tuning as your data distribution evolves. A useful heuristic: for tasks where volume is low and capability requirements are high, frontier API models are almost always the right choice. For tasks where volume is high and the task is well-defined and narrow, fine-tuned open-source models typically produce better TCO at scale. The crossover point varies by task and provider pricing, but is often around 1–5 million inferences per month.

The build-versus-buy decision framework

Build vs. buy in AI has a specific meaning: build means training or fine-tuning your own model (or building a custom RAG system), buy means using a model as-is via API with prompt engineering, and fine-tune sits between them — using a base model but adapting it with your data. The framework for making this decision has three questions: Can prompt engineering alone achieve the required performance? Before investing in fine-tuning or custom model development, invest seriously in prompt engineering. A well-engineered prompt — with clear instructions, few-shot examples, chain-of-thought scaffolding, and appropriate output formatting — can achieve performance close to fine-tuning for many tasks. Spend 1–2 weeks on prompt engineering before concluding that fine-tuning is necessary. The comparison should be made on your evaluation dataset, not on a heuristic. Do you have a proprietary data advantage? Fine-tuning is most defensible when you have proprietary training data that competitors cannot access — your own historical transaction data, your own labelled document corpus, your own customer interaction history. If you are fine-tuning on publicly available data or synthetic data, any competitor can do the same thing. A fine-tuned model built on proprietary data is a genuine competitive moat; a fine-tuned model built on public data is a temporary advantage at best. What is the cost of being wrong? High-stakes decisions — credit approvals, fraud determinations, identity verification — require a higher standard of reliability and auditability than low-stakes ones. For high-stakes applications, a well-evaluated API model with robust testing and human oversight is often more defensible than a fine-tuned model with higher average performance but less-understood failure modes. Regulators and auditors want to know what the system does when it gets it wrong; this is an easier question to answer for a well-understood frontier model than for a custom fine-tune.

Infrastructure constraints in African market deployments

Model selection cannot be made in isolation from the infrastructure environment in which the model will run. African market deployments impose a specific set of constraints that should be inputs to the model selection decision, not afterthoughts. Latency from data centre distance. The major AI API providers host their primary inference infrastructure in the US and Western Europe. Round-trip latency from East or West Africa to these data centres adds 200–600ms to every API call. For synchronous user-facing applications, this is material. Mitigation options: use providers with regional inference endpoints (AWS Bedrock in the af-south-1 Cape Town region; Google Cloud in the me-west1 Tel Aviv region, which is the closest to East Africa); implement aggressive caching of common responses; design flows to make AI calls asynchronously where possible. Connectivity reliability. Even with adequate bandwidth on average, African networks have higher variance in connectivity than Western networks. An AI feature that requires a reliable connection for every user interaction will have unpredictable failure rates. The design principle is: AI processing should fail gracefully, with a human-fallback path available for every AI-gated flow. A loan application that cannot proceed because the AI extraction API timed out is a worse experience than a loan application that routes to manual review. Cost sensitivity. In markets where average revenue per user is lower than in developed markets, the unit economics of AI inference are more constrained. A feature that costs $0.05 per interaction is trivially affordable in a US consumer product with $50 ARPU; it is material in a micro-lending product with $2 ARPU. This pushes toward smaller, more efficient models and aggressive prompt optimisation (shorter prompts at the same quality). Regulatory data residency. Several African markets have or are developing data residency requirements — mandating that certain categories of data (financial data, personal data) be stored and processed within the country or continent. A model hosted in the US that receives financial data about Kenyan customers may create compliance exposure under Kenya's Data Protection Act. Understanding data residency requirements in each operating market, and designing your AI infrastructure to satisfy them, is a legal requirement, not an optional consideration.

Self-check questions

Before moving to the next session, make sure you can answer these without re-reading. If you cannot, go back to the relevant section.

Session deliverable

Model Selection Decision Matrix

Complete a scored decision matrix comparing at least two model options for a specific AI use case. Include capability, cost, latency, and dependency risk dimensions.

Show worked example ↓
Mobile money statement income extraction — structured JSON output from PDF statements across 3 mobile money providers
Show worked example ↓
GPT-4o (OpenAI API) — prompt-engineered, no fine-tuning
Show worked example ↓
Mistral 7B fine-tuned on 12,000 labelled statement extractions, hosted on AWS EC2 g4dn.xlarge in af-south-1
Show worked example ↓
GPT-4o scores 91% accuracy on our 300-example evaluation set; Mistral 7B base scores 71%; fine-tuned Mistral 7B scores 88%. GPT-4o handles novel statement formats better (higher accuracy on the 50 edge-case examples: 83% vs 74%). Fine-tuned Mistral degrades more on out-of-distribution formats — needs retuning when new MNO statement formats are introduced.
Show worked example ↓
At 50,000 extractions/month: GPT-4o ≈ $750/month API cost + negligible infra. Fine-tuned Mistral: $320/month compute (g4dn.xlarge reserved) + ~$2,000 one-time fine-tuning investment + 3 weeks engineer time. Break-even at ~4 months. At 500,000/month, GPT-4o = $7,500; Mistral = $640. Cost case for fine-tuning is clear at high volume.
Show worked example ↓
GPT-4o: 800–1,400ms average from Nairobi (to EU endpoints). AWS af-south-1 inference: 180–320ms. For async processing (not real-time user-facing), GPT-4o latency is acceptable. For real-time statement assessment during USSD flow, af-south-1 deployment is required.
Show worked example ↓
Recommend GPT-4o for the next 6 months while volume is below 100,000/month and to avoid fine-tuning investment before evaluation set is larger. Invest in growing the labelled evaluation and training set to 20,000+ examples. Re-evaluate fine-tuned Mistral at 6-month review when volume trajectory is clear and fine-tuning dataset is sufficient. Use async processing architecture to make GPT-4o latency acceptable in the short term.
← Week 03 Week 05 →