Deploying an AI model is not the end of the ML lifecycle — it is the beginning of the hardest part. A model in production is a living system that degrades over time, requires maintenance, and must be governed across the different regulatory environments of each market it operates in.
MLOps is the discipline of managing this lifecycle with the rigour that production software demands. This session covers the full model lifecycle — from versioning and deployment to monitoring, drift detection, and rollback — with specific attention to the multi-market governance challenges faced by AI products operating across African markets with different regulatory frameworks.
The ML model lifecycle
A machine learning model goes through a defined lifecycle, and failure to manage any stage of that lifecycle produces downstream problems. The stages are:
Development and evaluation. Model development includes data preparation, feature engineering, training, and hyperparameter tuning. Evaluation against the test set gives you a pre-deployment performance baseline. This baseline is critical: it is the reference point against which all subsequent production performance is compared. A model released without a rigorous pre-deployment evaluation has no baseline to detect degradation against.
Registry and versioning. Every model version that reaches production should be logged in a model registry — a central store of model artefacts, their evaluation metrics, training data versions, and configuration parameters. A model registry answers the question 'what is running in production right now, and what evaluation data supports that version?' without requiring archaeology of someone's laptop or a shared drive. Without a registry, rollback is guesswork.
Staged deployment. New model versions should not go directly from training to 100% of production traffic. A staged deployment process moves the model through increasing traffic exposure — shadow mode (process requests but don't serve outputs), canary (serve a small percentage of traffic), gradual rollout (increase traffic percentage over days or weeks) — with monitoring gates at each stage. A regression detected at 5% traffic costs 5% of the potential business impact of a bad deployment.
Production monitoring. The monitoring requirements for AI systems differ from traditional software. In addition to infrastructure metrics (latency, error rate, throughput), you need model performance metrics (accuracy on a sampled subset of production inferences, output distribution statistics) and data quality metrics (input feature distributions, missing data rates). A model that processes requests without errors but has degraded accuracy is not healthy — infrastructure monitoring alone will not detect it.
Retraining and iteration. Models require periodic retraining to maintain performance as input distributions shift. The retraining schedule depends on how quickly the underlying distribution changes. For credit models in African markets, an economic shock (COVID, currency devaluation, agricultural season) can shift the relevant distribution rapidly. Monitoring should inform retraining triggers, not just calendar-based retraining.
Drift detection and production evaluation
Drift is the phenomenon where a model's inputs or outputs change over time in ways that degrade performance. There are two types:
Data drift (covariate shift) occurs when the statistical distribution of input features changes relative to the training distribution. For a mobile money income extractor, data drift might occur when a new mobile money provider enters the market (statement formats change) or when economic conditions shift customer behaviour (income levels and patterns change). Data drift does not immediately cause accuracy degradation — the model may still process inputs correctly — but it is an early warning signal.
Concept drift occurs when the relationship between inputs and outputs changes, even if input distributions remain stable. For a credit model, concept drift occurs when economic conditions change the relationship between historical income patterns and future default probability. The same income profile that was predictive of safe repayment during a growth period may be less predictive during a recession.
Detecting drift requires baseline statistics: the distribution of key input features as measured on the training or initial production dataset. Compare production input distributions to this baseline on a rolling basis using statistical tests (Population Stability Index is commonly used in credit contexts; KL divergence and chi-squared tests are also common). Set an alert threshold — e.g. PSI > 0.2 indicates significant drift requiring investigation.
Production accuracy evaluation is more difficult because you need ground truth labels for production inferences, which may not be available immediately. For credit models, the repayment outcome (the label) is only available 30–90 days after the inference. This creates a monitoring lag. The approaches to managing this lag: (1) monitor proxy metrics that are available sooner (application completion rate, loan officer override rate on AI recommendations); (2) implement a randomised holdout where a fraction of AI decisions are routed to human review, generating ground truth labels sooner; (3) use human spot-review of sampled AI outputs as an interim quality signal.
SLO design for AI features
A Service Level Objective (SLO) is a target value for a service metric. SLOs are the operational contract that defines what 'working correctly' means for a production AI feature. Without SLOs, 'the model is degrading' is a subjective judgment; with SLOs, it is an objective determination.
SLO design for AI features requires going beyond the standard infrastructure SLOs (availability, latency) to define model quality SLOs that reflect the business requirements.
Accuracy SLO. What minimum accuracy is required for the feature to deliver value? For a loan pre-screening system where inaccuracy routes to manual review, an accuracy of 80% may be acceptable (the other 20% is handled by humans). For a fully automated low-value loan decision, 95% may be the minimum acceptable. Define the accuracy threshold and the evaluation method explicitly.
Latency SLO. For AI features, latency has multiple components: inference latency (time to get a model response), end-to-end processing latency (time from input received to output delivered to the user), and queueing latency (time waiting in queue under load). Each should have a separate SLO. For a USSD-integrated AI feature, end-to-end latency SLO might be P95 < 2 seconds (95% of requests complete within 2 seconds) because of USSD session constraints.
Availability SLO. What is the acceptable rate of processing failures (API errors, format violations, system outages)? For a feature that has a human fallback, 99% availability may be sufficient. For a fully automated flow, 99.9% may be required.
Coverage SLO. What percentage of inputs should the AI successfully process (as opposed to routing to fallback)? A pre-screening system that routes 50% of applications to manual review is not delivering its intended efficiency value. Set a coverage SLO: the system should process at least X% of applications fully automatically.
Document SLOs formally, review them against production metrics on a weekly or monthly cadence, and treat SLO breaches as engineering incidents requiring post-mortems, not just dashboard numbers that drift past a threshold unnoticed.
CI/CD for machine learning
Continuous integration and continuous deployment (CI/CD) practices, standard in software engineering, have analogues in ML that are essential for maintaining model quality at production scale.
ML-specific CI should include: (1) data validation on every new training dataset (schema validation, statistical checks for data drift from previous training sets, label quality checks); (2) automated training reproducibility (the same training script on the same data should produce the same model, within defined tolerance); (3) model evaluation against the standard evaluation dataset with automated comparison to the current production model's baseline; (4) performance regression gate — new model versions that perform worse than the production model by more than a defined threshold should not advance in the pipeline without a deliberate override decision.
Prompt versioning is specific to LLM-based systems. System prompts should be version-controlled exactly like code — every change to a production prompt should be committed to version control with a change description, tested against the evaluation dataset, and deployed through the same staged process as a model update. The failure mode of treating prompts as informal configuration (edited directly in a dashboard, not tracked) is silently degrading production performance when prompts are changed without testing.
A/B testing infrastructure allows you to compare a new model version or prompt against the current production version on live traffic, with randomised assignment and automated statistical analysis. For AI features where a 1% improvement in accuracy translates to material business impact (credit decisioning, fraud detection), A/B testing should be the standard deployment mechanism, not an optional enhancement.
Multi-market AI governance
Operating AI systems across multiple African markets introduces governance complexity that single-market products do not face: different data protection laws, different credit regulation frameworks, different language requirements, and different levels of regulatory maturity and scrutiny.
A multi-market AI governance framework has three components:
Market compliance matrix. A documented mapping of each operating market to its relevant AI, data protection, and credit regulation obligations, with the specific system requirements that arise from each obligation. For each requirement, define whether it is currently satisfied, in progress, or not addressed. Examples: Kenya DPA requires data subject rights including explanation of automated decisions — this requires an explainability layer in the AI system and a process for handling explanation requests. Nigeria NDPR requires consent for secondary data use — this affects whether historical loan data can be used for model training.
Model governance by market. In markets where the regulatory environment is different, the model configuration may need to be different. A model that uses a feature that creates discrimination risk in a market with anti-discrimination regulation should exclude that feature in that market even if including it improves accuracy. This requires the model deployment architecture to support per-market configuration, not just a single global model.
Audit trails and explainability. Regulators in African markets are increasingly requesting the ability to audit AI decisions in the financial sector — to understand how a credit decision was made and to verify that it complied with applicable rules. Designing audit trails from the start (logging the input features, model version, and output for every production inference, with appropriate retention policies) is significantly easier than retrofitting them. Explainability — the ability to produce a human-understandable explanation of why a specific decision was made — is both a regulatory requirement in some markets and a customer service requirement in all markets.
Self-check questions
Before moving to the next session, make sure you can answer these without re-reading. If you cannot, go back to the relevant section.
- 1What is the difference between data drift and concept drift? Give an example of each in a credit model context.
- 2Design an SLO framework for an AI-powered loan pre-screening system, covering accuracy, latency, availability, and coverage.
- 3What specific MLOps practice prevents a new model version from degrading production performance silently?
- 4A regulator in one of your operating markets requests an explanation for why a specific customer was declined by your AI model. Walk through what your system needs to have in place to answer that request.
Session deliverable
MLOps Runbook Template
Build a runbook for one AI feature — covering how it gets deployed, monitored, and rolled back. This should be specific enough that a new team member could operate the system from the runbook.
Show worked example ↓
Mobile money income extraction + pre-screening recommendation (production v2.1)
Show worked example ↓
Accuracy SLO: ≥87% agreement with loan officer validation sample (measured weekly on 50-application sample). Latency SLO: P95 end-to-end processing < 90 seconds (async); P99 < 180 seconds. Availability SLO: ≥98% of submitted applications receive a processing outcome (success or routed-to-review) within 5 minutes. Coverage SLO: ≥75% of applications processed fully automatically (not routed to manual review). SLO breaches trigger an incident ticket; two consecutive weekly SLO breaches on accuracy trigger a model review board.
Show worked example ↓
1. New model or prompt change committed to git with evaluation results attached. 2. Automated CI run: data validation, evaluation against 300-example test set, comparison to production baseline — must match or exceed baseline within 2%. 3. Deployment to staging environment; QA team runs 20-application smoke test. 4. Shadow deployment: process 500 real applications, compare outputs to production model without serving shadow outputs to users. 5. Canary: serve 10% of traffic for 48 hours. Monitor SLOs. 6. Gradual rollout: 25% → 50% → 100% with 24-hour holds at each stage. Any SLO breach at any stage triggers automatic rollback.
Show worked example ↓
Dashboards: Grafana with Prometheus metrics. Alerts: PagerDuty, on-call rotation shared between Data Engineering and QE. Metrics monitored: (1) extraction success rate (alert if < 85% in 15-min window); (2) confidence score distribution (alert if mean drops > 10% vs 7-day rolling average); (3) API latency P95 (alert if > 120s); (4) extraction error types (classification failure, format violation, OCR quality) — tracked separately. Human quality review: 50 randomly sampled applications per week reviewed by a loan officer for accuracy rating.
Show worked example ↓
PSI calculated weekly on: monthly_income_estimate, recurrence_score, income_source distribution. PSI 0.1–0.2: log and monitor. PSI > 0.2: trigger drift investigation (data team reviews input distribution changes). PSI > 0.4 or sudden spike: trigger emergency review with option to rollback. Concept drift proxy: track loan officer override rate on AI recommendations (target: < 15%); sustained override rate > 25% for 2 weeks triggers model review.
Show worked example ↓
Trigger: any P1 incident (system down), SLO breach on accuracy < 80%, or rollback requested by Head of Data or CTO. Process: (1) traffic routed to previous model version in model registry (< 5 min, automated); (2) PagerDuty alert to on-call + Head of Data + Engineering Manager; (3) incident ticket created with timestamp and trigger; (4) postmortem scheduled within 48 hours. All traffic to manual review during rollback if previous version is also affected. Target time from trigger to rollback complete: < 10 minutes.