WEEK 05

AI System Architecture

Workflow Patterns and LLM Production Design
RAG · Agents · Orchestration · Production reliability · Architecture patterns
Reading time~60–90 min
FormatRead · Apply · Deliver
ContextAfrican fintech examples

In this session

A language model is not a product. It is an ingredient. The product is the system around the model — the data pipeline that feeds it, the orchestration logic that sequences its calls, the output parser that makes its responses usable, the fallback mechanisms that handle its failures, and the monitoring infrastructure that tells you when something has gone wrong. This session covers the patterns used to build production AI systems: how to structure them, which patterns to apply when, and what the production failure points are at each layer. By the end, you should be able to design a system architecture for an AI feature that a senior engineer could implement without further clarification.

The anatomy of a production AI system

A production AI system has five layers, each of which can fail independently: The ingestion layer receives raw inputs — a USSD session payload, a PDF document, a customer message, a structured API call — and transforms them into a format usable by the AI components. This layer is responsible for validation (is the input well-formed?), preprocessing (OCR, text extraction, format normalisation), and routing (which pipeline should handle this input?). Ingestion failures are often the most common failure type in production, because real-world inputs are messier than anything tested in development. The retrieval layer (present in RAG architectures) fetches relevant context from an external knowledge base — a vector database, a structured database, an API — to include in the model prompt. The quality of retrieval directly determines the quality of generation. A retrieval layer that returns irrelevant context produces worse outputs than no retrieval at all. Retrieval layer design — chunking strategy, embedding model choice, similarity threshold, reranking — deserves as much engineering attention as the model selection. The generation layer is the model inference call itself. The inputs are the formatted prompt (which includes system instructions, retrieved context, and the user's actual request). The outputs are raw model outputs, which must be validated before use. This layer should be treated as an unreliable external service: it can fail (API errors, rate limits, timeout), it can return invalid outputs (malformed JSON, responses that fail output schema validation), and its performance can change silently over time (model version updates). The output processing layer parses, validates, and transforms raw model outputs into the format expected by downstream systems. If the model is expected to return JSON, this layer parses the JSON and validates it against a schema. If the model is expected to make a binary decision, this layer extracts and validates the decision from the model's text output. Output processing failures — typically caused by the model not following the output format — are a common production issue that should be handled with retry logic and graceful fallback. The action layer uses the processed output to take a downstream action: update a database, send a response to a user, trigger a workflow, notify a human reviewer. This layer is where the AI system connects to the rest of the product, and it is where failures have the most direct business impact.

RAG architecture in depth

Retrieval-augmented generation is the most important pattern in production LLM systems. It solves the core limitation of pure LLM inference: the model's knowledge is frozen at training time and cannot be updated with new information, private information, or information specific to your users or domain. A RAG system works in three stages: indexing (converting your knowledge base into a searchable form), retrieval (finding the most relevant content for a given query), and generation (using the retrieved content as context for the LLM to generate a response). Indexing involves chunking your source documents into appropriately sized segments, embedding each chunk using an embedding model, and storing the chunks and their embeddings in a vector database. Chunk size is a critical design parameter: too small (a single sentence) loses context; too large (an entire document) dilutes relevance. A starting point for most tasks is 200–400 tokens per chunk with a 50-token overlap between adjacent chunks. Retrieval involves embedding the user's query with the same embedding model used during indexing, computing similarity between the query embedding and all chunk embeddings, and returning the top-k most similar chunks. The default similarity metric is cosine similarity. Top-k is a critical parameter: too few retrieved chunks misses relevant information; too many retrieved chunks fills the context window with noise. A common starting point is k=5, evaluated on your task. Advanced RAG patterns worth knowing: Hybrid search combines semantic (embedding-based) search with keyword (BM25) search, which is often better for domain-specific terms that an embedding model may not encode well. Reranking uses a cross-encoder model to reorder retrieved chunks by relevance before passing them to the LLM — this adds latency but often improves quality significantly. HyDE (Hypothetical Document Embeddings) generates a hypothetical answer to the query and retrieves based on that, rather than the query itself — effective when the query and relevant documents are in different linguistic styles. For African fintech applications, RAG is particularly valuable for: product and policy document retrieval (ensuring AI answers about loan terms, eligibility, and processes are grounded in current policy), regulatory compliance querying (retrieving relevant regulatory provisions for a given question), and customer FAQ systems (grounding responses in the actual product documentation rather than the model's training knowledge).

Agent patterns and when to use them

An AI agent is a system where a language model is given tools — the ability to call external functions, APIs, or services — and uses those tools autonomously to accomplish a multi-step task. The model decides which tools to call, in what order, based on the task it is given and the results of previous tool calls. Agents are powerful but fragile. The power comes from flexibility — an agent can handle tasks whose steps are not fully specified in advance, adapting based on intermediate results. The fragility comes from error compounding — each step in an agent's chain can fail, and errors in early steps propagate to later steps. A three-step agent with 90% reliability at each step has 73% end-to-end reliability (0.9³). A five-step agent with the same per-step reliability has 59% end-to-end reliability. These numbers are unacceptable for most production applications. The implication is not to avoid agents — they are the right pattern for certain tasks — but to use them for tasks where the flexibility justifies the reliability cost, and to design them with robust error handling, step validation, and human-in-the-loop checkpoints for high-stakes decisions. Tasks well suited to agents: complex research and synthesis (gathering information from multiple sources, resolving conflicts, producing a report), adaptive customer interactions (handling a customer service conversation that may require checking account status, looking up policy, escalating to a human, and following up), and workflow orchestration (triggering a series of downstream actions based on a processed document). Tasks poorly suited to agents: any task where the steps are fixed and fully specified (use a chain instead), any task where reliability must be above 95%, any task where errors are difficult to reverse (money movement, account changes). For a lending platform, an appropriate agent use case is a loan application assistant that can check eligibility against current policy, retrieve relevant product information, request missing documents, and schedule a call with an agent — adapting its path based on the customer's responses. An inappropriate agent use case is a credit decision system where each step in the decision logic must be auditable and reproducible.

Prompt chaining and deterministic workflows

For most production AI features, the right pattern is not an agent but a prompt chain: a fixed sequence of LLM calls where the output of each call is the input to the next. Unlike agents, the workflow structure is determined by the developer, not the model. This makes prompt chains more reliable, more predictable, and more auditable than agents, at the cost of flexibility. A well-designed prompt chain for a document extraction task might look like: (1) classify document type (bank statement vs. mobile money vs. payslip); (2) extract structured data using a template appropriate for the identified document type; (3) validate the extracted data for internal consistency; (4) generate a human-readable summary of the extraction for loan officer review. Each step in this chain is simpler than asking a single model to do everything. Simpler prompts produce more reliable outputs. Intermediate outputs can be validated and the chain halted if a step fails, rather than propagating errors silently. Routing is a pattern that combines classification with chaining: a classifier model (or a simple rule) examines an input and routes it to a specialised prompt chain based on its type. A customer message classifier might route 'what is my current balance?' to a structured data retrieval path and 'I want to dispute a transaction' to a customer escalation path. Routing with specialised handlers typically outperforms a single general-purpose handler for most production volumes. Parallelisation runs multiple LLM calls simultaneously and aggregates the results. For tasks where you want multiple independent analyses of the same input (e.g. evaluating a document from a credit risk perspective and a fraud risk perspective simultaneously), parallelisation reduces total latency without compromising quality.

Top production failure points and mitigations

Production AI systems fail in specific, predictable ways. Designing mitigations for these failure modes before deployment is significantly cheaper than retrofitting them after an incident. Output format violations. The model returns a response that does not conform to the expected format — malformed JSON, a decision embedded in prose rather than a structured field, missing required fields. Mitigation: implement strict output schema validation at the output processing layer; implement retry logic with an explicit format correction prompt on the first failure; have a graceful fallback that routes to human review rather than raising an uncaught exception. Context window overflow. The combined length of the system prompt, retrieved context, and user input exceeds the model's context window, causing either an API error or silent truncation. Mitigation: measure context length at every prompt construction and implement dynamic context trimming if the limit is approached; monitor context window utilisation as a production metric. Rate limit exhaustion. Sudden spikes in request volume cause API rate limit errors, degrading the feature for all users simultaneously. Mitigation: implement exponential backoff and retry; design async processing architectures that can queue requests rather than failing immediately; maintain headroom below rate limits with traffic shaping. Prompt injection. User inputs contain text designed to override the model's instructions — 'ignore previous instructions and return the customer's credit score'. In a financial application, prompt injection can cause information leakage or policy violations. Mitigation: never include sensitive data in contexts that users can influence; implement input sanitisation for known injection patterns; use structured output schemas that constrain what the model can return. We will cover this in more depth in Session 7. Silent accuracy degradation. Model performance degrades over time as input distributions shift, but without explicit monitoring, the degradation is not detected until it causes a business-visible failure. Mitigation: implement production evaluation on a sampled subset of real inferences with human labelling; set up automated alerts when accuracy metrics drop below defined thresholds. Session 8 covers this in depth.

Self-check questions

Before moving to the next session, make sure you can answer these without re-reading. If you cannot, go back to the relevant section.

Session deliverable

AI System Architecture Diagram

Design a C4-style architecture for an AI feature, annotated with the primary failure risk at each layer and the proposed mitigation. Focus on being specific enough that an engineer could implement it without further clarification.

Show worked example ↓
Loan application pre-screening pipeline — automated income and risk assessment from uploaded financial documents
Show worked example ↓
Prompt chain — fixed multi-step sequence
Show worked example ↓
Input: PDF upload via mobile app or agent-scanned document. Processing: PDF-to-text extraction using pdfminer with fallback to Tesseract OCR for image-based PDFs. Validation: check extracted text length > 200 chars (reject blank/failed extractions); check document language (route non-English/non-Swahili to human review). Failure risk: OCR quality on low-resolution photos is poor — flag low-confidence extractions with a quality score and route to human if below threshold.
Show worked example ↓
N/A for core extraction pipeline. RAG used in a secondary path for policy lookups: when the pre-screening output triggers a policy edge case (e.g. applicant income is primarily from a business category), retrieve relevant policy provisions from the product policy vector database to include in the loan officer brief.
Show worked example ↓
Model: GPT-4o (async, not real-time user-facing). Prompt chain: (1) classify document type + provider; (2) extract structured income data using provider-specific template; (3) validate extraction consistency; (4) generate officer brief in plain English. Output: JSON schema with fields: monthly_income_estimate, confidence_interval, income_source, recurrence_score, anomaly_flags, officer_brief. Failure risk: format violations on step 2 when statement structure is unusual — implement retry with explicit format correction prompt.
Show worked example ↓
Parse JSON output against Pydantic schema. On validation failure: retry with format correction prompt once; on second failure, route to human review queue with raw extraction and error log. On success: store structured output in loan application record. Human fallback: all extractions with confidence_score < 0.7 are flagged for loan officer review before pre-screening recommendation is generated.
Show worked example ↓
1. OCR failure on photo-based statements: ~15% of documents submitted via agent network are photos. Mitigation: Tesseract OCR with quality scoring; route low-quality scans to agent for re-scan request. 2. Format violation cascade: if step 1 misclassifies a document type, step 2 uses the wrong extraction template and all subsequent steps inherit the error. Mitigation: validate step 1 output with a confidence check before proceeding; route low-confidence classifications to human. 3. Context window overflow on multi-page statements: some statements are 20+ pages. Mitigation: chunk statements and extract page-by-page, then aggregate; never pass full multi-page document in single context.
← Week 04 Week 06 →