Learn AI Series (#155) - The Complete AI Stack: Architecture Review

avatar

Learn AI Series (#155) - The Complete AI Stack: Architecture Review

variant-c-07-purple.png

What will I learn?

  • You will learn how every technique from 154 episodes connects into a coherent whole;
  • common AI architectures for common problems: what to combine with what;
  • a technology selection framework that accounts for constraints (data, compute, latency, team);
  • the "boring AI" that actually ships in production versus the flashy AI that gets conference talks;
  • what we deliberately did NOT cover in this series and where to learn it;
  • how to think architecturally about AI systems rather than thinking in individual components.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • Python 3.10+ installed;
  • Familiarity with the full series - this episode ties everything together.

Difficulty

  • Advanced

Curriculum (of the Learn AI Series):

Learn AI Series (#155) - The Complete AI Stack: Architecture Review

154 episodes behind us. Nine arcs -- math foundations, classical ML, neural networks, language models, computer vision, audio, reinforcement learning, production engineering, and the frontier stuff. That is a LOT of individual parts. And here is the uncomfortable truth about learning a field one episode at a time: you can know every part and still not know how the machine goes together. So today we are not learning anything new. Today we climb the hill, turn around, and look at the whole map at once.

I want to be clear about what this episode is not. It is not a recap -- we did plenty of those at the tail of each arc, and I am not going to make you sit through another "remember when we covered gradient descent" tour. This is about the architecture: how the pieces snap together into complete systems, and (much more importantly) how you develop the intuition to snap them together yourself when a brand-new problem lands on your desk and nobody hands you the answer.

Having said that, let's climb. Here we go!

The dependency graph

Every technique we covered stands on top of something else. Nothing in this field is an island. If you draw out what-depends-on-what, you get a structure that looks like this:

NumPy/Linear Algebra (#2, #8)
    ├── Gradient Descent (#6-7)
    │   ├── Linear/Logistic Regression (#10-12)
    │   │   └── Regularization, Feature Engineering (#11, #15)
    │   ├── Neural Networks from Scratch (#37-39)
    │   │   ├── Training Tricks (#40-41)
    │   │   ├── CNNs (#45-47) ──────────────────────────┐
    │   │   ├── RNNs/LSTMs (#48-49) ───────┐            │
    │   │   └── Attention (#51) ────────────┤            │
    │   │       └── Transformers (#52-53) ──┤            │
    │   │           ├── GPT/LLMs (#57-76) ──┤            │
    │   │           ├── BERT (#59) ─────────┤            │
    │   │           ├── ViT (#54) ──────────┼────────────┤
    │   │           └── Diffusion (#84-85)  │            │
    │   └── PyTorch (#42-44)                │            │
    │                                       │            │
    ├── scikit-learn / Classical ML (#16-20) │            │
    │   ├── Trees -> Random Forest (#17-18)  │            │
    │   ├── Gradient Boosting (#19) ────────│────────────│──┐
    │   └── Ensembles (#33) ────────────────│────────────│──┤
    │                                       │            │  │
    ├── Unsupervised (#22-27) ──────────────│────────────│──│
    │   ├── Clustering (#22-23)             │            │  │
    │   ├── PCA/t-SNE/UMAP (#24-25)        │            │  │
    │   └── Anomaly Detection (#26)         │            │  │
    │                                       │            │  │
    ├── RL (#102-116) ──────────────────────┘            │  │
    │   ├── Value Methods (#104-107)                     │  │
    │   └── Policy Methods (#108-109)                    │  │
    │                                                    │  │
    └── Production (#117-136) ──────────────────── uses all ┘
        ├── System Design (#117)
        ├── Data Eng (#118)
        ├── Optimization (#120)
        ├── Serving (#121)
        └── Monitoring (#123)

Stare at that graph for a second, because there is a genuine insight hiding in it: gradient descent and the transformer are the two most connected nodes in the entire field. Almost everything reaches back to one or both of them. If you truly understand how gradient descent optimises a parameterised function (episodes #6-7 and #10, where we built it by hand before we ever touched a library) and how a transformer moves information around with attention (#51-53), then you can read almost any new architecture as "those two ideas, plus some domain-specific way of turning the input into numbers". That is not an exaggeration for effect -- it is genuinely how the field is built. The rest is bookkeeping and good engineering.

Nota bene: notice how much of the graph funnels through the transformer row. GPT, BERT, ViT, diffusion, multimodal -- all downstream of attention. That is not an accident of drawing; that is the actual history of the last several years compressed into an ASCII diagram ;-)

Common architectures for common problems

Here is something they do not tell you at the start: in practice, most AI systems are assembled from a tiny handful of architectural patterns. You do not invent a new architecture for every problem -- you reach for one of maybe five shapes and adapt it. These five cover, in my experience, something like 90% of what actually gets deployed in the real world:

# Architecture 1: Classification / Regression Pipeline
# Episodes: #10-20, #14-16
# When: structured data, clear input->output mapping, plenty of labels
classification_pipeline = {
    "data": "Tabular features (numbers, categories)",
    "preprocessing": "Imputation, encoding, scaling (#14-16)",
    "model": "Gradient boosting (XGBoost/LightGBM) (#19)",
    "evaluation": "Cross-validation, calibration (#13)",
    "deployment": "FastAPI + batch or real-time (#121)",
    "monitoring": "Feature drift + performance drift (#123)",
    "example_uses": [
        "Fraud detection",
        "Credit scoring",
        "Churn prediction",
        "Pricing optimization",
    ],
}

# Architecture 2: Embedding + Retrieval System
# Episodes: #31, #63-65
# When: "find similar things" or "answer questions from a knowledge base"
retrieval_system = {
    "data": "Documents, products, user profiles",
    "preprocessing": "Chunking, cleaning (#64)",
    "embedding_model": "Sentence transformer or OpenAI embeddings (#63)",
    "vector_store": "FAISS, Chroma, Qdrant (#63)",
    "retrieval": "Approximate nearest neighbor search",
    "optional_reranker": "Cross-encoder for precision (#65)",
    "optional_generator": "LLM for RAG (#64-65)",
    "example_uses": [
        "Semantic search",
        "Document Q&A (RAG)",
        "Recommendation systems",
        "Duplicate detection",
    ],
}

# Architecture 3: Foundation Model + Prompt/Fine-tune
# Episodes: #57-62, #66, #69
# When: any NLP task in 2026
llm_system = {
    "data": "Text (any format)",
    "approach_1_prompt": "API call with system prompt + examples (#62, #66)",
    "approach_2_finetune": "LoRA on domain data (#69)",
    "approach_3_rag": "Retrieve context + generate (#64-65)",
    "structured_output": "JSON mode / function calling (#66)",
    "evaluation": "LLM-as-judge + human eval (#73)",
    "example_uses": [
        "Content generation",
        "Summarization",
        "Classification",
        "Extraction / parsing",
        "Code generation",
        "Agents and tool use",
    ],
}

# Architecture 4: Pretrained Vision + Fine-tune
# Episodes: #45-47, #54, #78-80
# When: any image/video task
vision_system = {
    "data": "Images or video frames",
    "backbone": "ResNet, EfficientNet, ViT (#46, #54)",
    "task_head": {
        "classification": "Linear layer (#46)",
        "detection": "YOLOv8 (#79)",
        "segmentation": "SAM or U-Net (#80)",
    },
    "training": "Fine-tune from pretrained weights (#46)",
    "augmentation": "Random crop, flip, color jitter (#43)",
    "example_uses": [
        "Quality inspection",
        "Medical imaging",
        "Content moderation",
        "Autonomous systems",
    ],
}

# Architecture 5: Multimodal Pipeline
# Episodes: #75, #100, #138
# When: input or output spans multiple modalities
multimodal_system = {
    "encoders": {
        "text": "Transformer encoder (#52, #59)",
        "image": "ViT or CNN (#54, #46)",
        "audio": "Whisper encoder or spectrogram CNN (#93)",
    },
    "fusion": "Cross-attention or projection to shared space (#138)",
    "decoder": "Autoregressive transformer (#53, #58)",
    "example_uses": [
        "Visual question answering",
        "Image captioning",
        "Video understanding",
        "Audio-visual analysis",
    ],
}

Notice the pattern that runs through all five: every single one starts from a pretrained model. Training from scratch, in 2026, is the exception -- reserved for the labs with the GPU farms and the research budgets. For the rest of us the real skill is not building architectures at all. It is knowing which pretrained pieces exist, which ones to combine, and how to adapt them to your problem with the least amount of custom work. That is a very different skill from what most courses teach, and it is the one that pays the bills.

The technology selection framework

Alright, so a real problem lands on your desk. A stakeholder wants "AI" for something. You have five architectural shapes in your head and a hundred and fifty episodes of techniques. How do you decide? The mistake almost everyone makes -- and I made it too, for years -- is to start from the technology ("I want to use a transformer") and then go looking for a problem shaped like the tool. Wrong direction. You work backwards, from the constraints. Here is the framework I actually use:

def select_technology(problem):
    """
    Constraint-aware technology selection.
    Work BACKWARDS from constraints, not forwards from technology.
    """
    constraints = {
        # Data constraints
        "data_size": None,        # samples: 100? 10K? 1M?
        "data_labels": None,      # labeled, partially, unlabeled
        "data_modality": None,    # tabular, text, image, audio, mixed
        "data_sensitivity": None, # public, internal, regulated

        # Compute constraints
        "training_budget": None,  # laptop, single GPU, cluster, cloud
        "inference_latency": None,# <10ms, <100ms, <1s, >1s OK
        "inference_cost": None,   # per prediction budget

        # Team constraints
        "ml_expertise": None,     # beginner, intermediate, expert
        "maintenance_budget": None,# can you retrain monthly? quarterly?

        # Business constraints
        "accuracy_requirement": None,  # 80%? 95%? 99.9%?
        "explainability": None,   # nice-to-have, required, legally mandated
        "time_to_deploy": None,   # days, weeks, months
    }

    # The constraint that matters MOST determines your approach
    critical_constraint_rules = {
        "regulated_data": "Can't use external APIs. Local models or on-prem (#70, #128)",
        "sub_10ms_latency": "No LLM APIs. Optimized local model or classical ML (#120)",
        "no_labels": "Unsupervised (#22-26), self-supervised (#90), or LLM zero-shot (#144)",
        "tiny_dataset": "Transfer learning (#46), few-shot (#144), or feature engineering + small model",
        "must_explain": "Linear models (#10-12), trees (#17), SHAP (#146) - avoid black boxes",
        "deploy_in_days": "Use existing APIs and pretrained models only (#66, #74)",
        "99.9_accuracy": "Ensemble methods (#33), human-in-the-loop, fallback chains",
    }

    return constraints, critical_constraint_rules

Read that critical_constraint_rules dict slowly, because it is the whole game in one place. The single most common mistake in this field is choosing a technology because it is exciting rather than because the constraints demand it. If your data is tabular, your team is three people, and you need to ship in two weeks -- the answer is gradient boosting (episode #19), full stop. Not a transformer. If your data is text and you need good-enough accuracy fast -- the answer is an LLM API call (#66), not three months training a custom model. Match the tool to the constraints, not to your ambition. Your ambition is not the customer's problem ;-)

Let me make that concrete, because a framework in the abstract is easy to nod along to and hard to actually apply. Suppose a real request comes in: "we get thousands of support emails a day, route each one to the right team." Watch the constraints do the deciding for you:

# Turn a vague request into a stack by answering the constraint questions.
problem = {
    "data_modality": "text",          # support emails
    "data_labels": "partial",         # a few thousand historical, hand-tagged
    "data_size": "3000 labeled",      # small-ish
    "inference_latency": "<1s OK",    # emails are not real-time
    "data_sensitivity": "internal",   # customer PII -> be careful with APIs
    "ml_expertise": "intermediate",
    "time_to_deploy": "3 weeks",
}

def decide_stack(p):
    # Constraint 1: sensitive data narrows the field before anything else.
    if p["data_sensitivity"] in ("regulated", "internal"):
        base = "local model or a vendor with a data-processing agreement (#70, #128)"
    else:
        base = "any hosted LLM API is fair game (#66)"

    # Constraint 2: tiny labeled set -> do NOT train from scratch.
    if "3000" in p["data_size"]:
        approach = "zero/few-shot LLM classification first (#144), fine-tune only if it underperforms (#69)"
    else:
        approach = "fine-tune a small encoder like DistilBERT (#59, #69)"

    # Constraint 3: relaxed latency means we are not forced into a tiny model.
    latency_note = "sub-second budget is generous -- no need to over-optimize (#120)"
    return base, approach, latency_note

for line in decide_stack(problem):
    print("-", line)

See what happened? At no point did I ask "what is the coolest model". I answered a handful of boring questions and the stack fell out almost on its own. The email router is a few-shot LLM classifier with a fallback, running against a vendor that will sign a data agreement, and I have not written a line of training code yet. THAT is architectural discipline -- and it is way less glamorous than it sounds, which is exactly why it works.

The boring AI that actually ships

Now for the part nobody puts on a conference slide. After quite some years around production systems, here is an observation that took me an embarrassingly long time to accept: the AI that runs most of the world is boring. Not transformers. Not diffusion models. Not reinforcement learning with a clever reward. It is the stuff from the FIRST arc of this very series:

  • Logistic regression for click-through rate prediction (trillions of predictions per day across the internet)
  • Gradient-boosted trees for fraud detection, credit scoring, recommendation ranking
  • TF-IDF + simple classifiers for spam filtering, content categorization
  • k-nearest neighbors for recommendation systems at moderate scale
  • Moving averages and threshold rules for anomaly detection in monitoring systems

All of that came from episodes #10-20. The first arc. Before we ever touched a neural network. These techniques work, they are fast, they are explainable (a regulator can read them), they are debuggable at 3am, and they do not need a GPU cluster humming in a datacenter. A logistic regression that scores in microseconds and that you can reason about beats a transformer you cannot debug -- almost every time the stakes are real.

Here is why the boring stuff wins on the numbers, made painfully concrete:

# Two ways to score a click-through-rate prediction. Same job, wildly
# different cost. This is why "boring" runs the ad-serving internet.
def boring_ctr(features, weights, bias):
    """Logistic regression. Microseconds. Runs on a potato."""
    import math
    z = sum(f * w for f, w in zip(features, weights)) + bias
    return 1.0 / (1.0 + math.exp(-z))          # one dot product + a sigmoid

# A big neural model would need a matrix library, a loaded checkpoint,
# probably a GPU, and tens of milliseconds per call. At a MILLION requests
# per second (a normal ad exchange), those milliseconds are the whole budget.
example = boring_ctr([0.3, -1.2, 0.8], [1.5, 0.4, -0.9], bias=0.1)
print(round(example, 4))   # a probability, computed in the time it took to read this

That is not a toy -- that is (a stripped-down version of) what actually predicts trillions of ad clicks a day across the internet. Speed and explainability, not raw capability, are what put a model into the hot path.

The exciting stuff -- LLMs, diffusion, multimodal systems -- is increasingly deployed too, do not get me wrong. But it is deployed alongside the boring AI, not instead of it. A production recommendation system might use an LLM to understand messy product descriptions, gradient boosting to rank the candidates, and a dumb rule-based filter to enforce the business constraints that legal insists on. The system is a composition, not a single hero model:

class ProductionRecommender:
    """Real recommender systems are compositions, not single models."""

    def recommend(self, user, context, n_results=10):
        # Stage 1: Candidate generation (fast, broad)
        # Simple embedding similarity or collaborative filtering
        candidates = self.candidate_generator.get_candidates(
            user, n=1000
        )

        # Stage 2: Business rules (no ML needed)
        # Filter out-of-stock, age-restricted, already-purchased
        candidates = self.business_filter.apply(candidates, user, context)

        # Stage 3: Ranking (the ML part)
        # Gradient boosting on features: user history, item features,
        # context (time, device, location)
        features = self.feature_builder.build(user, candidates, context)
        scores = self.ranker.predict(features)

        # Stage 4: Diversification (rule-based)
        # Don't show 10 items from the same category
        results = self.diversifier.select(candidates, scores, n=n_results)

        # Stage 5: Explanation (optional, for transparency)
        explanations = self.explainer.explain(user, results)

        return results, explanations

Count them: four of the five stages are not machine learning at all. Candidate generation, business rules, diversification, explanation -- plain code, filters, and heuristics. The actual ML (stage 3) is a gradient-boosted tree that scores in about 2ms, and the whole pipeline lands around 50ms end to end. It works. It is boring. It ships. And if you had walked into that room proposing to replace all five stages with one giant end-to-end model, you would have been (correctly) shown the door.

What we didn't cover

Let's be honest with each other for a minute. This has been a long road -- more episodes than either of us probably want to count -- but the field is genuinely vast, and no series covers all of it. Pretending otherwise would be an insult to your intelligence. So here are the honest gaps, the things I either skipped or barely grazed:

not_covered = {
    "in_depth": [
        "Bayesian deep learning (beyond episode #32's basics)",
        "Neural ODEs and continuous-time models",
        "Geometric deep learning beyond basic GNNs (#131)",
        "Diffusion for non-image modalities (protein design, molecular gen)",
        "Compiler-level ML optimization (TVM, Apache TVM, Triton kernels)",
        "Hardware-specific optimization (CUDA kernel writing beyond #125)",
        "Federated learning at scale (only covered concepts in #128)",
    ],
    "entire_subfields": [
        "Computational biology / bioinformatics",
        "Computational linguistics (formal syntax/semantics)",
        "Signal processing (beyond audio fundamentals)",
        "Operations research / mathematical optimization",
        "Probabilistic programming (Pyro, Stan, NumPyro)",
    ],
    "practical_skills": [
        "ML interviewing and system design interviews",
        "Managing ML teams and projects (touched in #135)",
        "ML regulation compliance in depth (GDPR, EU AI Act specifics)",
        "Cost optimization for large-scale ML pipelines",
    ],
}

Here is the thing though, and it is the reason I am not the least bit worried about those gaps: for every one of them, this series already handed you the foundation to go learn it on your own. You know how neural networks train (#37-44), so Neural ODEs are an extension of a thing you understand, not an alien concept. You know transformers cold (#52-53), so any shiny new architecture built on attention is readable to you now. You know how to read the paper behind it (that was the previous episode). The foundation was always the point. I was never going to teach you all of AI -- nobody can. I was teaching you how to teach yourself the rest.

Thinking architecturally

So what was the actual meta-skill this whole series was secretly building? It is architectural thinking: looking at a fresh problem and seeing which pieces from your toolkit snap together into a solution. Not "I'll use a transformer because transformers are powerful" (the beginner's reflex), but "this problem has sequential dependencies, variable-length inputs, and needs to attend to long-range context -- so attention is the right inductive bias here, and a pretrained encoder gives me a running start." That shift, from thinking in models to thinking in systems, is the difference between someone who took a course and someone you can hand a vague problem to and actually trust with it.

class ArchitecturalThinking:
    """The questions that guide system design."""

    questions = [
        # Input
        "What is the raw input? (text, images, numbers, mixed)",
        "How much of it do I have?",
        "How noisy / messy is it?",
        "Does the input have structure I should exploit? (sequential, spatial, graph)",

        # Output
        "What am I predicting? (class, number, sequence, image, action)",
        "How precise does it need to be?",
        "Does the output need to be explainable?",

        # Constraints
        "What's the latency budget?",
        "What's the compute budget (training and inference)?",
        "Are there privacy / regulatory constraints on the data?",
        "How often does the data distribution change?",

        # System
        "Is this a one-shot prediction or part of a loop?",
        "Does the system need to improve over time?",
        "What happens when the model is wrong?",
        "Who maintains this after I'm done?",
    ]

    @staticmethod
    def design_principle():
        return (
            "Start with the simplest approach that could work. "
            "Add complexity only when you have evidence that simplicity fails. "
            "Every component you add is a component that can break."
        )

That design_principle is not a throwaway docstring -- it is the single thread running through this ENTIRE series, from the very first episode to this one. We started with the simplest possible prediction (just the average, all the way back in episode #4) and added complexity only when the problem forced our hand. Linear regression when we needed continuous outputs. Logistic regression when we needed classes. Neural networks when we needed to learn features automatically in stead of hand-crafting them. Transformers when we finally needed variable-length sequences with long-range dependencies. Every single step was earned by evidence that the previous step was not enough.

That is the discipline in practice -- and it is worth having as a literal checklist you run before adding anything to a system:

def should_i_add_complexity(current_system, proposed_addition):
    """Run this BEFORE reaching for the fancier model. Be honest."""
    # Do you have EVIDENCE the simple version fails? Not a hunch -- evidence.
    if not current_system["has_measured_failure"]:
        return "NO -- measure the simple baseline first (#13). You are guessing."
    # Will the addition fix the SPECIFIC failure you measured?
    if proposed_addition["targets"] != current_system["measured_failure"]:
        return "NO -- you are solving a problem you do not actually have."
    # Every part you add is a part that can break at 3am. Worth it?
    cost = proposed_addition["extra_latency_ms"] + proposed_addition["extra_ops_burden"]
    if cost > proposed_addition["expected_gain"]:
        return "NO -- the complexity costs more than it buys."
    return "YES -- earned. Add it, and measure again."

print(should_i_add_complexity(
    {"has_measured_failure": False, "measured_failure": None},
    {"targets": "accuracy", "extra_latency_ms": 40, "extra_ops_burden": 5, "expected_gain": 3},
))  # -> "NO -- measure the simple baseline first (#13). You are guessing."

Each step earned, not assumed. THAT is architectural thinking, boiled down to a habit you can actually run in your head.

And that, my friend, is very nearly the whole journey. We have one more stop on this road together -- a quieter, more personal one about where you go from here. But the map is complete. You are holding it now.

What to remember from this one

  • The dependency graph of AI techniques centers on two nodes: gradient descent and transformers - deep understanding of both lets you learn almost everything else;
  • five architectural patterns cover 90% of real-world AI systems: classification pipeline, embedding + retrieval, LLM + prompt/fine-tune, pretrained vision + task head, and multimodal pipeline;
  • select technology by working backwards from constraints (data, compute, latency, team, explainability), not forwards from what's exciting;
  • the boring AI (logistic regression, gradient boosting, TF-IDF) runs most of the world's production systems and is deployed alongside, not replaced by, frontier models;
  • production systems are compositions: multiple stages, some ML, some rules, some filtering - the ML model is rarely more than one component in a larger pipeline;
  • architectural thinking is the meta-skill: asking the right questions about input, output, constraints, and system context, then snapping together the right pieces from your toolkit.

155 episodes in and you are still here -- that says everything about you and very little about me. Go build something boring that ships ;-) De groeten!

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments