Learn AI Series (#153) - Building AI Products

avatar

Learn AI Series (#153) - Building AI Products

variant-a-05-hotpink.png

What will I learn

  • the gap between a working model and a product -- why a model that aces the test set can still ship a product nobody comes back to;
  • designing UX for uncertainty -- how to tell a user "I might be wrong" without destroying their trust, and why calibration (#13) is a product problem, not just a metric;
  • the data flywheel -- turning every click, thumbs-up and correction into training signal, so the product quietly gets better while you sleep;
  • the money side -- why AI products have a per-prediction cost that traditional software does not, and what that does to pricing;
  • build vs buy -- the honest framework for "train our own model" versus "call an API", and why the answer is almost always the same;
  • what actually works -- patterns I have watched succeed and fail, with the boring truth about why.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • Python 3.10+ installed;
  • You have read episode #117 (ML System Design) and #151 (Building Something That Matters) -- this one leans on both.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#153) - Building AI Products

There is a pattern I have watched play out more times than I can count. Someone builds a model. The model scores beautifully on the test set. They wrap it in an API, slap a UI on top, and show it to real users. The users try it three times and never come back. The model was fine. The product was broken ;-)

Building an AI product is not the same thing as building a model and bolting an interface onto it. The model is maybe 20% of the work. The other 80% is understanding what the user actually needs (not what you assume they need), designing for a system that WILL be wrong sometimes, and building the machinery that makes the whole thing improve over time. Last episode we packed our toolkit -- the stuff you reach for to build models. Today we deal with everything that stands between a good model and a thing people pay for. This is the 80% nobody puts on a slide.

The model-to-product gap

The distance between "the model works" and "the product works" shows up in a few painfully predictable places. Let me name them, because once you can name them you start seeing them everywhere.

Latency. Your model takes 3 seconds per prediction. In a notebook, who cares -- you go grab coffee. In a product, where a human is sitting there watching a spinner, 3 seconds feels broken. Users expect sub-second responses for most interactions, and their patience for "AI magic" is far shorter than the hype suggests. If your model is slow you need caching (#121), async processing with an honest progress indicator, or simply a smaller model that is fast enough. Fast-and-good-enough beats slow-and-perfect almost every time.

Error handling. In a notebook, an error means you re-run the cell and mutter to yourself. In a product, an error means a user stares at a blank screen and quietly loses faith. Every single prediction path needs a fallback. The user must NEVER see a raw stack trace. Here is the shape I reach for -- a predictor that degrades gracefully in stead of crashing:

class RobustPredictor:
    """Prediction with graceful degradation: cache -> primary -> fallback -> default."""
    def __init__(self, primary_model, fallback_model=None):
        self.primary = primary_model
        self.fallback = fallback_model
        self.cache = {}

    def predict(self, input_data, timeout_ms=1000):
        # Level 1: cache hit - free and instant
        cache_key = self._hash_input(input_data)
        if cache_key in self.cache:
            return {'prediction': self.cache[cache_key],
                    'source': 'cache', 'confidence': None}

        # Level 2: the real model
        try:
            result = self._timed_predict(self.primary, input_data, timeout_ms)
            self.cache[cache_key] = result['prediction']
            return {**result, 'source': 'primary'}
        except Exception:
            pass  # fall through, never surface the raw error

        # Level 3: a simpler, faster backup model
        if self.fallback:
            try:
                result = self._timed_predict(self.fallback, input_data, timeout_ms * 2)
                return {**result, 'source': 'fallback'}
            except Exception:
                pass

        # Level 4: a sane default - always better than a 500 page
        return {'prediction': self._default_prediction(input_data),
                'source': 'default', 'confidence': 0.0}

    def _timed_predict(self, model, input_data, timeout_ms):
        import time
        start = time.monotonic()
        prediction = model.predict(input_data)
        elapsed_ms = (time.monotonic() - start) * 1000
        if elapsed_ms > timeout_ms:
            raise TimeoutError(f"Prediction took {elapsed_ms:.0f}ms")
        return {'prediction': prediction, 'latency_ms': elapsed_ms}

    def _hash_input(self, data):
        import hashlib, json
        return hashlib.md5(json.dumps(data, sort_keys=True).encode()).hexdigest()

    def _default_prediction(self, input_data):
        return None  # a safe, boring default beats a crash

Notice the ladder: cache, then the good model, then a cheap backup, then a sensible default. At every rung the user gets something -- a result, a degraded result with a note, or a clear "we could not process this and here is why". What they never get is a blank page. That laddered fallback is the difference between a product that feels solid and one that feels like a science experiment.

Designing UX for uncertainty

Here is the hardest thing about AI products, and the one engineers underestimate the most. AI predictions are probabilistic. Sometimes they are wrong. And you are shipping them to users who have spent thirty years being trained by deterministic software that either works or throws an error you can Google. Managing that mismatch is a design problem, not a modelling one.

Two rules I hold to. First, do not hide uncertainty. If your model is 60% confident, do not dress the answer up as fact. "This might be X (moderate confidence)" is both more honest and, weirdly, more useful than a flat "This is X" that turns out wrong. Users forgive a system that admits doubt. They do not forgive one that was confidently wrong.

Second, calibrate your confidence scores. A model that says "90% sure" and is actually right 70% of the time is worse than useless -- it is actively lying to your users with a straight face. We covered calibration back in #13; in product terms it becomes non-negotiable. If your confidence numbers do not map to real accuracy, fix them before you ship a single screen. Platt scaling is the simple, boring workhorse here:

import numpy as np

class ConfidenceCalibrator:
    """Map raw model scores to honest probabilities via Platt scaling."""
    def __init__(self):
        self.a, self.b = 1.0, 0.0

    def fit(self, raw_scores, true_labels):
        from scipy.optimize import minimize
        def nll(params):
            a, b = params
            p = 1 / (1 + np.exp(-(a * raw_scores + b)))
            p = np.clip(p, 1e-7, 1 - 1e-7)
            return -np.mean(true_labels * np.log(p)
                            + (1 - true_labels) * np.log(1 - p))
        self.a, self.b = minimize(nll, [1.0, 0.0], method='Nelder-Mead').x

    def calibrate(self, raw_score):
        return 1 / (1 + np.exp(-(self.a * raw_score + self.b)))

Calibrated numbers are only half the job though. A user does not want to read "0.43". They want words. So the last mile is translating a probability into plain human language -- something they can actually act on:

class UncertaintyPresenter:
    """Turn a calibrated confidence into language a human understands."""
    LEVELS = [
        (0.9, "high confidence",     "We're fairly sure about this"),
        (0.7, "moderate confidence", "This looks likely, but we're not certain"),
        (0.5, "low confidence",      "Best guess -- take it with a grain of salt"),
        (0.0, "very low confidence", "We're really not sure -- please verify this one"),
    ]

    def present(self, prediction, confidence):
        for threshold, level, message in self.LEVELS:
            if confidence >= threshold:
                return {'prediction': prediction,
                        'confidence_level': level,
                        'user_message': message,
                        'raw_confidence': round(confidence, 3)}
        return {'prediction': prediction, 'confidence_level': "very low confidence",
                'user_message': self.LEVELS[-1][2], 'raw_confidence': round(confidence, 3)}

The UncertaintyPresenter is almost embarrassingly simple, and yet it moves the needle on trust more than another two points of accuracy would. "We're really not sure about this one" is something a person can work with. "confidence: 0.43" is something a person ignores or misreads. Having said that, the words only earn their keep if the number behind them is calibrated -- honest language wrapped around a lying score is just a nicer lie.

The data flywheel

Now the single most powerful idea in the whole business of AI products. Every user interaction can improve the model. That loop -- predict, observe what the user does, feed it back, retrain -- is the data flywheel, and it is precisely what separates products that quietly compound over the years from products that freeze the day they launch.

class DataFlywheel:
    """Collect feedback from live usage so the model gets better over time.

    The cycle: user gets a prediction -> user reacts (accept/reject/correct)
    -> reaction becomes a training signal -> model improves -> better
    predictions pull in more users -> more data -> and around we go.
    """
    def __init__(self, feedback_path='./feedback'):
        from pathlib import Path
        self.path = Path(feedback_path)
        self.path.mkdir(parents=True, exist_ok=True)

    def log_prediction(self, request_id, input_data, prediction, confidence):
        import json
        from datetime import datetime
        record = {'request_id': request_id,
                  'timestamp': datetime.now().isoformat(),
                  'input': input_data, 'prediction': prediction,
                  'confidence': confidence, 'feedback': None}
        with open(self.path / 'predictions.jsonl', 'a') as f:
            f.write(json.dumps(record) + '\n')

    def log_feedback(self, request_id, feedback_type, correction=None):
        # feedback_type: 'accepted' | 'rejected' | 'corrected'
        import json
        from datetime import datetime
        record = {'request_id': request_id,
                  'timestamp': datetime.now().isoformat(),
                  'feedback_type': feedback_type, 'correction': correction}
        with open(self.path / 'feedback.jsonl', 'a') as f:
            f.write(json.dumps(record) + '\n')

    def get_training_candidates(self):
        """The corrected, low-confidence cases are the gold - exactly where
        the model struggled and a human told us the right answer."""
        import json
        preds, fb = {}, {}
        with open(self.path / 'predictions.jsonl') as f:
            for line in f:
                r = json.loads(line); preds[r['request_id']] = r
        with open(self.path / 'feedback.jsonl') as f:
            for line in f:
                r = json.loads(line); fb[r['request_id']] = r

        candidates = []
        for rid, p in preds.items():
            if rid in fb and fb[rid]['feedback_type'] == 'corrected':
                candidates.append({'input': p['input'],
                                   'model_prediction': p['prediction'],
                                   'correct_answer': fb[rid]['correction'],
                                   'confidence': p['confidence']})
        return candidates

The flywheel spins fastest when feedback is nearly free for the user to give. A thumbs up/down costs one click and hands you a real signal. An explicit correction ("no, the answer was X") is worth ten thumbs, but it costs the user more effort, so you spend those requests carefully -- ask for corrections only on the predictions that matter most. And notice which records get_training_candidates fishes out: the LOW-confidence ones the user corrected. Those are the cases the model already knew it was shaky on, now with a ground-truth label attached. That is the highest-value training data money can buy, except you did not buy it -- your users handed it to you for free.

One caution, because I have seen this bite people. A raw feedback log is not a training set. Users who bother to click "wrong" are not a random sample -- angry people click more than happy ones (#35, bias in your data). So the flywheel gives you a fantastic stream of hard cases, but you still have to sanity-check that stream before you retrain on it, or you will happily teach your model to over-correct for the loudest 5% of your users.

Pricing: the AI product math

Here is where AI products quietly break the mental model most software people carry around. Traditional SaaS has near-zero marginal cost -- once the server is up, one more user costs you basically nothing. AI products do NOT work like that. Every single prediction has a real cost: compute, an API call, storage, GPU time. That one fact reshapes how you have to price the thing:

def estimate_unit_economics(monthly_users=1000,
                            predictions_per_user=50,
                            cost_per_prediction_usd=0.002,   # compute + API
                            monthly_infrastructure_usd=200,  # servers, storage, monitoring
                            monthly_price_per_user_usd=10):
    """Back-of-envelope unit economics for an AI product."""
    total_predictions = monthly_users * predictions_per_user
    variable_cost = total_predictions * cost_per_prediction_usd
    total_cost = variable_cost + monthly_infrastructure_usd
    revenue = monthly_users * monthly_price_per_user_usd
    profit = revenue - total_cost
    margin = profit / revenue if revenue > 0 else 0
    per_user_margin = (monthly_price_per_user_usd
                       - predictions_per_user * cost_per_prediction_usd)
    return {'monthly_predictions': total_predictions,
            'variable_cost_usd': round(variable_cost, 2),
            'total_cost_usd': round(total_cost, 2),
            'revenue_usd': revenue,
            'profit_usd': round(profit, 2),
            'margin_pct': round(margin * 100, 1),
            'cost_per_user_usd': round(total_cost / monthly_users, 2),
            'break_even_users': (int(monthly_infrastructure_usd / per_user_margin)
                                 if per_user_margin > 0 else None)}

# A document-analysis product billing per prediction to a model API:
print(estimate_unit_economics(monthly_users=500,
                              predictions_per_user=100,
                              cost_per_prediction_usd=0.01,
                              monthly_infrastructure_usd=150,
                              monthly_price_per_user_usd=29))
# {'monthly_predictions': 50000, 'variable_cost_usd': 500.0,
#  'total_cost_usd': 650.0, 'revenue_usd': 14500, 'profit_usd': 13850.0,
#  'margin_pct': 95.5, 'cost_per_user_usd': 1.3, 'break_even_users': 8}

Run that with different numbers and the danger jumps out. Watch what happens when a user gets heavy:

# Same $29/month price, but a power user hammering the model 2000x/month:
print(estimate_unit_economics(monthly_users=1,
                              predictions_per_user=2000,
                              cost_per_prediction_usd=0.01,
                              monthly_infrastructure_usd=0,
                              monthly_price_per_user_usd=29))
# variable_cost_usd = 20.0 on 29.0 of revenue - margin collapses to ~31%,
# and if they push past ~2900 calls they cost you MORE than they pay. Ouch.

The critical insight: heavy users are expensive. In classic SaaS a power user who logs in every day is a badge of honour and costs you nothing. In an AI product, a power user firing 500 predictions a day can cost more to serve than they pay you. That is why you see usage-based pricing, tiered limits, and a relentless push toward cheaper models everywhere in this industry -- it is not greed, it is survival. Price on a flat monthly fee and let a whale discover your product, and you can literally lose money on your biggest fan. Nota bene: model your worst-case power user BEFORE you publish a pricing page, not after.

Build vs buy: the honest framework

The most common question I get asked about AI products, by a mile: should we train our own model, or just use an API? People want a clever answer. The honest answer is boring, and here is the framework I lay the decision out on:

build_vs_buy = {
    "buy_api": {
        "when": [
            "General tasks - translation, summarisation, Q&A, extraction",
            "Prototyping and validation (ALWAYS start here)",
            "Small team, little in-house ML muscle",
            "The task keeps changing (a prompt tweak beats a retrain)",
        ],
        "risks": [
            "The provider changes pricing or terms out from under you",
            "Your data leaves your infrastructure (a privacy question)",
            "Latency now depends on someone else's uptime",
            "Cost scales linearly forever with usage",
        ],
    },
    "fine_tune": {
        "when": [
            "Domain-specific language - legal, medical, finance",
            "You need a consistent, rigid output format",
            "Volume is high enough to amortise the training cost",
            "Prompting has hit a ceiling you cannot prompt past",
        ],
        "risks": [
            "Your training-data quality sets the hard ceiling",
            "It needs ongoing care as the data drifts (#123)",
            "You now need real ML engineering on the team",
        ],
    },
    "train_from_scratch": {
        "when": [
            "A genuinely unusual data modality nobody pretrained on",
            "Brutal latency limits (sub-10ms) an API can't meet",
            "The model itself is your competitive moat",
            "Scale so large that API bills would bankrupt you",
        ],
        "risks": [
            "The highest upfront cost by far",
            "You need a serious, expensive ML team",
            "Real risk of building something WORSE than the API you replaced",
        ],
    },
}

My honest default advice, and I will die on this hill: start with an API. Always. You can find out whether the product idea even works before you spend one euro on training infrastructure. If it works on an API -- then, and only then -- you ask whether owning the model would improve your margins or unlock a capability you cannot rent. Most products never need to leave the API stage at all. The ones that genuinely do usually KNOW it, because their API bill is visibly eating their profit margin alive. That is a nice problem to have, and it is a much later problem than most founders think.

Ship it carefully: shadow and canary

One more thing that lives squarely in product-land, not model-land: how you release a model matters as much as the model. You do not flip a new model to 100% of users and pray. You ship it quietly alongside the old one first. Run the new model in the shadows -- it makes predictions, you log them, but users never see them -- and compare. Only when it clearly wins does it get real traffic, and even then, a slice at a time:

import hashlib

class ShadowCanary:
    """Compare a candidate model to production without risking users."""
    def __init__(self, production, candidate, canary_pct=0.0):
        self.production = production
        self.candidate = candidate
        self.canary_pct = canary_pct   # 0.0 = pure shadow, no user sees candidate
        self.log = []

    def _in_canary(self, user_id):
        bucket = int(hashlib.md5(str(user_id).encode()).hexdigest(), 16) % 100
        return bucket < self.canary_pct * 100

    def predict(self, user_id, x):
        prod = self.production.predict(x)
        cand = self.candidate.predict(x)             # always run it - for comparison
        self.log.append({'prod': prod, 'cand': cand, 'agree': prod == cand})
        # Users see the candidate ONLY if they fall in the canary slice:
        return cand if self._in_canary(user_id) else prod

    def agreement_rate(self):
        if not self.log:
            return None
        return sum(r['agree'] for r in self.log) / len(self.log)

With canary_pct=0.0 nobody is exposed -- you are just quietly recording where the new model disagrees with the old one, on real live inputs, at zero risk. When you have seen enough and the disagreements look like improvements in stead of regressions, you nudge the canary up to 5%, watch your product metrics (not just accuracy -- retention, complaints, latency), and roll forward only if the real world agrees with your offline test. This is exactly the systematic-doubt habit from last episode, wearing its deployment clothes.

Case study: what works, what doesn't

Let me be concrete, because patterns are easier to trust with examples attached.

What works: AI products that augment a human decision in stead of replacing it. A radiologist reviewing AI-flagged scans catches more anomalies than either the AI or the doctor working alone. A support agent with AI-suggested replies clears more tickets at higher quality. The human stays in the loop, catches the machine's mistakes, and the machine eats the tedious part. Everybody wins, and crucially nobody feels replaced.

What fails: AI products that promise full automation of messy human judgement. "AI writes all your marketing copy" delivers bland, detectable mush. "AI handles every customer complaint" delivers furious customers who can smell the bot from the first sentence. The technology is not the bottleneck here -- the expectation is. Promise a human-level experience and deliver 80%, and users feel cheated by the missing 20%. Promise an assistant that handles the routine stuff, deliver the same 80%, and users feel empowered. Same model, same accuracy, opposite outcome, decided entirely by the promise you made.

The quiet winners: the AI products nobody calls AI. Spam filters. Recommendation feeds. Search ranking. Fraud detection. The user has no idea a model is involved and does not care -- they just know the thing works. These win precisely because they are judged on outcomes, not on "AI-ness". When someone asks me what a great AI product looks like, I point at a spam folder that just quietly does its job. No chat bubble, no sparkle icon, no "powered by AI" badge -- just a problem that stopped bothering you ;-)

Where this leaves us

We turned a model into something a human can actually use and pay for: latency and fallbacks so it never feels broken, honest uncertainty so it never lies, a flywheel so it sharpens itself, pricing that survives a power user, and a careful release so a new version cannot torch your Tuesday. What is left in this series is not more model tricks -- you have plenty. What is left is how to keep yourself sharp: how to read the research so it feeds you in stead of drowning you, how all 150-plus pieces we have built click into one coherent stack, and where you point all of this next. Bring what you built today -- we are nearly at the end of the road, and the last stretch is about you, not the models.

The important bits

  • the model-to-product gap is real -- latency, error handling and honest UX decide user satisfaction far more than another point of accuracy; every prediction path needs a fallback so a user never sees a raw error;
  • communicate uncertainty honestly -- calibrated confidence (#13) wrapped in plain human language builds trust; hiding uncertainty, or wrapping nice words around a lying score, destroys it;
  • the data flywheel is the whole game -- log predictions, make feedback nearly free, and mine the corrected low-confidence cases for gold; just remember the feedback stream is biased, so sanity-check before you retrain (#35);
  • AI products carry a marginal cost per prediction -- this reshapes pricing and means a heavy user on a flat fee can cost you more than they pay; model your worst-case whale before you publish a price;
  • build vs buy has a default answer: start with an API, always -- validate the idea first, then own the model only when cost or capability genuinely forces your hand;
  • ship carefully -- shadow then canary a new model against production on real traffic, and roll forward on product metrics, not offline accuracy;
  • augment, don't replace -- human-in-the-loop products consistently beat full-automation ones, and the quiet invisible winners (spam filters, recommendations) succeed because they are judged on outcomes, not on being "AI".

Bedankt for reading all the way down here -- now go take one model you built in this series and wrap it in a single honest fallback and a confidence message, because feeling a product not-break is worth more than any paragraph of mine. Tot de volgende keer! ;-)

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments