Learn AI Series (#123) - Monitoring ML in Production

avatar

Learn AI Series (#123) - Monitoring ML in Production

variant-a-03-red.png

What will I learn

  • You will learn why monitoring an ML system is a fundamentally different job from monitoring ordinary software -- models fail silently, not loudly;
  • data drift: how to catch it when the input distribution quietly shifts out from under a model that was perfectly fine yesterday;
  • model drift (concept drift): how predictions rot over time even when not a single line of your code changed;
  • feature importance drift: reading which inputs are gaining or losing their predictive punch;
  • prediction distribution monitoring: the cheap leading indicator that screams before your ground-truth labels ever arrive;
  • shadow mode: testing a new model on real production traffic without a single user ever feeling it;
  • and a sane alerting strategy -- when to page someone at 3 AM, when to just file a ticket, and when to do nothing at all.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • An installed Python 3(.11+) distribution with NumPy, SciPy and scikit-learn (pip install numpy scipy scikit-learn covers it);
  • You've been through episode #13 (evaluation), episode #117 (ML system design), and episodes #121 and #122 (serving that model, then shipping it to the edge) -- today we ask the nervous follow-up question those posts kept dodging: is the thing still working?

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#123) - Monitoring ML in Production

Here's a nasty little truth that separates ML systems from every other kind of software you've ever shipped. Ordinary software fails loudly. A function gets a bad input and throws an exception, a service runs out of memory and crashes, a request times out and someone's pager goes off. The failure announces itself. You get a stack trace, a red graph, an angry Slack message -- something to grab onto.

Machine learning does not do you that courtesy. ML systems fail silently. The model keeps returning predictions. The API keeps answering with a cheerful 200 OK. The dashboards stay green. And meanwhile the predictions are getting quietly, steadily worse -- until three weeks later somebody in the business notices that conversions are down 8% and nobody has the faintest idea why. That is the whole reason this episode exists.

So let me be blunt about the mental shift. Monitoring a normal service means watching for crashes and latency. Monitoring an ML system means watching for something much sneakier: the world changing in ways your model never saw during training. Your code is fine. Your infrastructure is fine. Your model is fine, in the narrow sense that it's doing exactly what it was trained to do. The problem is that reality moved and the model didn't get the memo. Let's learn how to hear the alarm before the fire spreads.

Data drift: the input distribution moved

Start with the most common failure mode. Your model trained on data from January through June. It's December now. New product categories exist that didn't before, a holiday shopping season warped everyone's behaviour, some macro event shifted the whole population. The model still makes predictions -- it just makes worse ones, because the inputs it's seeing no longer look like the inputs it learned from. This is data drift, and the good news is it's measurable without needing any ground-truth labels at all.

import numpy as np
from scipy import stats

class DataDriftDetector:
    """Detect when the input distribution wanders away from training."""
    def __init__(self, reference_data, feature_names):
        self.reference = reference_data          # the training distribution
        self.feature_names = feature_names
        self.reference_stats = self._compute_stats(reference_data)

    def _compute_stats(self, data):
        return {
            'means': np.mean(data, axis=0),
            'stds': np.std(data, axis=0),
            'quantiles': np.percentile(data, [5, 25, 50, 75, 95], axis=0),
        }

    def check_drift(self, current_data, method='psi'):
        """Compare live data against the reference, feature by feature."""
        results = {}
        for i, name in enumerate(self.feature_names):
            ref_col = self.reference[:, i]
            cur_col = current_data[:, i]

            if method == 'ks':
                # Kolmogorov-Smirnov: statistically pure, trigger-happy at scale
                stat, p_value = stats.ks_2samp(ref_col, cur_col)
                drifted = p_value < 0.01
            else:
                # Population Stability Index: the industry workhorse
                stat = self._psi(ref_col, cur_col)
                p_value = None
                drifted = stat > 0.25

            results[name] = {
                'statistic': stat,
                'p_value': p_value,
                'drifted': drifted,
                'ref_mean': float(np.mean(ref_col)),
                'cur_mean': float(np.mean(cur_col)),
            }
        return results

    def _psi(self, reference, current, bins=10):
        """Population Stability Index -- the drift metric you'll actually ship."""
        ref_pct = np.histogram(reference, bins=bins)[0] / len(reference)
        cur_pct = np.histogram(current, bins=bins)[0] / len(current)

        # log(0) is a party-pooper, so clip the empty bins
        ref_pct = np.clip(ref_pct, 1e-4, None)
        cur_pct = np.clip(cur_pct, 1e-4, None)

        return np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct))

# Usage: a reference that shifted by +0.3 on every feature
reference = np.random.randn(10000, 5)
current = np.random.randn(1000, 5) + 0.3          # the world moved

detector = DataDriftDetector(reference, ['f1', 'f2', 'f3', 'f4', 'f5'])
for name, r in detector.check_drift(current).items():
    tag = "DRIFTED" if r['drifted'] else "stable"
    print(f"{name}: PSI={r['statistic']:.4f} [{tag}]")

PSI (Population Stability Index) is the metric you'll meet in every risk and fraud team on the planet, and it's worth memorising the rules of thumb because they barely change between shops: PSI below 0.1 means no meaningful shift, 0.1 to 0.25 means a moderate shift worth investigating, and above 0.25 means a major shift that warrants urgent attention -- retrain, or at least go find out why. It's just measuring how much probability mass moved between the histogram bins of your training data and your live data.

Now, why did I default to PSI instead of the more famous Kolmogorov-Smirnov test? Because KS has a trap waiting for you. It's more statistically rigorous, yes -- but with large samples it becomes almost uselessly sensitive. Feed it 100,000+ rows and it will flag a trivially small shift as "statistically significant" (p < 0.001) even when nothing meaningful changed. You'll drown in alerts for drift that doesn't matter. PSI cares about the size of the shift, not just whether one exists, which is exactly what you want when a human has to act on the number. Having said that, KS is lovely for smaller batches -- so keep both in the toolbox.

Data quality: the failures that aren't even drift

Before we go chasing subtle statistical drift, let me flag the dumbest and most common production failure of all, because it embarrasses everybody eventually: the data pipeline breaks and starts feeding your model garbage. A column that used to be a percentage in [0, 1] starts arriving as [0, 100]. A feature quietly goes 40% null because an upstream service changed its schema. Your fancy drift detector will absolutely light up -- but it'll blame the model, when the real culprit is a broken pipe two systems away. So check the plumbing first.

class DataQualityMonitor:
    """Catch the boring, high-frequency failures before blaming the model."""
    def __init__(self, feature_names, expected_ranges):
        self.feature_names = feature_names
        self.expected_ranges = expected_ranges       # name -> (low, high)

    def check_batch(self, X):
        issues = []
        for i, name in enumerate(self.feature_names):
            col = X[:, i]

            # 1. Null / NaN rate -- the classic silent killer
            null_rate = float(np.mean(np.isnan(col)))
            if null_rate > 0.05:
                issues.append({'feature': name, 'issue': 'high_null_rate',
                               'value': null_rate})

            # 2. Out-of-range values -- schema or unit changes upstream
            low, high = self.expected_ranges[name]
            valid = col[~np.isnan(col)]
            oor = float(np.mean((valid < low) | (valid > high))) if len(valid) else 0.0
            if oor > 0.01:
                issues.append({'feature': name, 'issue': 'out_of_range',
                               'value': oor})
        return issues

# Usage
ranges = {'age': (0, 120), 'score': (0.0, 1.0)}
monitor = DataQualityMonitor(['age', 'score'], ranges)
batch = np.array([[35, 0.7], [42, 1.4], [np.nan, 0.2]])   # a 1.4 score, a null age
for problem in monitor.check_batch(batch):
    print(problem)

This is unglamorous and it will save your bacon more often than any of the clever stuff. A shocking share of "the model degraded" incidents turn out to be a renamed column, a timezone bug, or a service that started returning -1 for "unknown" instead of null. Catch those at the door with a range-and-null check and you'll spend your actual drift budget on actual drift.

Model drift: the predictions are rotting

Data drift is about the inputs. Concept drift (often called model drift) is nastier: the relationship between inputs and outputs changes. The inputs can look completely normal while the pattern your model learned quietly stops being true. Think of a spam filter -- the emails look the same, but spammers evolve their tactics, and last year's "spammy" signals become this year's normal newsletter. The only way to catch this cold is to compare predictions against ground truth over time.

class ModelDriftMonitor:
    """Track prediction quality across rolling windows of time."""
    def __init__(self, window_size=1000):
        self.window_size = window_size
        self.predictions = []

    def log_prediction(self, prediction, confidence, timestamp):
        self.predictions.append({
            'pred': prediction,
            'confidence': confidence,
            'timestamp': timestamp,
        })

    def log_label(self, index, true_label):
        """Ground truth shows up late -- hours, days, or never."""
        if index < len(self.predictions):
            self.predictions[index]['true_label'] = true_label

    def rolling_metrics(self):
        """Accuracy and mean confidence over sliding windows."""
        labeled = [p for p in self.predictions if 'true_label' in p]
        if len(labeled) < self.window_size:
            return None

        windows = []
        step = self.window_size // 4
        for i in range(0, len(labeled) - self.window_size, step):
            w = labeled[i:i + self.window_size]
            correct = sum(1 for p in w if p['pred'] == p['true_label'])
            windows.append({
                'accuracy': correct / len(w),
                'avg_confidence': float(np.mean([p['confidence'] for p in w])),
                'timestamp': w[-1]['timestamp'],
            })
        return windows

    def detect_performance_drop(self, threshold=0.05):
        """Alert when recent accuracy falls below the historical baseline."""
        metrics = self.rolling_metrics()
        if not metrics or len(metrics) < 4:
            return {'alert': False}

        baseline = np.mean([m['accuracy'] for m in metrics[:len(metrics) // 2]])
        recent = np.mean([m['accuracy'] for m in metrics[-2:]])
        drop = baseline - recent

        return {
            'alert': drop > threshold,
            'baseline_accuracy': float(baseline),
            'recent_accuracy': float(recent),
            'drop': float(drop),
        }

And here's the wrinkle that makes model drift genuinely hard in the real world: your labels almost always arrive late. A fraud model flags a transaction right now, but you don't know whether it was really fraud until the chargeback lands weeks later. A loan-default model makes a call today that reality won't confirm for months. Sometimes the label never comes at all. So you can't just wait for accuracy to tank -- by the time you've got enough labels to be sure, the damage has been compounding for a month. Which is precisely why the next two sections matter so much: they let you smell trouble without waiting for the truth.

Feature importance drift

Your model leans on certain features more than others -- back in episode #13 we cared about whether a model works, and part of "works" is which signals it's actually using. Over time, a feature that used to carry the model can lose its predictive power, or a feature the model was trained to ignore can suddenly become the most informative thing in the dataset. Watching that shift tells you why performance is moving, not just that it is.

class FeatureImportanceDrift:
    """Monitor which features are actually driving predictions."""
    def __init__(self, model, feature_names):
        self.model = model
        self.feature_names = feature_names

    def permutation_importance(self, X, y, n_repeats=5):
        """Shuffle each feature and watch how much the score falls."""
        baseline = self._score(X, y)
        importances = {}

        for i, name in enumerate(self.feature_names):
            drops = []
            for _ in range(n_repeats):
                X_perm = X.copy()
                np.random.shuffle(X_perm[:, i])          # break this feature
                drops.append(baseline - self._score(X_perm, y))
            importances[name] = {
                'mean': float(np.mean(drops)),
                'std': float(np.std(drops)),
            }
        return importances

    def _score(self, X, y):
        return float(np.mean(self.model.predict(X) == y))

    def compare(self, training_imp, current_imp):
        """Flag features whose importance changed a lot since training."""
        alerts = []
        for name in self.feature_names:
            was = training_imp[name]['mean']
            now = current_imp[name]['mean']
            if was > 0.01 and now < 0.001:
                alerts.append({'feature': name, 'issue': 'lost importance',
                               'training': was, 'current': now})
            elif was < 0.001 and now > 0.01:
                alerts.append({'feature': name, 'issue': 'gained importance',
                               'training': was, 'current': now})
        return alerts

The idea behind permutation importance is delightfully simple: to measure how much a model relies on a feature, you shuffle that feature's values into nonsense and see how much the score drops. Big drop, important feature. No drop, the model wasn't really using it. Do that on your training data and your live data, then compare. When a once-important feature goes quiet, something in the world changed and your model is now leaning on a crutch that's been kicked away. When a once-useless feature lights up, there's a fresh signal sitting right there that retraining would let you exploit. Either way, the drift in importance is often your earliest, most diagnostic clue -- it doesn't just say "something's wrong", it points a finger.

Prediction distribution monitoring: the cheap early warning

Right, so labels are late and permutation importance needs labels too. What can you watch in real time, for basically free, that needs no ground truth whatsoever? The distribution of your model's own outputs. This is the single highest-value-per-effort monitor you can build, and I'd put it in place before anything else.

class PredictionDistributionMonitor:
    """Watch the shape of the model's outputs, no labels required."""
    def __init__(self):
        self.reference = None

    def set_reference(self, predictions):
        """Freeze the expected output distribution from your validation set."""
        if hasattr(predictions[0], '__len__'):
            # Classification: store per-class rates
            classes = [int(np.argmax(p)) for p in predictions]
            uniq, counts = np.unique(classes, return_counts=True)
            self.reference = dict(zip(uniq.tolist(), (counts / len(classes)).tolist()))
        else:
            # Regression: store summary statistics
            self.reference = {
                'mean': float(np.mean(predictions)),
                'std': float(np.std(predictions)),
            }

    def check(self, current):
        """Compare today's predictions against the frozen reference."""
        if 'mean' in self.reference:                       # regression path
            cur_mean = float(np.mean(current))
            z = abs(cur_mean - self.reference['mean']) / (self.reference['std'] + 1e-8)
            return {'z_score': z, 'alert': z > 3,
                    'message': f"output mean moved {z:.1f} sigma from reference"}

        # classification path -- biggest per-class rate change
        classes = [int(np.argmax(p)) for p in current]
        uniq, counts = np.unique(classes, return_counts=True)
        cur = dict(zip(uniq.tolist(), (counts / len(classes)).tolist()))
        max_shift = max(abs(cur.get(c, 0) - r) for c, r in self.reference.items())
        return {'max_class_shift': max_shift, 'alert': max_shift > 0.1,
                'message': f"max class-rate shift: {max_shift:.1%}"}

Picture your spam classifier. On a normal day it labels maybe 5% of messages as spam. This morning it's suddenly flagging 30%. You do not need the ground-truth labels to know something has gone badly sideways -- the output distribution alone is screaming. Maybe an upstream data pipeline started feeding garbage, maybe a real spam wave hit, maybe a feature is arriving as null and defaulting to something toxic. You don't know the cause yet, but you know to look, and you knew it within minutes instead of weeks. That's what makes prediction distribution monitoring a leading indicator: it fires before the labels that would confirm the damage have even been collected.

Shadow mode: testing on reality without touching users

Say your monitors have been nagging you and you've trained a shiny new model. How do you know it's actually better before you dare route real users to it? In episode #121 we met canary deployments -- send 5% of live traffic to the new model and watch. Shadow mode is the even-safer sibling: run the new model on 100% of production traffic, but throw its answers away. Users only ever see the old model. You get a full-scale dress rehearsal with a blast radius of exactly zero.

class ShadowDeployment:
    """Run a candidate model alongside production, discard its output."""
    def __init__(self, production_model, shadow_model):
        self.production = production_model
        self.shadow = shadow_model
        self.comparisons = []

    def predict(self, features):
        prod = self.production.predict(features)

        # The shadow runs on the same real input -- but its answer never ships
        shadow = self.shadow.predict(features)

        self.comparisons.append({
            'production': prod,
            'shadow': shadow,
            'agree': prod == shadow,
        })
        return prod                                  # always the production answer

    def agreement_report(self):
        if not self.comparisons:
            return {'note': 'no traffic yet'}
        agree = sum(c['agree'] for c in self.comparisons) / len(self.comparisons)
        return {
            'total': len(self.comparisons),
            'agreement_rate': agree,
            'disagreements': sum(1 for c in self.comparisons if not c['agree']),
        }

Run it for a week or two, then go read the disagreements -- because the disagreements are where all the information lives. Where the two models agree, who cares. Where they disagree, that's exactly the set of cases where switching would change a user's experience, and it's where you find out whether the new model is genuinely smarter or just differently wrong. If the shadow wins on the cases that matter, and its mistakes are less costly than the incumbent's, then you graduate it to a canary and start easing real traffic across. Shadow first, canary second, full rollout last. Rushing that ladder is how quit some teams have shipped a regression to everyone at once ;-)

An alerting strategy that doesn't cry wolf

You can measure all of the above, but measurement without a decision framework just gives you a wall of noisy graphs nobody trusts. The hardest part of monitoring isn't computing the metrics -- it's deciding what actually deserves a human's attention, and at what hour. Here's a compact threshold table to anchor the thinking.

alert_thresholds = {
    'data_drift_psi': {
        'warning':  0.10,   # moderate shift -> investigate this week
        'critical': 0.25,   # major shift    -> investigate today
    },
    'accuracy_drop': {
        'warning':  0.03,   # 3% down  -> schedule a retrain
        'critical': 0.10,   # 10% down -> retrain now, this is real
    },
    'prediction_shift': {
        'warning':  0.05,   # output rates moved 5%
        'critical': 0.15,   # moved 15% -> something upstream is broken
    },
    'latency_p99': {
        'warning':  200,    # ms -- creeping toward the SLA
        'critical': 500,    # ms -- violating the SLA
    },
    'null_rate': {
        'warning':  0.01,   # 1% nulls in a critical feature
        'critical': 0.05,   # 5% -- your data pipeline is bleeding
    },
}

def triage(metric_name, value):
    """Turn a raw number into an action, not just a colour on a graph."""
    t = alert_thresholds[metric_name]
    if value >= t['critical']:
        return 'PAGE'      # wake the on-call human
    if value >= t['warning']:
        return 'TICKET'    # file it for the next sprint
    return 'OK'            # do nothing, and mean it

The philosophy in one sentence: warnings create tickets, criticals wake people up, and most drift is neither. That last part is the one juniors get wrong. Not every distribution shift means the model is broken -- seasonality is real, new user segments are real, a marketing campaign that changes your traffic mix is real. If you page someone every time PSI twitches to 0.11, they'll mute your alerts inside a week and then miss the one that mattered. Nota bene: the goal of monitoring was never "zero alerts". The goal is zero surprises -- you want to be the person explaining the dip in the meeting, not the person hearing about it there.

Wiring it together: a monitoring loop

None of these pieces is much use sitting in a drawer. In production they run on a schedule -- hourly, daily, per-batch -- reading a slice of recent traffic, updating the metrics, and routing anything alarming to the right place. Here's the skeleton that ties the whole episode into one honest little loop.

def monitoring_cycle(recent_features, recent_predictions,
                     drift_detector, pred_monitor):
    """One pass of a scheduled monitoring job -- run it hourly in prod."""
    report = {'actions': []}

    # 1. Did the inputs drift? (no labels needed)
    drift = drift_detector.check_drift(recent_features)
    worst_psi = max(r['statistic'] for r in drift.values())
    report['worst_feature_psi'] = worst_psi
    report['actions'].append(('data_drift', triage('data_drift_psi', worst_psi)))

    # 2. Did the outputs drift? (no labels needed -- the leading indicator)
    dist = pred_monitor.check(recent_predictions)
    shift = dist.get('max_class_shift', 0)
    report['actions'].append(('prediction_shift', triage('prediction_shift', shift)))

    # 3. Anyone need waking up?
    report['page'] = any(action == 'PAGE' for _, action in report['actions'])
    return report

Look at what that gives you for almost no code. Two of the three checks need no ground truth at all, so they run every hour and catch the fast failures early. The label-dependent accuracy check runs whenever your labels actually land -- daily, weekly, whenever the chargebacks come in -- and it's the slow, authoritative confirmation. Layer the fast-cheap monitors on top of the slow-certain ones and you've got defence in depth: the cheap monitors buy you time, the expensive one gives you truth. That's the whole game.

So, what do you know now?

  • ML systems fail silently -- the API stays green while predictions rot -- so monitoring here means watching the world change, not just watching for crashes;
  • data drift (the input distribution moving) is caught with PSI or KS tests, needs no labels, and PSI above 0.25 is your urgent-investigate line -- prefer PSI over KS at scale, where KS turns into an alert firehose;
  • model / concept drift (predictions degrading) needs ground truth, which arrives late or never, so you lean on proxy signals while you wait;
  • feature importance drift via permutation is your most diagnostic clue -- it tells you which signal changed, not merely that something did;
  • prediction distribution monitoring is the cheap leading indicator that fires before any label arrives -- build it first;
  • shadow mode runs a candidate on 100% of real traffic with zero user impact, and the disagreements are where the truth hides;
  • and a real alerting strategy separates page-worthy criticals from file-it-later warnings, accepts that a lot of drift is just life, and chases zero surprises, not zero alerts.

Exercises

Three to chew on before the next one. Get your hands dirty first -- I'll walk full solutions next time, as always.

  1. Make PSI earn its reputation. Generate a reference array of 10,000 samples from np.random.randn. Then create three "current" batches: one identical in distribution, one shifted by +0.2, and one shifted by +0.8. Run the DataDriftDetector PSI on all three and confirm the numbers land in the no-shift / moderate / major buckets. Now rerun with the ks method and write one sentence on which one you'd trust to page a human at 2 AM, and why.

  2. Prove the leading-indicator claim. Simulate a classifier whose spam rate is 5% for the first 900 predictions and then jumps to 30% for the next 100 (no labels anywhere). Feed the stream through PredictionDistributionMonitor in windows and show that it raises an alert from the output distribution alone. Then explain in two sentences why this beats waiting for accuracy to drop.

  3. Design the triage, not just the metric. Take the alert_thresholds table and add a new metric of your own -- say feature_null_rate for a single critical feature. Pick defensible warning and critical values, wire it into the triage function, and write a short paragraph justifying your thresholds. Bonus: describe one real situation where a warning-level null rate is actually fine and you'd choose to do nothing.

We now have a model that trains, gets served, runs on the edge, and tells us the moment it starts drifting. But notice we've been doing all the retraining and redeploying by hand this whole time -- a human noticing an alert, kicking off a training run, eyeballing the metrics, pushing the new model out. That doesn't scale, and it's exactly the kind of manual, error-prone ritual that begs to be automated end to end. How you build that automated pipeline -- the one that tests and ships a model the way good teams test and ship code -- is where we head next.

Thanks for your time, and happy monitoring!

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments