Learn AI Series (#130) - Causal Inference and ML

avatar

Learn AI Series (#130) - Causal Inference and ML

variant-c-11-teal.png

What will I learn

  • You will learn why correlation is not causation and why that gap quietly breaks ML systems the moment somebody uses their predictions to act instead of just to watch;
  • Simpson's Paradox -- the genuinely unsettling result where a treatment that helps every subgroup can look harmful in the pooled data, and how a naive model walks straight into the trap;
  • causal graphs (DAGs) and the do-calculus -- Pearl's language for writing down what causes what so you can reason about interventions in stead of mere observations;
  • propensity score matching, the most practical trick in the toolbox -- pair up treated and untreated units that look alike on the confounders, then read off the effect;
  • Double Machine Learning, the beautiful idea that lets you use any ML model (gradient boosting, a neural net, whatever) for the heavy lifting while still getting an honest, unbiased causal estimate;
  • and the honest part -- when causal thinking earns its keep in a real ML pipeline, and when a plain predictive model was fine all along.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • An installed Python 3(.10+) distribution with scikit-learn (pip install scikit-learn numpy) -- everything here runs on a laptop in seconds, there is not a GPU in sight today;
  • You've been through episode #10-#12 (linear and logistic regression, which we lean on constantly here), episode #14 (data preparation, where we first met leakage), and especially episode #32 (Bayesian Methods) -- today is where all that machinery stops answering "what will happen?" and starts answering "what should we do?".

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#130) - Causal Inference and ML

I closed episode #129 with a warning dressed up as a cliffhanger. Every technique in that AutoML episode -- random search, Bayesian optimization, the whole architecture-search circus -- quietly assumed one thing: that correlation in the data is enough. Find whatever pattern predicts the target best and ship it. And for a huge slice of ML that assumption is perfectly fine. But it has a fault line running straight through the middle, and the fault line shows up the moment somebody stops watching the predictions and starts acting on them.

Here is the whole episode in one example. Your model, trained beautifully, predicts that customers who received a discount are more likely to churn. So somebody in a meeting says: "great, stop sending discounts, they cause churn." Do they? Or were the customers who got the discount already halfway out the door -- flagged as churn-risky by some retention system, which is why they got the discount in the first place? The model cannot tell you. It only saw that discount and churn move together. Whether pulling the discount lever changes churn is a completely different question, and standard ML has literally no machinery to answer it. Causal inference does. That is the gap we close today ;-)

Solutions to episode #129's exercises

As promised, let's clear last time's AutoML homework before we open the causal toolbox.

Exercise 1 -- race random search against Bayesian. The task was to tune a GradientBoostingClassifier two ways -- 40 iterations of random search versus 40 Optuna trials -- and watch which one is ahead early and which wins by the end. The shape you are meant to see is that random search often leads in the first handful of trials (pure luck, it is sampling widely) but Bayesian optimization overtakes it and pulls away, because it learns from every trial instead of forgetting them.

import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score
import optuna

X, y = load_breast_cancer(return_X_y=True)

def score(lr, depth, n_est):
    model = GradientBoostingClassifier(learning_rate=lr, max_depth=depth,
                                       n_estimators=n_est, random_state=0)
    return cross_val_score(model, X, y, cv=5).mean()

# Random search: independent guesses, zero memory.
rng = np.random.default_rng(0)
best_random = -np.inf
for _ in range(40):
    s = score(rng.uniform(0.01, 0.3), rng.integers(2, 6), rng.integers(50, 300))
    best_random = max(best_random, s)

# Optuna: a Bayesian study that aims each next guess.
def objective(trial):
    return score(trial.suggest_float('lr', 0.01, 0.3),
                 trial.suggest_int('depth', 2, 5),
                 trial.suggest_int('n_est', 50, 300))
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=40)

print(f"random best : {best_random:.4f}")
print(f"optuna best : {study.best_value:.4f}")

The one-sentence answer I was fishing for: random search is a fine sprinter over the first ten trials but a poor marathoner, because Bayesian optimization spends its later trials aiming at the promising region instead of re-rolling the dice. On an easy dataset like this the two often finish close (there is not much score left to squeeze), which is itself the lesson -- the harder and more expensive the objective, the more the aiming pays off.

Exercise 2 -- watch the alphas move. Instantiate the DARTSCell from last episode, train it to imitate a fixed target, and print the softmax over the architecture weights before and after. The point was to feel one operation win the popularity contest as gradient descent flows into the alphas.

import torch

cell = DARTSCell(channels=4, n_ops=6)          # from episode #129
x = torch.randn(1, 4, 8, 8)
target = torch.randn(1, 4, 8, 8)
opt = torch.optim.Adam(cell.parameters(), lr=0.05)

print("before:", torch.softmax(cell.alphas, dim=0).detach().round(decimals=3))
for _ in range(200):
    opt.zero_grad()
    loss = ((cell(x) - target) ** 2).mean()
    loss.backward()
    opt.step()
print("after :", torch.softmax(cell.alphas, dim=0).detach().round(decimals=3))

Before training the softmax is nearly uniform (every operation roughly 0.16 -- the cell has no opinion yet). After 200 steps one or two operations climb toward the top while the rest sag. Discretizing this trained cell means keeping only the operation with the largest alpha and throwing the others away, turning the soft blend into a single concrete layer. That discretize-after-search step is exactly where DARTS turns a differentiable search back into a real network.

Exercise 3 -- decide: AutoML or not? No single right answer, only defensible reasoning. (i) A startup needing a strong churn baseline on tabular data by Friday -> run full AutoML (FLAML), because tabular is precisely where AutoML reliably matches expert tuning and the deadline rewards walking away while it works. (ii) A research lab inventing a model for a brand-new sensor modality -> design by hand, because AutoML only searches known spaces and a novel modality has none. (iii) An embedded team needing the smallest classifier that fits in 2MB -> Bayesian HPO on a hand-picked tiny architecture, because the hard size constraint means you pick the model family yourself and let the optimizer tune within it. If you justified why each choice fit that constraint, you nailed it.

Correlation is not causation -- and why it actually bites

Everybody nods along at "correlation is not causation" as if it were a fortune cookie. Then they train a model, read its coefficients, and make a decision as though it were causation. So let me show you the failure with numbers, because a demonstration sticks where a slogan slides off. Behold Simpson's Paradox -- a treatment that genuinely helps can look actively harmful in the pooled data.

import numpy as np
from sklearn.linear_model import LogisticRegression

np.random.seed(42)
n = 2000

# The CONFOUNDER: disease severity. It drives BOTH who gets treated AND who recovers.
severity = np.random.beta(2, 5, n)

# Sicker patients are more likely to receive the treatment (doctors triage).
treatment_prob = 0.3 + 0.5 * severity
treatment = np.random.binomial(1, treatment_prob)

# Outcome: the treatment genuinely HELPS (+0.2), but severity HURTS (-0.4).
recovery_prob = 0.6 - 0.4 * severity + 0.2 * treatment
recovery = np.random.binomial(1, np.clip(recovery_prob, 0.01, 0.99))

# Naive analysis -- ignore severity entirely. Treatment looks HARMFUL.
naive = LogisticRegression().fit(treatment.reshape(-1, 1), recovery)
print(f"naive coefficient      : {naive.coef_[0][0]:+.3f}")   # negative!

# Correct analysis -- control for the confounder. Treatment looks HELPFUL.
correct = LogisticRegression().fit(np.column_stack([treatment, severity]), recovery)
print(f"severity-adjusted coeff: {correct.coef_[0][0]:+.3f}") # positive!

Look at what happened. In the raw data the treatment correlates with worse recovery -- not because it hurts, but because the sickest patients are the ones who get it. Severity is a confounder: it reaches into both the treatment and the outcome, and it poisons the naive comparison. The instant we hold severity fixed, the treatment's true, helpful effect surfaces. A model that reads the naive coefficient and recommends "stop treating people" would be catastrophically wrong -- and notice, its accuracy on held-out data might be perfectly good. This is not a bug you catch with a validation set. It is a bug you catch only by asking a causal question.

Causal graphs: writing down what causes what

The fix begins with honesty about your assumptions, and the tool for that is the Directed Acyclic Graph (DAG). You draw an arrow from cause to effect, and the graph becomes a map of where confounding hides. This is the heart of Judea Pearl's framework, and the lovely part is that a tiny bit of Python makes the ideas concrete.

class CausalGraph:
    """A minimal DAG: an edge (parent -> child) means 'parent causes child'."""
    def __init__(self):
        self.edges = []

    def add_edge(self, parent, child):
        self.edges.append((parent, child))

    def parents(self, node):
        return [p for p, c in self.edges if c == node]

    def children(self, node):
        return [c for p, c in self.edges if p == node]

    def is_confounder(self, node, treatment, outcome):
        """A confounder is a common cause of BOTH treatment and outcome."""
        return (treatment in self.children(node) and
                outcome in self.children(node))

# Encode the medical example: severity is a common cause; treatment affects recovery.
dag = CausalGraph()
dag.add_edge('severity', 'treatment')
dag.add_edge('severity', 'recovery')
dag.add_edge('treatment', 'recovery')

print("severity is a confounder:",
      dag.is_confounder('severity', 'treatment', 'recovery'))

The graph earns its keep by telling you which variables you must adjust for. Here it says plainly: to isolate the effect of treatment on recovery, block the backdoor path that runs treatment <- severity -> recovery by controlling for severity. Pearl formalises the difference with the do-operator. The observational quantity P(recovery | treatment) asks "among people who happened to be treated, what is recovery?" -- and that is contaminated by why they were treated. The interventional quantity P(recovery | do(treatment)) asks "if we reach in and treat, what is recovery?" -- and that is the number a decision-maker actually wants. They are not the same object; conflating them is the original sin of applied ML.

And once the graph tells you what to adjust for, you can compute the interventional answer by hand -- the backdoor adjustment formula says: estimate the effect within each stratum of the confounder, then average over how common those strata are.

import numpy as np

# P(recovery | do(treatment)) via backdoor adjustment: bucket by severity,
# compute the treated-minus-control gap INSIDE each bucket, then weight-average.
buckets = np.linspace(0, 1, 6)          # 5 severity strata
strata = np.digitize(severity, buckets)

effect, weight_total = 0.0, 0
for s in np.unique(strata):
    m = strata == s
    t, c = recovery[m & (treatment == 1)], recovery[m & (treatment == 0)]
    if len(t) and len(c):               # need both groups present in the stratum
        effect += (t.mean() - c.mean()) * m.sum()
        weight_total += m.sum()

print(f"adjusted causal effect: {effect / weight_total:+.3f}")   # positive again

Notice this uses no fancy model at all -- just conditioning on the confounder the graph flagged, exactly as the do-operator demands. It agrees with everything else: the treatment helps once severity is held fixed.

Having said that, a DAG is only as honest as the person drawing it. It is a statement of belief about the world, not a fact extracted from data -- draw the arrows wrong and every estimate downstream inherits the mistake. That is a feature, not a weakness: it forces your assumptions into the open where a colleague can argue with them.

Propensity score matching: the workhorse

So we know we must adjust for confounders. The most intuitive way to actually do it is matching: for each treated individual, find an untreated individual who looked just like them on the confounders, and compare their outcomes. The snag is that with many confounders "just like them" becomes impossible -- nobody matches on twenty variables at once (the curse of dimensionality, which we met back in episode #24). Rosenbaum and Rubin's trick is to collapse all those confounders into a single number -- the propensity score, the modelled probability of receiving treatment -- and match on that.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import NearestNeighbors

class PropensityScoreMatching:
    """Estimate a treatment effect by matching on the propensity score."""
    def __init__(self):
        self.propensity_model = LogisticRegression()

    def fit_propensity(self, X, treatment):
        """Learn P(treatment = 1 | confounders) -- the propensity score."""
        self.propensity_model.fit(X, treatment)
        return self.propensity_model.predict_proba(X)[:, 1]

    def match(self, scores, treatment, outcome, n_neighbors=1):
        """Pair each treated unit with its nearest untreated unit, by score."""
        treated_idx = np.where(treatment == 1)[0]
        control_idx = np.where(treatment == 0)[0]

        nn = NearestNeighbors(n_neighbors=n_neighbors)
        nn.fit(scores[control_idx].reshape(-1, 1))

        treated_out, matched_out = [], []
        for idx in treated_idx:
            _, m = nn.kneighbors(scores[idx].reshape(1, -1))
            treated_out.append(outcome[idx])
            matched_out.append(outcome[control_idx[m[0]]].mean())

        # Average Treatment effect on the Treated (ATT).
        return np.mean(np.array(treated_out) - np.array(matched_out))

X = severity.reshape(-1, 1)
psm = PropensityScoreMatching()
scores = psm.fit_propensity(X, treatment)
att = psm.match(scores, treatment, recovery)
print(f"estimated treatment effect (ATT): {att:+.3f}")   # positive -- treatment helps

The estimate that pops out is the Average Treatment effect on the Treated -- how much the treatment helped the people who actually got it -- and it lands positive, agreeing with the severity-adjusted regression and flatly contradicting the naive slogan-level analysis. Nota bene: matching is only as trustworthy as the confounders you fed it. If some sneaky common cause is missing from X, the propensity model never sees it, and the bias walks right back in through the unguarded door. There is no statistical test that saves you from an unmeasured confounder -- only domain knowledge does, which is why causal inference is a conversation between the data and someone who understands the problem.

Double Machine Learning: ML muscle, causal honesty

Propensity matching leans on a plain logistic model. But what if the relationship between confounders and treatment is a tangled, non-linear mess that only gradient boosting or a neural net can capture? Naively throwing a flexible ML model at the problem reintroduces bias, because these models overfit and their errors leak into the effect estimate. Double Machine Learning (Chernozhukov et al., 2018) is the elegant repair, and it rests on two ideas: residualization and cross-fitting.

import numpy as np
from sklearn.ensemble import GradientBoostingRegressor, GradientBoostingClassifier
from sklearn.model_selection import cross_val_predict

class DoubleML:
    """Debiased/orthogonal ML for a causal effect estimate."""
    def __init__(self):
        self.outcome_model = GradientBoostingRegressor(n_estimators=100)
        self.treatment_model = GradientBoostingClassifier(n_estimators=100)

    def estimate(self, X, treatment, outcome):
        # 1. Predict the OUTCOME from confounders, cross-fitted, and take residuals.
        y_resid = outcome - cross_val_predict(self.outcome_model, X, outcome, cv=5)

        # 2. Predict the TREATMENT from confounders, cross-fitted, and take residuals.
        t_hat = cross_val_predict(self.treatment_model, X, treatment,
                                  cv=5, method='predict_proba')[:, 1]
        t_resid = treatment - t_hat

        # 3. Regress the outcome residuals on the treatment residuals: that slope
        #    IS the causal effect, orthogonal to errors in either nuisance model.
        ate = np.mean(y_resid * t_resid) / np.mean(t_resid ** 2)

        r = y_resid - ate * t_resid
        se = np.sqrt(np.mean(r ** 2 * t_resid ** 2)) / abs(np.mean(t_resid ** 2)) / np.sqrt(len(X))
        return {'ate': ate, 'se': se, 'ci_95': (ate - 1.96 * se, ate + 1.96 * se)}

dml = DoubleML()
result = dml.estimate(severity.reshape(-1, 1), treatment, recovery)
print(f"ATE: {result['ate']:+.3f}  "
      f"(95% CI: {result['ci_95'][0]:+.3f}, {result['ci_95'][1]:+.3f})")

Read the three steps slowly, because they are the whole idea. You predict the outcome from the confounders and keep the leftover (the part the confounders do not explain). You predict the treatment from the confounders and keep its leftover. Then you relate the two leftovers -- and because you have already scrubbed out everything the confounders could account for, whatever is left is the causal signal. The forementioned cross-fitting (predicting each point using models trained on the other folds) is what stops the flexible ML models from overfitting their own residuals. The result is almost magical: you get the raw predictive power of gradient boosting or deep nets in the nuisance steps, yet a final estimate that stays statistically honest, complete with a confidence interval you can actually defend. This is the technique that quietly powers a lot of modern experimentation platforms -- it is the bridge between the ML world of episodes past and the causal world of today.

Where causal thinking sharpens real ML systems

None of this is econometrics trivia locked away from practitioners. Causal reasoning improves the ML systems you already build -- often by preventing a mistake rather than adding a feature:

causal_payoffs = {
    'recommendation': (
        "Naive: 'users who bought X also bought Y' (pure correlation). "
        "Causal: 'does SHOWING X actually cause buying Y?'. "
        "The gap: popularity bias -- best-sellers correlate with everything."
    ),
    'feature_selection': (
        "Naive: keep features highly correlated with the target. "
        "Causal: keep CAUSES of the target, not its EFFECTS. "
        "The gap: an effect of the target is leakage (episode #14)."
    ),
    'model_debugging': (
        "Naive: feature importance shows what the model leans on. "
        "Causal: causal analysis shows what SHOULD matter. "
        "The gap: spurious correlations in training data mislead the model."
    ),
    'ab_testing': (
        "A/B tests: randomization GIVES you causality for free. "
        "Causal methods: recover the effect when you CANNOT randomize. "
        "Same goal, different tools -- reach for the test when you can."
    ),
}
for area, story in causal_payoffs.items():
    print(f"[{area}]\n  {story}\n")

That feature_selection row is the one I want you to tattoo somewhere visible. Blindly keeping every feature that correlates with the target is how leakage sneaks into production -- you accidentally include a variable that is a consequence of the outcome (say, "number of retention calls received") and your offline metrics look glorious right up until the model faces the real world, where that feature is not available yet or means something different. Causal thinking is the discipline that asks, for every feature, "is this a cause of what I am predicting, or a symptom of it?" -- and that single question has saved more production models than any amount of hyperparameter tuning.

And the honest caveat, because I promised you honesty: causal inference is not free lunch. Every method here rests on the assumption that you have measured the confounders that matter (the "no unmeasured confounding" assumption), and that assumption is untestable from the data alone. When you can randomize -- a proper A/B test -- do that instead, because randomization severs the confounding at the source and hands you causality with no assumptions. Causal inference is what you reach for when randomization is impossible, unethical, or already too late: observational data is all you have, and you would rather reason carefully than pretend correlation was causation all along.

TL;DR - from predicting to intervening

  • Correlation is not causation, and it is not a slogan -- Simpson's Paradox shows a genuinely helpful treatment looking harmful in pooled data, and a naive model reading that coefficient recommends the exact opposite of what works;
  • a confounder is a common cause of both treatment and outcome; it contaminates naive comparisons, and the whole game is to adjust it away;
  • causal graphs (DAGs) make your assumptions explicit and reveal which variables to control for; the do-operator P(Y | do(T)) is the interventional question decision-makers actually care about, and it is NOT the observational P(Y | T);
  • propensity score matching collapses many confounders into one score and pairs similar treated/untreated units -- practical, intuitive, and only as good as the confounders you measured;
  • Double Machine Learning lets you use any flexible ML model for the nuisance steps while staying unbiased, via residualization plus cross-fitting -- ML muscle with causal honesty;
  • causal thinking sharpens everyday ML: it kills feature leakage, exposes popularity bias in recommenders, and answers "what should we do?" instead of merely "what will happen?" -- and when you can randomize, an A/B test beats all of it.

Exercises

Three to chew on before next time -- and as always I'll walk through full solutions in the following episode.

  1. Break the naive analysis yourself. Regenerate the Simpson's Paradox dataset above but crank the confounding up: make the sickest patients far more likely to be treated (change treatment_prob to 0.1 + 0.8 * severity). Refit both the naive and the severity-adjusted logistic regressions and report the two coefficients. Does the naive coefficient get more misleading as the confounding strengthens? Write one sentence explaining why.

  2. Match, then check your matches. Using the PropensityScoreMatching class, after fitting the propensity scores, plot (or just print summary statistics of) the score distributions for the treated and control groups separately. Then estimate the ATT with n_neighbors=1 and again with n_neighbors=5. Do the estimates differ? Explain in two sentences what "poor overlap" between the two score distributions would mean for whether matching is even trustworthy here.

  3. Cause or effect? For a churn-prediction model, you have these candidate features: (a) account age, (b) number of support tickets filed last month, (c) whether the customer's subscription was cancelled, (d) monthly spend, (e) number of retention calls the company made to them. For each, decide whether it is a plausible cause of churn, an effect of (likely) churning, or genuine leakage that must be dropped -- and justify each in one line. There is no single perfect answer; the reasoning is the whole point.

We spent this whole episode drawing little graphs -- severity pointing at treatment, treatment pointing at recovery -- and treating the graph as a map we reason over by hand. But notice the quiet question sitting underneath: if a graph is such a natural way to encode how things relate, could a neural network learn to operate on graph-structured data directly, passing information along the edges instead of forcing everything into a flat feature vector? Molecules, social networks, road maps, even the causal DAGs from today -- all of them are graphs first and tables second. That is precisely where we are headed next ;-)

Bedankt en tot de volgende keer!

@scipio

stem stemsocial steemstem python programming
0
0
0.000
1 comments
avatar

Congratulations @scipio! You have completed the following achievement on the Hive blockchain And have been rewarded with New badge(s)

You distributed more than 25000 upvotes.
Your next target is to reach 26000 upvotes.

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

0
0
0.000
Reply