Learn AI Series (#128) - Privacy-Preserving AI

avatar

Learn AI Series (#128) - Privacy-Preserving AI

variant-a-13-lime.png

What will I learn

  • You will learn differential privacy (DP) -- the one idea in this whole area that hands you an actual mathematical guarantee instead of a hopeful shrug, and the epsilon knob that measures exactly how strong that guarantee is;
  • DP-SGD, the recipe that turns ordinary training into private training with two small moves (clip, then add noise), and how Opacus does it for you in about four lines;
  • federated learning -- the genuinely different idea of bringing the model to the data instead of hoarding everyone's data in one big tempting pile (this is how your phone keyboard learns without shipping your messages anywhere);
  • secure multi-party computation (SMPC), where several parties compute a joint answer while each keeps its own inputs secret -- with a tiny secret-sharing scheme you can actually read;
  • homomorphic encryption, the almost-magical trick of computing directly on encrypted data without ever decrypting it (and the sobering reason it is not everywhere yet);
  • and how to budget privacy -- because every private query spends something you can never get back.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • An installed Python 3(.10+) distribution with PyTorch (pip install torch), and pip install opacus if you want to run the differentially private training for real -- but as usual you do NOT need a GPU, every idea here is easiest to understand on a small model you could train on a laptop;
  • You've been through episode #35 (data ethics and bias, where we first admitted data is people), and especially episode #127 (AI security) -- today is the constructive sequel to the membership-inference and model-inversion attacks we met there.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#128) - Privacy-Preserving AI

I ended episode #127 on a deliberately uncomfortable note. We had just watched a model quietly betray the people it was trained on -- membership inference figuring out whether your record was in the training set, model inversion clawing back blurry-but-recognisable faces -- and I asked the obvious follow-up question: could we train useful models without ever piling everyone's raw data into one big, tempting heap in the first place? Today is the constructive answer to that question. We stop attacking and start defending, and this time the defenses come with receipts ;-)

Here is the framing I want you to carry through the whole episode. Every technique below is chasing the same goal from a different angle: extract the pattern from the data without leaking the individuals in it. Hospitals want a model trained on patient records from a dozen institutions, but no hospital may ship a single row to another. Banks want to spot fraud rings that span competitors, without handing rivals their customer lists. Your phone wants a keyboard that learns how you type, without your late-night messages ever leaving the device. In every case the data cannot move, for reasons that are legal, competitive or plain ethical. Privacy-preserving AI is the toolbox that makes learning happen anyway -- and the important thing, the thing that separates it from a corporate pinky-promise, is that it leans on mathematics and cryptography rather than trust. Nobody has to believe you won't peek; the math makes peeking provably useless.

Solutions to episode #127's exercises

As promised, let's clear last time's homework before we build anything new.

Exercise 1 -- break your own model with FGSM. The point was to sweep the perturbation budget and find where the label flips:

import torch
import torch.nn as nn

def epsilon_sweep(model, image, true_label):
    """Run FGSM at increasing epsilon and watch the prediction crumble."""
    for epsilon in [0.0, 0.01, 0.03, 0.1, 0.3]:
        img = image.clone().detach()
        img.requires_grad = True
        loss = nn.functional.cross_entropy(model(img), true_label)
        model.zero_grad()
        loss.backward()

        adv = torch.clamp(img + epsilon * img.grad.sign(), 0, 1)
        with torch.no_grad():
            probs = torch.softmax(model(adv), dim=1)
            pred = probs.argmax(dim=1).item()
            conf = probs[0, pred].item()
        flipped = "FLIPPED" if pred != true_label.item() else "ok"
        print(f"eps={epsilon:>4}: pred={pred} conf={conf:.2f} [{flipped}]")

Almost everyone finds the label survives epsilon=0.0 and 0.01, wobbles around 0.03, and is comprehensively wrong by 0.1 -- while your own eyes still see the original picture. That gap between "the model is fooled" and "I can see nothing wrong" is the whole reason episode #127 called adversarial examples a property of high-dimensional geometry, not a bug.

Exercise 2 -- watch a model betray its training set. The membership signal is the confidence gap between data the model has seen and data it hasn't:

def membership_gap(model, train_batch, test_batch):
    """Average confidence on training data minus on unseen data."""
    model.eval()
    def mean_conf(x, y):
        with torch.no_grad():
            probs = torch.softmax(model(x), dim=1)
        return probs.gather(1, y.unsqueeze(1)).mean().item()
    seen = mean_conf(*train_batch)
    unseen = mean_conf(*test_batch)
    print(f"seen={seen:.3f}  unseen={unseen:.3f}  gap={seen - unseen:.3f}")
    return seen - unseen

The overfit model (tiny dataset, no regularisation, too many epochs) shows a fat gap -- it is far too confident on data it memorised. The well-regularised model shows a thin one. That is the punchline I wanted you to feel in your bones: regularisation, which back in episode #40 was just an accuracy trick, is quietly also a privacy defense. Hold onto that thought, because differential privacy is about to make it rigorous.

Exercise 3 -- design the defense stack. There was no single right answer here, only defensible reasoning. A public paid image API: top threats are model stealing (rate-limit plus top-k outputs) and adversarial inputs (input sanitisation, maybe adversarial training). An internal LLM assistant that reads documents and sends email: indirect prompt injection (sandbox the email tool behind a human/permission check) and data exfiltration (output filtering). A diagnostic model on hospital records: membership inference / inversion (which is precisely today's episode) and data poisoning (provenance and vetting). If you justified why each defense matched that system's biggest risk, you nailed it.

Differential privacy: a guarantee, not a vibe

Right, on to the star of the show. Every defense in episode #127 was a heuristic -- helpful, sensible, but ultimately a "trust me, this usually helps". Differential privacy is a different species entirely. It makes a formal promise: the output of a computation should be essentially the same whether or not any single individual's data was included. If one person's presence cannot meaningfully move the result, then no downstream attack can reliably pull that person back out. The strength of the promise is dialled by a single number, epsilon:

import numpy as np

def explain_epsilon():
    """What epsilon means, in plain terms."""
    return {
        'epsilon=0.1': 'Very strong: essentially impossible to single anyone out',
        'epsilon=1.0': 'Strong: individual influence bounded by a factor ~2.7',
        'epsilon=10':  'Weak: some privacy, but detectable in principle',
        'epsilon=100': 'Essentially no guarantee at all',
    }
    # Formal statement -- for ANY output O and any two datasets D1, D2
    # that differ by exactly one record:
    #     P(mechanism(D1) = O)  <=  e**epsilon * P(mechanism(D2) = O)

Read that formal line in the comment slowly, because it is the entire idea compressed into one inequality. It says: swapping one person in or out can change the probability of any given output by at most a factor of e**epsilon. Small epsilon means that factor is close to 1, which means the two worlds -- you-in, you-out -- are nearly indistinguishable. Large epsilon means the worlds are easy to tell apart, which means your presence shows. Smaller epsilon is stronger privacy. That is the one sentence to tattoo on the inside of your eyelids.

So how do you actually achieve it? By adding carefully calibrated noise, scaled to something called sensitivity -- the most that any one individual could possibly change the answer:

class DPMechanism:
    """The two classic ways to privatise a numeric answer."""

    @staticmethod
    def laplace_mechanism(true_value, sensitivity, epsilon):
        """Laplace noise -- the textbook mechanism for pure epsilon-DP."""
        scale = sensitivity / epsilon
        return true_value + np.random.laplace(0, scale)

    @staticmethod
    def gaussian_mechanism(true_value, sensitivity, epsilon, delta=1e-5):
        """Gaussian noise -- the workhorse for machine learning."""
        sigma = sensitivity * np.sqrt(2 * np.log(1.25 / delta)) / epsilon
        return true_value + np.random.normal(0, sigma)

# Releasing the average age of 1000 patients, privately.
true_avg = 45.3
sensitivity = 100 / 1000     # ages capped at 100, so one person shifts the mean by <= 100/n
private_avg = DPMechanism.gaussian_mechanism(true_avg, sensitivity, epsilon=1.0)
print(f"True average: {true_avg}   Private average: {private_avg:.1f}")

Notice the shape of the trade-off baked into scale = sensitivity / epsilon. Want stronger privacy (smaller epsilon)? The scale goes UP, so you inject more noise, so your answer gets fuzzier. Want a crisper answer? You spend more epsilon and protect people less. There is no free lunch here, only a dial -- and the entire craft of differential privacy is choosing where to set that dial for the data in front of you. For a mean of n values bounded in [0, 100] the sensitivity is a tidy 100/n. For the gradient of a neural network it is not tidy at all, which is exactly the problem the next section solves.

DP-SGD: private training in two moves

Here is the lovely part. To make ordinary neural-network training differentially private, you do NOT need to rederive backpropagation or invent a new optimiser. You bolt two small moves onto the training loop you have used since episode #41: clip each example's gradient so no single person can shove the model too hard, then add noise to the aggregate so their fingerprint blurs away.

class DPSGD:
    """Differentially Private SGD, spelled out the slow, readable way."""
    def __init__(self, model, lr=0.01, max_grad_norm=1.0, noise_multiplier=1.1):
        self.model = model
        self.lr = lr
        self.max_grad_norm = max_grad_norm          # the clipping bound
        self.noise_multiplier = noise_multiplier    # how much Gaussian noise

    def train_step(self, batch_x, batch_y):
        per_example = self._per_example_grads(batch_x, batch_y)

        # MOVE 1: clip every example's gradient to bound its influence
        clipped = []
        for grads in per_example:
            total_norm = torch.sqrt(sum((g ** 2).sum() for g in grads))
            factor = min(1.0, self.max_grad_norm / (total_norm + 1e-6))
            clipped.append([g * factor for g in grads])

        n = len(clipped)
        avg = [sum(clipped[i][j] for i in range(n)) / n
               for j in range(len(clipped[0]))]

        # MOVE 2: add calibrated Gaussian noise to the averaged gradient
        for param, grad in zip(self.model.parameters(), avg):
            noise_std = self.noise_multiplier * self.max_grad_norm / n
            noisy = grad + torch.randn_like(grad) * noise_std
            param.data -= self.lr * noisy

    def _per_example_grads(self, batch_x, batch_y):
        out = []
        for x, y in zip(batch_x, batch_y):
            self.model.zero_grad()
            loss = nn.functional.cross_entropy(self.model(x.unsqueeze(0)),
                                               y.unsqueeze(0))
            loss.backward()
            out.append([p.grad.clone() for p in self.model.parameters()])
        return out

The two moves earn their keep together, and it is worth being precise about why. Clipping guarantees a bounded sensitivity: after clipping, no single training example can move the gradient by more than max_grad_norm, full stop. That is what makes one person's influence finite instead of unlimited. Then the noise, scaled to that same bound, hides whether any particular person was in the batch at all. Take away either move and the guarantee collapses -- noise without clipping is noise scaled to infinity, and clipping without noise still lets a determined attacker average away the clipping.

Now, that per-example loop above is honest but slow (it runs one backward pass per sample, which for a batch of 256 is 256 backward passes). Nobody trains like that in production. In real life you reach for Opacus, PyTorch's DP library, which computes per-example gradients efficiently with hooks and tracks your spent epsilon for you:

from opacus import PrivacyEngine

privacy_engine = PrivacyEngine()
model, optimizer, data_loader = privacy_engine.make_private(
    module=model,
    optimizer=optimizer,
    data_loader=data_loader,
    noise_multiplier=1.1,
    max_grad_norm=1.0,
)

# ... your ORDINARY training loop, unchanged ...

# and afterwards, ask how much privacy you actually spent:
epsilon = privacy_engine.get_epsilon(delta=1e-5)
print(f"Trained with (epsilon={epsilon:.2f}, delta=1e-5)")

Four lines wrap your model, your optimiser and your data loader, and the training loop itself does not change one character. That is the bit that always makes people sit up: private training is not a different algorithm, it is your algorithm wearing a seatbelt. The bill, in practice, is roughly 5-15% accuracy versus non-private training, depending on the model and how tight you set epsilon. For cat photos, do not bother. For genomes, that 15% is the difference between a responsible system and a headline.

Federated learning: bring the model to the data

Differential privacy fixes what the model memorises. But look back at the hospital and the phone from the intro -- their problem starts one step earlier. They cannot even gather the data in one place. So flip the whole pipeline around: instead of bringing the data to the model, bring the model to the data. Each participant trains locally on data that never leaves their machine, and only the model update travels back. This is federated learning, and its simplest recipe is FedAvg -- literally averaging everyone's updated weights:

class FederatedServer:
    """Coordinates training without ever seeing a raw data point."""
    def __init__(self, model_fn):
        self.global_model = model_fn()

    def aggregate(self, client_updates):
        """FedAvg: average the client weight dictionaries."""
        global_dict = self.global_model.state_dict()
        for key in global_dict:
            global_dict[key] = torch.stack(
                [update[key] for update in client_updates]
            ).mean(dim=0)
        self.global_model.load_state_dict(global_dict)
        return global_dict


class FederatedClient:
    """Trains on private local data, ships back ONLY the weights."""
    def __init__(self, model_fn, local_data, local_labels):
        self.model = model_fn()
        self.data, self.labels = local_data, local_labels

    def local_train(self, global_weights, epochs=5, lr=0.01):
        self.model.load_state_dict(global_weights)
        opt = torch.optim.SGD(self.model.parameters(), lr=lr)
        for _ in range(epochs):
            loss = nn.functional.cross_entropy(self.model(self.data), self.labels)
            opt.zero_grad()
            loss.backward()
            opt.step()
        return self.model.state_dict()   # raw data stays home


def federated_training(server, clients, rounds=50):
    for r in range(rounds):
        weights = server.global_model.state_dict()
        updates = [c.local_train(weights) for c in clients]
        server.aggregate(updates)
        print(f"Round {r}: aggregated {len(clients)} client updates")

This is not a toy from a textbook -- it is how Gboard, Google's Android keyboard, learns next-word prediction across a billion-odd phones without your typing ever being uploaded, and how Apple nudges bits of Siri. The model visits your device, learns a little from your data locally, and only the update goes home.

Having said that, do NOT walk away thinking federated learning is privacy by itself. It has real teeth-grinding challenges. Client data is non-IID (your phone's data looks nothing like mine, so naive averaging can wobble). Communication is expensive (shipping a full set of weights every round eats bandwidth). Stragglers -- one slow phone on a train tunnel -- hold up the whole round. And here is the sting: those innocent-looking weight updates can still leak information about the local data that produced them (there are gradient-inversion attacks that reconstruct inputs from updates alone). The grown-up move is to combine federated learning with the differential privacy from two sections ago, so each update is clipped and noised before it ever leaves the device. Belt AND braces.

Secure multi-party computation

Federated learning still trusts a central server to do the averaging. What if you do not want to trust anyone -- not even the aggregator? Enter secure multi-party computation (SMPC): several parties jointly compute a function while each keeps its own input completely secret. The gateway idea is additive secret sharing. You split your private number into random-looking shares that sum back to it, hand one share to each party, and no single share reveals a thing:

class SimpleSecretSharing:
    """Additive secret sharing -- the 'hello world' of SMPC."""
    def __init__(self, n_parties):
        self.n_parties = n_parties

    def share(self, value):
        """Split a value into n shares that look like noise individually."""
        shares = [np.random.uniform(-1000, 1000) for _ in range(self.n_parties - 1)]
        shares.append(value - sum(shares))    # last share makes the sum come out right
        return shares

    def reconstruct(self, shares):
        return sum(shares)

    def private_sum(self, values_per_party):
        """Compute the SUM of everyone's value without revealing any single one."""
        all_shares = [self.share(v) for v in values_per_party]
        # Each party adds up the shares it was handed...
        party_totals = [sum(all_shares[j][i] for j in range(self.n_parties))
                        for i in range(self.n_parties)]
        # ...and only the aggregated totals get combined.
        return self.reconstruct(party_totals)

# Three hospitals compute the combined average patient age,
# and NONE of them learns another's number.
sharing = SimpleSecretSharing(3)
hospital_means = [45.2, 52.1, 48.7]
total = sharing.private_sum(hospital_means)
print(f"Average across hospitals: {total / 3:.1f}")

The magic is that a single share is just a uniform random number -- it tells you nothing about the secret. Only when the shares are combined does the real answer reappear. This little example only does addition, mind you. Real protocols (SPDZ, ABY3 and friends) handle multiplication, comparison and, in principle, arbitrary computation -- but the honesty tax is steep: SMPC typically runs 100 to 1000 times slower than plaintext, because every multiplication turns into a flurry of network messages between the parties. Beautiful, rigorous, and something you reach for only when the trust problem genuinely demands it.

Homomorphic encryption: computing in the dark

If SMPC felt like sorcery, homomorphic encryption is the part where you start checking whether the universe is allowed to work this way. The promise: compute directly on encrypted data, never decrypting, and get back an encrypted result that -- once you and only you decrypt it -- is exactly the answer you wanted. Send your encrypted numbers to an untrusted cloud, let it crunch, get encrypted output back, and the cloud learns nothing:

# Conceptual toy -- real HE uses lattice libraries like SEAL, TFHE, OpenFHE.
class ConceptualHE:
    """Illustrates the ONE property that matters: additive homomorphism."""
    def __init__(self, key=42):
        self.key = key

    def encrypt(self, value):
        return value + self.key      # real HE: hard lattice problems, not addition

    def decrypt(self, ciphertext):
        return ciphertext - self.key

    def add_encrypted(self, ct1, ct2):
        # The whole point: we add WITHOUT knowing the key or the plaintexts.
        # decrypt(ct1 + ct2) == decrypt(ct1) + decrypt(ct2)
        return ct1 + ct2

# What it looks like with a real library (TenSEAL, CKKS scheme):
# import tenseal as ts
# ctx = ts.context(ts.SCHEME_TYPE.CKKS, poly_modulus_degree=8192, ...)
# enc = ts.ckks_vector(ctx, [1.0, 2.0, 3.0])
# enc_result = enc + enc            # arithmetic on ciphertext
# print(enc_result.decrypt())       # -> [2.0, 4.0, 6.0]

Fully Homomorphic Encryption (FHE) goes all the way -- both addition and multiplication on ciphertext, which is enough to compute anything in principle. So why is your bank not doing everything this way? One word: cost. FHE runs somewhere between 10,000 and 1,000,000 times slower than plaintext, which makes training a network on encrypted data a non-starter today. Where it starts to make sense is inference on a small model in a genuinely high-stakes setting -- a medical diagnosis, a financial computation -- where the privacy is worth paying a fortune in compute for. It is the most theoretically powerful tool in the box and the least practically used, and it is worth knowing that tension exists.

Budgeting privacy, because it runs out

One last idea, and it is the one people forget until it bites them. Privacy is a consumable. Every differentially private query you answer spends a little epsilon, and once it is gone, it is gone -- you cannot answer more questions without weakening every earlier answer. So you keep an accountant:

class PrivacyAccountant:
    """Track cumulative privacy expenditure against a fixed budget."""
    def __init__(self, total_epsilon=10.0):
        self.total_epsilon = total_epsilon
        self.spent = 0.0
        self.log = []

    def record_query(self, query_epsilon, description):
        if self.spent + query_epsilon > self.total_epsilon:
            raise ValueError(
                f"Budget exceeded: spent {self.spent:.2f}, "
                f"requested {query_epsilon:.2f}, limit {self.total_epsilon}"
            )
        self.spent += query_epsilon
        self.log.append((description, query_epsilon, self.spent))
        remaining = self.total_epsilon - self.spent
        print(f"{description}: eps={query_epsilon:.2f}  remaining={remaining:.2f}")

acct = PrivacyAccountant(total_epsilon=10.0)
acct.record_query(2.0, "Train model v1")
acct.record_query(1.0, "Release aggregate statistics")
acct.record_query(3.0, "Train model v2")
# 4.0 epsilon left in the tank.

The reason this matters is the composition theorem: privacy degrades every time you use the data, and if you naively add up the epsilons you run out fast. The clever bit -- and the reason Opacus can train for many epochs without blowing the budget -- is advanced composition (the "moments accountant"), which tracks the spend far more tightly than plain addition and lets you squeeze many more queries out of the same total epsilon. Nota bene: this is a governance question as much as a maths one. Somebody has to own the budget and decide what it gets spent on, exactly like a team owns a cloud bill.

TL;DR - defending the people inside the data

  • Differential privacy trades a heuristic for a mathematical guarantee: the epsilon knob bounds how much any single individual can affect the output -- smaller epsilon, stronger privacy, more noise, fuzzier answers;
  • DP-SGD makes training private with just two moves, clip per-example gradients (bound one person's influence) then add calibrated noise (blur the fingerprint) -- and Opacus does it in about four lines while tracking your spent epsilon for you;
  • federated learning brings the model to the data instead of hoarding the data (this is how phone keyboards learn) -- but weight updates can still leak, so pair it with DP;
  • secure multi-party computation lets parties compute a joint answer while each keeps its input secret, via secret sharing, at roughly 100-1000x the cost of plaintext;
  • homomorphic encryption computes on encrypted data without decrypting -- theoretically able to compute anything, but 10,000x+ slower, so realistic only for small-model inference in high-stakes cases;
  • privacy is a budget: every private query spends epsilon you never get back, so account for it (advanced composition tracks the spend tightly enough to actually train with).

Exercises

Three to chew on before next time -- and as always I'll walk through full solutions in the following episode.

  1. Feel the privacy-utility trade-off. Take the gaussian_mechanism above and privately release the mean of a synthetic dataset of 1000 values in [0, 100]. For each epsilon in [0.1, 0.5, 1.0, 5.0, 10.0], run the mechanism 200 times and record the average absolute error versus the true mean. Plot error against epsilon (or just print the table) and write one sentence describing the shape of the curve you see.

  2. Train something for real with Opacus. Install Opacus, train a small classifier on MNIST twice -- once normally, once wrapped with make_private at noise_multiplier=1.1, max_grad_norm=1.0. Report both test accuracies AND the final epsilon from get_epsilon(delta=1e-5). Then re-run the private version at a larger noise multiplier and describe what happens to both the accuracy and the epsilon.

  3. Reason about the right tool. For each scenario, pick ONE technique from this episode (DP, federated learning, SMPC, or HE) and justify it in two sentences: (a) three banks want a shared fraud model but will not share customer data with each other or a central server; (b) a research team wants to publish aggregate statistics from a sensitive survey; (c) a cloud provider offers to run inference for a hospital but must never see the patient scans. There is no single correct mapping -- the reasoning is the point.

We now have a model that can learn from data without memorising the individuals inside it, spread across machines that never share a raw row, on inputs it may never even see in the clear. That is a serious amount of engineering. Which raises a slightly mischievous question for next time: we have spent this whole series making humans do the hard design work -- choosing architectures, tuning knobs, deciding what to try. What if we handed that job to the machine as well, and let it search for the best model on its own? That is where we head next ;-)

Bedankt en tot de volgende keer!

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments