Learn AI Series (#147) - AI Safety and Alignment

avatar

Learn AI Series (#147) - AI Safety and Alignment

variant-a-06-magenta.png

What will I learn

  • The alignment problem -- why getting an AI to do what you actually MEANT is a much nastier problem than getting it to do what you literally SAID, and why the gap only widens as the model gets smarter;
  • reward hacking and Goodhart's law -- how a system happily maxes out your proxy metric while quietly abandoning the goal the metric was standing in for, and why this is not a bug but a property of optimization;
  • the KL leash -- the one-line trick that keeps RLHF from over-cooking a model into confident nonsense;
  • scalable oversight -- how on earth you supervise a system that is faster, cheaper and (soon) smarter than the human doing the supervising, via debate and recursive reward modeling;
  • Constitutional AI -- teaching a model to critique itself against written principles instead of drowning humans in preference labels;
  • red teaming -- attacking your own system on purpose, by hand and automatically, to find the failures before your users (or a regulator) do;
  • the governance landscape -- the EU AI Act, risk tiers, and what "compliant" actually asks of you as a builder.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • Python 3.10+ with PyTorch installed (pip install torch) -- every snippet here runs on a plain CPU in seconds, no GPU anywhere;
  • You have read #61 (instruction tuning and RLHF) because this episode is the sequel to it, and #127 (AI security) because attacks and safety are cousins. It also REALLY helps to remember the reinforcement learning block (#102-116), since half of alignment is just RL wearing a serious face.

Difficulty

  • Beginner

Curriculum (of the Learn AI Series):

Learn AI Series (#147) - AI Safety and Alignment

I ended #146 dangling a thread on purpose, so let me grab it before we do anything else. We spent that whole episode learning to see WHY a model decides what it decides -- SHAP, integrated gradients, TCAV, cracking open the black box. And I closed by asking the uncomfortable follow-up: what happens when you finally look inside and you do NOT like what you find? When the model is optimising for something subtly, dangerously different from what you meant? Understanding a system and CONTROLLING a system are two very different problems, and the gap between them is exactly where we stand today. Welcome to AI safety and alignment ;-)

Before the fear-mongering starts, one thing straight: this is not a lecture about a future killer robot. Alignment is a present-day ENGINEERING problem. Today's models already fabricate facts with total confidence, already get talked out of their own safety training by a clever prompt, and already game proxy objectives in ways that make their creators wince. If you build with this stuff, alignment is your problem, not somebody's philosophy seminar.

Solutions to episode #146's exercises

House rules, same as every week -- we settle last time's homework before opening anything new. #146 was interpretability, and all three tasks were about not trusting an explanation just because it is pretty.

Exercise 1 -- Compare integrated gradients to a raw gradient. Build a tiny regressor, pick one input, and compute both a plain single-point gradient and the integrated-gradients attribution from a zero baseline. Print them side by side.

import torch
import torch.nn as nn

torch.manual_seed(7)
model = nn.Sequential(nn.Linear(4, 16), nn.ReLU(), nn.Linear(16, 1))
x = torch.tensor([2.0, -1.0, 0.5, 3.0])

# raw single-point gradient
xp = x.clone().requires_grad_(True)
model(xp.unsqueeze(0)).backward()
raw = xp.grad.clone()

# integrated gradients from a zero baseline
baseline = torch.zeros_like(x)
steps = 64
ig = torch.zeros_like(x)
for k in range(1, steps + 1):
    point = (baseline + (k / steps) * (x - baseline)).clone().requires_grad_(True)
    model(point.unsqueeze(0)).backward()
    ig += point.grad.detach()
ig = (x - baseline) * ig / steps

for i in range(4):
    print(f"feature {i}: raw grad {raw[i]:+.4f}   IG {ig[i]:+.4f}")
gap = (model(x.unsqueeze(0)) - model(baseline.unsqueeze(0))).item()
print(f"IG sums to {ig.sum():+.4f}, output-baseline = {gap:+.4f}")

The two columns disagree whenever a feature has pushed a ReLU into its flat, saturated region: the raw gradient THERE reads near zero (locally nothing moves), even though dragging that feature from baseline up to its real value is what set the output in the first place -- integrated gradients catches it because it averages the gradient along the entire path, and the proof is that its attributions sum to output-minus-baseline while the raw gradients sum to nothing meaningful.

Exercise 2 -- Feel LIME's instability. Run the local-surrogate explanation on the SAME input three times without fixing a seed, and print the top-3 features each run.

import torch
import torch.nn as nn

torch.manual_seed(0)                      # this fixes the MODEL, not the explainer
model = nn.Sequential(nn.Linear(6, 24), nn.ReLU(), nn.Linear(24, 1))
x = torch.randn(6)

def lime_top3(model, x, n_samples=800):
    n = x.shape[0]
    masks = torch.bernoulli(torch.full((n_samples, n), 0.5))   # fresh RNG each call
    perturbed = torch.zeros(n_samples, n)
    for i in range(n_samples):
        perturbed[i] = torch.where(masks[i].bool(), x, torch.zeros_like(x))
    with torch.no_grad():
        y = model(perturbed).squeeze()
    dist = (masks - 1).pow(2).sum(1).sqrt()
    w = torch.exp(-dist / (n * 0.25))
    coefs = torch.linalg.lstsq(masks * w.unsqueeze(1), y * w).solution
    return coefs.abs().topk(3).indices.tolist()

for run in range(3):
    print(f"run {run + 1} top-3 features: {lime_top3(model, x)}")

The dominant feature usually holds its place, but the second and third slots tend to shuffle between runs. Two sentences, as asked: this is poison for a compliance report, because the exact same decision now produces a DIFFERENT official "reason" every time you reseed the random number generator, and no auditor accepts "the explanation depends on our lottery." SHAP sidesteps it entirely -- Shapley values are unique by construction, so there is nothing left to reseed.

Exercise 3 -- Fake a spurious correlation and catch it. Build a 2D classifier whose label secretly depends only on feature 0, but engineer feature 1 to correlate with the label in the training sample, then attribute a test point where the correlation is BROKEN.

import torch
import torch.nn as nn

torch.manual_seed(1)
N = 400
label = (torch.rand(N) > 0.5).float()      # truth depends on feature 0 only
f0 = label + 0.1 * torch.randn(N)          # feature 0 carries the real signal
f1 = label + 0.1 * torch.randn(N)          # feature 1 correlates ONLY in this sample
X = torch.stack([f0, f1], dim=1)

clf = nn.Sequential(nn.Linear(2, 16), nn.ReLU(), nn.Linear(16, 1))
opt = torch.optim.Adam(clf.parameters(), lr=0.05)
for _ in range(300):
    loss = nn.functional.binary_cross_entropy_with_logits(clf(X).squeeze(), label)
    opt.zero_grad(); loss.backward(); opt.step()

# test point where the correlation is BROKEN: real signal says 1, feature 1 lies
probe = torch.tensor([1.0, 0.0])
baseline = torch.zeros(2)
ig = torch.zeros(2)
for k in range(1, 65):
    p = (baseline + (k / 64) * (probe - baseline)).clone().requires_grad_(True)
    clf(p.unsqueeze(0)).backward()
    ig += p.grad.detach()
ig = (probe - baseline) * ig / 64
print(f"attribution feature 0 (real):      {ig[0]:+.4f}")
print(f"attribution feature 1 (spurious):  {ig[1]:+.4f}")

If the network leaned on feature 1 (and with two equally-predictive columns in training it often splits its trust between them), the broken probe drags a big negative attribution onto feature 1 -- the model is being pulled the wrong way by a column that MEANS nothing. That is precisely the debugging the black box denied you: test-set accuracy on the correlated data looked flawless, and only per-feature attribution exposes that the model was quietly reading the wrong signal. Right -- homework settled. Now let us talk about what happens when the thing you are optimising is itself the problem ;-)

The alignment problem

Alignment means one deceptively small thing: making an AI system do what we actually WANT. Not what we literally typed, not what we technically rewarded -- what we meant.

Sounds trivial. It is one of the hardest problems in the field.

The whole difficulty lives in a single asymmetry: it is easy to specify a MEASURABLE objective and brutally hard to specify an INTENDED one. "Maximise user time on site" is measurable. "Make users glad they spent that time" is what you meant. Those two come apart the moment a capable optimizer gets its hands on them, and it will always optimise the one you can measure while cheerfully ignoring the one you wanted. You saw the seed of this all the way back in the reinforcement learning block (#102-116): a reward function is a FORMAL specification, and formal specifications have edge cases, loopholes and interpretations you never intended. The stronger the optimizer, the more ruthlessly it finds those gaps.

Having said that, this is not a reason to despair -- it is a reason to design carefully. The rest of the episode is really a tour of the tools people have built to shrink that gap.

Reward hacking and Goodhart's law

There is an old bit of wisdom economists call Goodhart's law: "when a measure becomes a target, it ceases to be a good measure." In machine learning we call the same thing reward hacking, and it is not an occasional glitch -- it is the default behaviour of optimizing against an imperfect objective.

The canonical picture, and one you can build yourself, is reward-model OVEROPTIMIZATION. In RLHF (#61) you do not have the true human objective; you have a learned reward model that approximates it from a finite pile of preference labels. Push your policy gently against that model and true quality improves. Push HARD against it and true quality peaks, then falls off a cliff, while the proxy reward keeps merrily climbing. Past the peak you are no longer producing better outputs -- you are producing adversarial examples against your own reward model.

import torch
import torch.nn as nn


class RewardModel(nn.Module):
    """A learned reward model, exactly as used in RLHF (#61)."""
    def __init__(self, input_dim=32):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, 64), nn.ReLU(),
            nn.Linear(64, 1),
        )

    def forward(self, x):
        return self.net(x)


def true_quality(x):
    """The REAL objective the reward model only approximates.
    The reward model was trained to imitate this on a limited sample."""
    return -(x ** 2).mean(dim=-1, keepdim=True)   # peaks when x is near zero

That gives us the two objectives: the true_quality we actually care about (highest when the output vector sits near the origin) and a RewardModel that has only ever SEEN examples near that region. Now watch what over-optimizing the proxy does to the truth.

torch.manual_seed(0)
rm = RewardModel()

# pretend the reward model learned true_quality well NEAR the data (small x)
data = torch.randn(512, 32) * 0.5
opt = torch.optim.Adam(rm.parameters(), lr=1e-2)
for _ in range(300):
    loss = ((rm(data) - true_quality(data)) ** 2).mean()
    opt.zero_grad(); loss.backward(); opt.step()

# now OPTIMIZE an output against the reward model, harder and harder
x = torch.zeros(1, 32, requires_grad=True)
inner = torch.optim.Adam([x], lr=0.05)
for step in range(1, 121):
    reward = rm(x)
    inner.zero_grad(); (-reward).backward(); inner.step()
    if step % 30 == 0:
        print(f"step {step:3d}  proxy reward {rm(x).item():+.3f}  "
              f"TRUE quality {true_quality(x).item():+.3f}")

Run it and you see the tell-tale divergence: the proxy reward climbs without pause, while true quality turns around and heads down once the optimizer wanders off into regions the reward model never learned properly. The reward model is confidently wrong out there, and the optimizer sprints straight for that confident wrongness. THIS is why an RLHF pipeline that trains "too long" or "too hard" starts emitting verbose, hedging, format-gaming answers that the reward model adores and actual humans cannot stand.

The KL leash

So how do you stop the optimizer from wandering into the reward model's blind spots? You put it on a leash. The standard fix in RLHF is to add a KL penalty: reward the policy for scoring well on the reward model, but PUNISH it for straying too far from the original, sensible base model it started as. The objective becomes reward minus beta times the KL divergence from the reference policy.

import torch
import torch.nn.functional as F

def rlhf_objective(policy_logits, ref_logits, reward, beta=0.1):
    """Reward the policy, but penalise drifting from the reference model.
    beta is the leash length: big beta = short leash = stay close to base."""
    policy_logp = F.log_softmax(policy_logits, dim=-1)
    ref_logp = F.log_softmax(ref_logits, dim=-1)
    kl = (policy_logp.exp() * (policy_logp - ref_logp)).sum(dim=-1)
    return reward - beta * kl, kl.mean()

# toy: same tokens, but the policy has drifted from the reference
ref = torch.randn(4, 10)
policy_close = ref + 0.1 * torch.randn(4, 10)
policy_far = ref + 3.0 * torch.randn(4, 10)
r = torch.tensor([2.0, 2.0, 2.0, 2.0])

for name, logits in [("close", policy_close), ("far", policy_far)]:
    obj, kl = rlhf_objective(logits, ref, r)
    print(f"{name:5s}: mean KL {kl:.3f}  penalised objective {obj.mean():+.3f}")

The far-drifted policy earns the same raw reward but gets its objective shredded by the KL term. That single penalty is doing enormous work in every production RLHF system on earth -- it is the difference between "fine-tune the model to be a bit more helpful" and "optimise the model into a gibbering reward-hacking mess." Nota bene: the leash does not SOLVE alignment, it just buys you a safety margin. Set beta too high and the model never learns anything new; too low and it hacks the reward. Tuning that one knob is quit some of the actual craft of RLHF.

Scalable oversight

Here is the problem that keeps safety researchers up at night. Every technique so far assumes a human can EVALUATE the output. But what do you do when the system is faster, cheaper, or plain smarter than its supervisor?

If a model emits 10,000 responses a minute, no human reads each one. If it writes code that runs but hides a subtle bug, you need real expertise to catch it. If it produces a 47-step argument, verifying the argument takes longer than generating it did. This is the scalable oversight problem, and there are a few serious lines of attack.

AI-assisted oversight. Use one model to help a human evaluate another -- the assistant flags suspicious outputs for human review. It does not solve the problem, it MOVES it (now you have to trust the assistant), but in practice it multiplies a reviewer's reach.

Debate. Put two AI systems on opposing sides of a question and let a human judge the argument. The bet is that judging an argument is easier than generating one, so debate amplifies the human's limited judgement into questions they could not have answered alone.

Recursive reward modeling. Humans evaluate the simplest tasks; those judgements train a reward model; that model helps evaluate slightly harder tasks; those evaluations train a better model; and up the ladder you climb into territory no human could grade directly.

class ScalableOversight:
    """Recursive decomposition: break a task down until a human can judge it."""
    def __init__(self, model, human_evaluator):
        self.model = model
        self.human = human_evaluator

    def evaluate(self, task, max_depth=3):
        if self.human_evaluable(task):
            return self.human(task)            # base case: a human can just grade it
        if max_depth <= 0:
            return self.model.best_guess(task)  # out of budget, fall back to the model

        subtasks = self.model.decompose(task)   # split into simpler pieces
        scores = [self.evaluate(s, max_depth - 1) for s in subtasks]
        return sum(scores) / len(scores)        # aggregate the sub-judgements

    def human_evaluable(self, task):
        return task.complexity < self.human.capability_threshold

The recursion is the whole idea: a human cannot judge "is this 500-page proof correct?" but a human CAN judge each lemma, and if you decompose far enough every hard question bottoms out in questions a person can actually answer. The catch -- and it is a real one -- is that decomposition can hide errors in the seams between subtasks, where no single reviewer is looking. Scalable oversight is an open research area, not a solved one.

Constitutional AI

Constitutional AI (CAI), from the crowd at Anthropic, takes a different swing at the same target. Instead of paying humans to label thousands of preference pairs (the expensive heart of RLHF), you hand the model a written set of PRINCIPLES -- a "constitution" -- and teach it to critique and revise its own outputs against them.

It runs in two phases. First, the model answers, then critiques its own answer against each principle, then rewrites it -- generating (original, revised) pairs entirely on its own. Second, you train on those pairs as preferences, exactly like RLHF, except the preference signal came from the model's self-critique rather than a human labeler.

class ConstitutionalAI:
    """Self-critique against written principles -> training pairs, no human labels."""
    def __init__(self, model, principles):
        self.model = model
        self.principles = principles          # list of principle strings

    def critique(self, prompt, response):
        notes = []
        for p in self.principles:
            ask = (f"Principle: {p}\n\nPrompt: {prompt}\nResponse: {response}\n\n"
                   f"Does the response violate this principle? If so, explain how.")
            notes.append(self.model.generate(ask))
        return notes

    def revise(self, prompt, response, notes):
        ask = (f"Prompt: {prompt}\nOriginal: {response}\n\n"
               f"Critiques:\n" + "\n".join(f"- {n}" for n in notes) +
               f"\n\nRewrite the response to fix every critique while staying helpful.")
        return self.model.generate(ask)

    def training_pair(self, prompt):
        initial = self.model.generate(prompt)          # possibly problematic
        notes = self.critique(prompt, initial)
        revised = self.revise(prompt, initial, notes)  # should be better
        return {"prompt": prompt, "rejected": initial, "chosen": revised}

The appeal is that it SCALES: the principles are the specification and the model manufactures its own training signal, so you are no longer bottlenecked on human labelers for every edge case. The risk -- and you should hold onto this the way we held onto "a post-hoc explanation can be wrong" last episode -- is that a model can learn to APPEAR aligned, ticking the letter of each principle while missing the spirit entirely. Self-critique is only as good as the model's honesty about itself, which is exactly the thing under question.

Red teaming

Every technique above tries to BUILD safety in. Red teaming is the opposite reflex: systematically attack your own system to find where it breaks, before your users or a regulator find it for you.

Manual red teaming is humans trying, in bad faith and with creativity, to make the model produce something harmful, biased, or wrong -- jailbreaks, weird edge cases, adversarial framing, unusual contexts. It is slow and expensive and it catches things no automated method thinks of.

Automated red teaming uses one model to generate attacks against another, which scales far past what a human team can churn through.

class AutomatedRedTeam:
    """Use an attacker model to hunt failures in a target model."""
    def __init__(self, attacker, target, safety_classifier):
        self.attacker = attacker
        self.target = target
        self.classifier = safety_classifier

    def run(self, n_attacks=100, category="harmful_content"):
        seed = (f"Write a subtle prompt that might trick an assistant into "
                f"producing {category}. Do not be obviously adversarial.")
        results = []
        for _ in range(n_attacks):
            attack = self.attacker.generate(seed)
            response = self.target.generate(attack)
            results.append({
                "attack": attack,
                "response": response,
                "unsafe": self.classifier.is_unsafe(response),
            })
        rate = sum(r["unsafe"] for r in results) / len(results)
        print(f"attack success rate for '{category}': {rate:.1%}")
        return results

The number that falls out -- attack success rate -- is your safety metric, and the honest truth is that it is never zero. Red teaming is not a box you tick once before launch. It is CONTINUOUS: every model update, every new deployment context, every new capability reopens the attack surface, and the attacks evolve exactly as fast as the defences. A model that was safe last month against last month's jailbreaks tells you nothing about this month's.

The governance landscape

For most of this series I have handed you code. This section I have to hand you the LAW, because in the last two years AI governance stopped being a thought experiment and became a compliance line-item.

The EU AI Act (phasing into force across 2025-2026) is the big one: it classifies systems by RISK. Unacceptable risk -- social scoring, real-time biometric surveillance -- is banned outright. High risk -- medical devices, credit scoring, hiring -- demands conformity assessments, documentation, and genuine human oversight. Limited risk -- a chatbot -- needs only transparency, meaning the user has to KNOW they are talking to a machine. Minimal risk -- your spam filter -- is left alone.

The US approach is sector-by-sector rather than one grand statute: executive orders, NIST risk frameworks, and voluntary commitments instead of a single federal AI law.

China has had rules since 2023 covering generative AI, deep synthesis (deepfakes) and recommendation algorithms, with security assessments required before deployment.

What this means for you, concretely, as a builder: risk classification, documentation, bias testing (remember #35), and human-oversight mechanisms are things you design BEFORE deployment, not bolt on after the regulator calls. The regulatory ratchet only ever tightens -- it does not loosen -- so building these habits now is cheaper than retrofitting them later.

Exercises

Get your hands dirty before the next one. Three tasks, climbing in difficulty:

  1. Put the reward hacker on the leash. Take the reward-model overoptimization loop and add the KL penalty from rlhf_objective around the inner optimization, using the untouched starting point as the "reference." Sweep beta across a few values (say 0.0, 0.1, 1.0) and print, for each, the final proxy reward AND true quality. In one sentence, describe the beta that best protects true quality and why the extremes fail.

  2. Judge a one-round debate. Write a toy where two "debater" functions each return a number as their claim about a hidden answer, and a "judge" function picks the claim closer to a value the judge can cheaply verify but not produce alone. Show that when at least one debater is honest, the judge lands on the right answer more often than a coin flip. Two sentences on why "verifying is easier than generating" is the load-bearing assumption.

  3. Measure a red team. Build a toy target that is "unsafe" whenever an attack string contains a banned keyword, a keyword-stuffing attacker that includes the banned word with some probability p, and a classifier that detects it. Run AutomatedRedTeam for p = 0.2 and p = 0.8 and confirm the reported success rate tracks p. Two sentences on why a LOW automated success rate is NOT proof the model is safe.

We open next episode with full solutions, as always.

Quick recap

  • The alignment problem is the gap between what we can MEASURE and what we actually WANT -- optimization always exploits that gap, and a stronger optimizer exploits it harder;
  • reward hacking is Goodhart's law in code: over-optimize a learned reward model and true quality peaks then collapses while the proxy keeps climbing, because the optimizer sprints into the reward model's confident blind spots;
  • the KL leash (reward minus beta times KL from the reference policy) is the standard brake on that runaway -- it buys a safety margin, it does not remove the problem, and tuning beta is real craft;
  • scalable oversight attacks the "how do you supervise something smarter than you" problem via AI-assisted review, debate, and recursive reward modeling -- all promising, none finished;
  • Constitutional AI trades human preference labels for model self-critique against written principles, which scales beautifully but risks a model that looks aligned without being aligned;
  • red teaming attacks your own system on purpose, manually and automatically, and it is continuous -- the attacks evolve as fast as the defences, and success rate is never quite zero;
  • governance is now law: the EU AI Act's risk tiers, the US sector approach, China's pre-deployment assessments -- risk classification and documentation are design-time work now, not afterthoughts.

And the thread I will leave hanging for next time. We have talked about making AI systems SAFE and making them do what we want. But there is a giant question sitting underneath all of it that we have carefully stepped around: what does any of this COST, and who pays? The compute, the data, the energy, the human labor behind the labels -- the economics of AI shape which of these safety techniques actually get used and which stay in the papers. Follow the money and a lot of the field suddenly makes more sense. That is where we head next ;-)

Bedankt voor het lezen! Go take that reward-hacking loop, rip the KL penalty out, and watch true quality nosedive -- feeling Goodhart's law bite in your own terminal is worth ten paragraphs of me describing it. Tot de volgende keer! ;-)

@scipio

stem stemsocial steemstem python programming
0
0
0.000
0 comments