Learn Zig Series (#105) - Mini Project: Reverse Proxy - Health Checks
Learn Zig Series (#105) - Mini Project: Reverse Proxy - Health Checks

What will I learn?
- Why last episode's failover -- discovering a dead backend by dialing it and getting refused -- is a reactive safety net, and what changes when the proxy knows a box is sick before it ever sends a client there;
- The difference between active health checks (probe every backend on a schedule) and passive ones (learn from real requests that just failed), and why a serious proxy runs both;
- How to model a backend's health as a small three-state machine instead of a bool, and why "we haven't checked yet" is genuinely different from "we checked and it's down";
- What hysteresis is, and why
rise/fallcounters are the single most important knob for stopping a backend from flapping in and out of rotation on one unlucky timeout; - How to run the checker on its own thread (episode 64) with the single-writer discipline that lets the hot path read health state locklessly (episode 30);
- How to fold health into the
pickfrom last episode so a drained pool returns an honest503in stead of knowingly dialing a corpse; - How this same machinery lives inside nginx, HAProxy, Envoy, and Kubernetes readiness probes -- so you'll recognise it everywhere once you've built it once.
Requirements
- A working modern computer running macOS, Windows or Ubuntu;
- An installed Zig 0.14+ distribution (download from ziglang.org);
- The load-balancing proxy we built last episode -- health checks bolt directly onto that pool-and-strategy code, so keep it handy;
- Two or three of the little HTTP servers from episodes 51 through 54, running on different ports, so you have real backends to kill and revive;
- The ambition to learn Zig programming.
Difficulty
- Advanced
Curriculum (of the Learn Zig Series):
- Zig Programming Tutorial - ep001 - Intro
- Learn Zig Series (#2) - Hello Zig, Variables and Types
- Learn Zig Series (#3) - Functions and Control Flow
- Learn Zig Series (#4) - Error Handling (Zig's Best Feature)
- Learn Zig Series (#5) - Arrays, Slices, and Strings
- Learn Zig Series (#6) - Structs, Enums, and Tagged Unions
- Learn Zig Series (#7) - Memory Management and Allocators
- Learn Zig Series (#8) - Pointers and Memory Layout
- Learn Zig Series (#9) - Comptime (Zig's Superpower)
- Learn Zig Series (#10) - Project Structure, Modules, and File I/O
- Learn Zig Series (#11) - Mini Project: Building a Step Sequencer
- Learn Zig Series (#12) - Testing and Test-Driven Development
- Learn Zig Series (#13) - Interfaces via Type Erasure
- Learn Zig Series (#14) - Generics with Comptime Parameters
- Learn Zig Series (#15) - The Build System (build.zig)
- Learn Zig Series (#16) - Sentinel-Terminated Types and C Strings
- Learn Zig Series (#17) - Packed Structs and Bit Manipulation
- Learn Zig Series (#18b) - Addendum: Async Returns in Zig 0.16
- Learn Zig Series (#19) - SIMD with @Vector
- Learn Zig Series (#20) - Working with JSON
- Learn Zig Series (#21) - Networking and TCP Sockets
- Learn Zig Series (#22) - Hash Maps and Data Structures
- Learn Zig Series (#23) - Iterators and Lazy Evaluation
- Learn Zig Series (#24) - Logging, Formatting, and Debug Output
- Learn Zig Series (#25) - Mini Project: HTTP Status Checker
- Learn Zig Series (#26) - Writing a Custom Allocator
- Learn Zig Series (#27) - C Interop: Calling C from Zig
- Learn Zig Series (#28) - C Interop: Exposing Zig to C
- Learn Zig Series (#29) - Inline Assembly and Low-Level Control
- Learn Zig Series (#30) - Thread Safety and Atomics
- Learn Zig Series (#31) - Memory-Mapped I/O and Files
- Learn Zig Series (#32) - Compile-Time Reflection with @typeInfo
- Learn Zig Series (#33) - Building a State Machine with Tagged Unions
- Learn Zig Series (#34) - Performance Profiling and Optimization
- Learn Zig Series (#35) - Cross-Compilation and Target Triples
- Learn Zig Series (#36) - Mini Project: CLI Task Runner
- Learn Zig Series (#37) - Markdown to HTML: Tokenizer and Lexer
- Learn Zig Series (#38) - Markdown to HTML: Parser and AST
- Learn Zig Series (#39) - Markdown to HTML: Renderer and CLI
- Learn Zig Series (#40) - Key-Value Store: In-Memory Store
- Learn Zig Series (#41) - Key-Value Store: Write-Ahead Log
- Learn Zig Series (#42) - Key-Value Store: TCP Server
- Learn Zig Series (#43) - Key-Value Store: Client Library and Benchmarks
- Learn Zig Series (#44) - Image Tool: Reading and Writing PPM/BMP
- Learn Zig Series (#45) - Image Tool: Pixel Operations
- Learn Zig Series (#46) - Image Tool: CLI Pipeline
- Learn Zig Series (#47) - Build a Shell: Parsing Commands
- Learn Zig Series (#48) - Build a Shell: Process Spawning
- Learn Zig Series (#49) - Build a Shell: Built-in Commands
- Learn Zig Series (#50) - Build a Shell: Job Control and Signals
- Learn Zig Series (#51) - HTTP Server: Accept Loop and Parsing
- Learn Zig Series (#52) - HTTP Server: Router and Responses
- Learn Zig Series (#53) - HTTP Server: Static Files and MIME
- Learn Zig Series (#54) - HTTP Server: Middleware and Logging
- Learn Zig Series (#55) - ECS Game Engine: Architecture
- Learn Zig Series (#56) - ECS Game Engine: Component Storage
- Learn Zig Series (#57) - ECS Game Engine: Systems and Queries
- Learn Zig Series (#58) - ECS Game Engine: Terminal Rendering
- Learn Zig Series (#59) - Assembler: Instruction Encoding
- Learn Zig Series (#60) - Assembler: Two-Pass Assembly
- Learn Zig Series (#61) - Assembler: Disassembler and Binary Inspector
- Learn Zig Series (#62) - File Systems: Reading Directories and Metadata
- Learn Zig Series (#63) - File Watching: Detecting Changes
- Learn Zig Series (#64) - Process Management: Fork, Exec, Wait
- Learn Zig Series (#65) - Pipes and Inter-Process Communication
- Learn Zig Series (#66) - Shared Memory and Semaphores
- Learn Zig Series (#67) - Signal Handling Deep Dive
- Learn Zig Series (#68) - Unix Domain Sockets
- Learn Zig Series (#69) - Daemonization: Background Services
- Learn Zig Series (#70) - Timers and Scheduling
- Learn Zig Series (#71) - Resource Limits and Capabilities
- Learn Zig Series (#72) - System Call Wrappers
- Learn Zig Series (#73) - seccomp and Sandboxing
- Learn Zig Series (#74) - ptrace: Process Tracing
- Learn Zig Series (#75) - Reading Kernel State from /proc and /sys
- Learn Zig Series (#76) - Mini Project: Process Monitor
- Learn Zig Series (#77) - Mini Project: File Sync Tool - Part 1
- Learn Zig Series (#78) - Mini Project: File Sync Tool - Part 2: Delta Transfer
- Learn Zig Series (#79) - Mini Project: File Sync Tool - Part 3: Network Protocol
- Learn Zig Series (#80) - Mini Project: File Sync Tool - Part 4: Polish
- Learn Zig Series (#81) - UDP Sockets and Datagrams
- Learn Zig Series (#82) - DNS Resolver from Scratch
- Learn Zig Series (#83) - DNS Server Implementation
- Learn Zig Series (#84) - HTTP/1.1 Deep Dive
- Learn Zig Series (#85) - HTTP/2 Frames and Streams
- Learn Zig Series (#86) - TLS via C Interop
- Learn Zig Series (#87) - WebSocket Protocol
- Learn Zig Series (#88) - WebSocket Server
- Learn Zig Series (#89) - MQTT Messaging Protocol
- Learn Zig Series (#90) - Protocol Buffers Serialization
- Learn Zig Series (#91) - MessagePack Format
- Learn Zig Series (#92) - gRPC Service in Zig
- Learn Zig Series (#93) - SOCKS5 Proxy
- Learn Zig Series (#94) - NAT Traversal and Hole Punching
- Learn Zig Series (#95) - Mini Project: Chat Server - Protocol Design
- Learn Zig Series (#96) - Mini Project: Chat Server - Server Core
- Learn Zig Series (#97) - Mini Project: Chat Server - Client TUI
- Learn Zig Series (#98) - Mini Project: Chat Server - Rooms and History
- Learn Zig Series (#99) - Mini Project: DNS-over-HTTPS Proxy
- Learn Zig Series (#100) - Mini Project: Port Scanner
- Learn Zig Series (#101) - Mini Project: HTTP Load Tester - Part 1
- Learn Zig Series (#102) - Mini Project: HTTP Load Tester - Part 2
- Learn Zig Series (#103) - Mini Project: Reverse Proxy - Routing
- Learn Zig Series (#104) - Mini Project: Reverse Proxy - Load Balancing
- Learn Zig Series (#105) - Mini Project: Reverse Proxy - Health Checks (this post)
Learn Zig Series (#105) - Mini Project: Reverse Proxy - Health Checks
Last episode we taught the proxy to spread traffic across a pool and to fail over when a backend refused a connection. That failover is real resilience, and it's better than most hand-rolled proxies ever get -- but read it back and you'll notice something uncomfortable: we only find out a backend is dead by walking into the burning room. A client's request is the thing that discovers the fire. We dial the corpse, the dial fails, and only then do we shrug and try the next box. The client whose request happened to draw the dead backend eats the latency of a failed connect before we route around it. Do that at a thousand requests a second and "one dead box" becomes "a thousand clients each paying a connect-timeout tax until the pool sorts itself out". Having said that, this is the episode where the humble middleman finally grows up.
The fix has a name every ops person knows: health checking. Instead of learning a backend is sick from the requests that stumble onto it, the proxy checks each backend proactively, on its own schedule, and pulls the sick ones out of rotation before a single client is sent their way. When a backend recovers, the proxy notices that too and quietly slides it back into the pool. The client never sees any of it. That invisibility -- the same invisibility that made load balancing feel like magic last time -- is again the whole product. Let's build it, and let's build it honestly, because health checking is a swamp of subtle failure modes and I'd rather show you where the mud is than pretend the path is dry.
Active versus passive: two ways to know
There are exactly two ways a proxy can learn a backend's health, and a grown-up proxy uses both. Active health checks are the proxy reaching out on a timer -- every couple of seconds it opens a probe connection (or sends a tiny GET /health) to each backend and notes whether it answered. Passive health checks are the proxy learning from the traffic it's already carrying -- when a real client request fails to connect to a backend, that's a data point too, and a fresher one than any scheduled probe.
Active checks catch a dead box proactively (you know within one probe interval, before any client is harmed) but they cost a trickle of extra connections and they only test what the probe tests. Passive checks cost nothing extra (you're already making the request) and they reflect real traffic, but they're reactive -- you only learn from a request that already suffered. Neither is enough alone. Active-only means a backend can die the instant after a probe and stay "healthy" for a full interval. Passive-only is exactly last episode's failover with a memory. Run both and the windows shrink until they nearly close: active checks bound the worst case, passive checks catch death the very next request. We'll wire up both.
Health as a state machine, not a bool
Your first instinct is healthy: bool, and your first instinct is wrong -- or at least too coarse. A backend is not simply up or down; there's a genuinely-different third state, which is we have not checked it yet. At startup, before the first probe has run, a fresh backend deserves the benefit of the doubt: refusing traffic while we warm up would be a self-inflicted outage. So health is a small three-state enum, and thinking of it that way (episode 6's discipline again) prevents a whole category of "why is my brand-new pool serving 503s for the first two seconds" head-scratchers.
const std = @import("std");
/// Health status for a single backend. Three states, NOT a bool, because "we haven't
/// probed it yet" is genuinely distinct from "we probed it and it's down". A fresh
/// backend starts .unknown and is treated as ELIGIBLE until proven guilty -- we'd
/// rather send a request to an unprobed box than refuse traffic during warm-up.
const Health = enum { unknown, healthy, unhealthy };
Now we extend last episode's Backend to carry that health plus the little bit of bookkeeping the checker needs. Two of the new fields matter a lot, and one subtlety hides in which of them are atomic:
/// Last episode's Backend, now health-aware. `health` is read on the hot path by
/// pick() and written by the checker thread, so it MUST be atomic (ep30). The two
/// consecutive counters, by contrast, are touched ONLY by the checker thread -- a
/// single-writer discipline -- so they can stay plain integers. Arranging ownership
/// so that shared state is minimal beats slapping a mutex on everything.
const Backend = struct {
address: std.net.Address,
weight: u32 = 1,
inflight: std.atomic.Value(u32) = std.atomic.Value(u32).init(0),
// Shared across threads: the checker writes it, pick() reads it. Atomic.
health: std.atomic.Value(Health) = std.atomic.Value(Health).init(.unknown),
// Private to the checker thread. Single-writer, so plain ints are safe.
consecutive_ok: u32 = 0,
consecutive_fail: u32 = 0,
};
That split is the design in miniature. The decision (is this backend eligible?) is read constantly, from many threads, on the hottest path in the program -- so it's one atomic load. The reasoning behind the decision (how many probes in a row has it passed or failed?) is only ever computed in one place, by one thread, so it needs no synchronisation at all. Get the ownership right and the concurrency stops being scary.
Hysteresis: the anti-flapping knob
Here's the single most important idea in this whole episode, and the one that separates a health check you'd trust from one that'll page you at 3am for no reason. Networks are noisy. A single probe can time out because a switch hiccuped, because the backend was briefly busy with a garbage-collection pause, because a cosmic ray looked at your packet funny. If one failed probe yanks a backend out of rotation, and one good probe slams it back in, your pool will flap -- backends oscillating in and out of service on random noise, which is worse than useless because it also churns connection state and confuses every dashboard you own.
The cure is hysteresis: require several consecutive results in the same direction before you believe them. A backend must fail fall probes in a row before we declare it unhealthy, and pass rise probes in a row before we declare it healthy again. Asymmetry is normal and deliberate -- you usually want to pull a sick box fast (low-ish fall) but re-admit a recovering one cautiously (higher rise), so a backend that's still thrashing doesn't get traffic the instant it manages one lucky response.
/// Health-check policy. `rise`/`fall` are the hysteresis knobs: a backend must pass
/// `rise` probes IN A ROW to be marked healthy, and fail `fall` IN A ROW to be marked
/// unhealthy. Requiring several consecutive results is what stops a backend flapping
/// in and out of rotation on one noisy timeout -- the most important tuning here.
const HealthPolicy = struct {
interval_ms: u64 = 2000, // how often to probe each backend
timeout_ms: u64 = 1000, // how long to wait for a single probe to answer
rise: u32 = 2, // consecutive successes needed to (re)admit a backend
fall: u32 = 3, // consecutive failures needed to evict a backend
};
The transition logic itself is a pure function of (current counters, this probe's outcome) -- no sockets, no clock, no I/O -- which, as last episode taught us with the balancing strategies, is exactly what makes it testable to exhaustion. Fold a probe result in, update the counters, and flip the atomic health only when a threshold is crossed:
/// Fold ONE probe result into a backend's health, applying hysteresis. Pure logic:
/// counters in, maybe a health transition out, zero I/O. The checker thread calls this
/// after every probe. Note the reset-on-flip: a success zeroes the failure streak and
/// vice-versa, so "consecutive" really means consecutive and a mixed run never trips.
fn applyProbe(b: *Backend, ok: bool, policy: HealthPolicy) void {
if (ok) {
b.consecutive_fail = 0;
b.consecutive_ok += 1;
if (b.consecutive_ok >= policy.rise) {
b.health.store(.healthy, .monotonic);
}
} else {
b.consecutive_ok = 0;
b.consecutive_fail += 1;
if (b.consecutive_fail >= policy.fall) {
b.health.store(.unhealthy, .monotonic);
}
}
}
Notice we reset the opposite counter on every result. That's what makes "consecutive" honest: a backend that goes fail, ok, fail, ok, fail never accumulates three fails in a row, so it never trips -- and a genuinely flapping backend like that arguably shouldn't be confidently marked either way. The counters encode "how sure are we, and in which direction", and the thresholds are where uncertainty becomes a decision.
The probe itself
What actually is a probe? At the simplest, layer 4: can we open a TCP connection to the backend at all? If the box is down, the kernel refuses the SYN or the connect times out. That's a cheap, meaningful liveness signal and it's where I'd start. At layer 7 you go further -- send GET /health HTTP/1.1 and insist on a 200 -- which catches the nastier failure where the process is accepting connections but is internally broken (database handle dead, disk full, deadlocked worker pool). The shape of the code is identical; the L7 version just writes a few bytes and reads the status line back. We'll do the L4 version and I'll be candid about the timeout.
/// One active probe: can we even open a TCP connection to this backend? A bare connect
/// is the cheapest liveness signal -- refused or timed-out both mean "down". For an L7
/// check you'd send `GET /health` and require a 200; same structure, a few more bytes.
/// We treat ANY error as a failed probe, because from the balancer's chair a refused
/// connect, a timeout, and no-route-to-host all mean the identical thing: don't send
/// users here. Nota bene: a production probe needs a real connect timeout (non-blocking
/// connect + poll), otherwise a black-holed backend hangs the checker for the OS default.
fn probeTcp(address: std.net.Address, timeout_ms: u64) bool {
_ = timeout_ms; // see the note above -- wire SO_SNDTIMEO / non-blocking connect here
const stream = std.net.tcpConnectToAddress(address) catch return false;
stream.close();
return true;
}
I'm flagging that discarded timeout_ms on purpose, because it's the exact spot a naive health checker quietly betrays you. If a backend doesn't refuse the connection but just swallows it -- a black hole, a machine whose network stack is wedged, a firewall dropping packets silently -- then a blocking connect waits for the operating system's default timeout, which can be over a minute. One black-holed backend can stall the entire single-threaded checker, so every other backend's probe is delayed too, and your fast health checker becomes a slow one for the whole pool. The fix is a non-blocking connect followed by a poll with your own deadline, which I'm leaving as the obvious hardening step rather than drowning this listing in socket-option ceremony. Knowing the trap is there is half the battle.
The checker thread
The probes run on their own thread so the accept loop never blocks waiting for one (episode 64 gave us std.Thread.spawn, episode 70 the timing intuition). The loop is deliberately dull: wake, probe every backend of every route, fold each result in, sleep, repeat -- until someone sets the stop flag. And it is the only writer of health state and the only toucher of the consecutive counters, which is precisely the single-writer discipline that lets the hot path read health locklessly and never see a torn value.
/// The active health-check loop. Runs on its own thread (ep64) and is the SOLE writer
/// of health state + the sole toucher of the consecutive counters -- that single-writer
/// discipline is what lets pick() read `health` locklessly. `stop` is checked with
/// .acquire so we see the main thread's .release store promptly on shutdown (ep30).
fn healthLoop(routes: []Route, policy: HealthPolicy, stop: *std.atomic.Value(bool)) void {
while (!stop.load(.acquire)) {
for (routes) |*route| {
for (route.backends) |*b| {
const ok = probeTcp(b.address, policy.timeout_ms);
applyProbe(b, ok, policy);
}
}
std.Thread.sleep(policy.interval_ms * std.time.ns_per_ms);
}
}
A real deployment would probe backends concurrently (one slow probe shouldn't delay the rest, per the black-hole warning above) and would jitter the interval a touch so a hundred proxies don't all hammer the same backend on the same tick. But the skeleton is exactly this: a background heartbeat that keeps everyone's health field honest.
Teaching pick to skip the sick
All that probing is worthless unless the balancer acts on it. Last episode's pick chose among all backends blindly; now it must only ever hand out an eligible one. Eligibility is a one-liner with a deliberate subtlety baked in:
/// A backend is eligible for traffic unless we've AFFIRMATIVELY marked it unhealthy.
/// The subtlety: .unknown counts as eligible. A backend we haven't probed yet gets the
/// benefit of the doubt -- pulling traffic during warm-up would be a self-own. Only a
/// backend we positively decided is down gets excluded. A single relaxed atomic load.
fn isEligible(b: *const Backend) bool {
return b.health.load(.monotonic) != .unhealthy;
}
Now round-robin becomes health-aware round-robin: advance the shared cursor exactly as before (still one atomic fetchAdd, still lock-free, straight out of last episode), but skip any backend that isn't eligible, trying at most backends.len steps. If we walk the whole pool and everyone is down, we return null -- and returning null is the honest thing, because knowingly dialing a backend we just decided is dead would be lying to ourselves.
/// Round-robin that respects health. Same atomic cursor as ep104, but we SKIP backends
/// that aren't eligible, bounded to backends.len attempts so a fully-down pool can't
/// loop forever. Returning null means "everyone is down" -- the caller then serves an
/// honest 503 instead of pretending it can help. THIS is where all the probing pays off.
fn pickHealthyRoundRobin(route: *Route) ?usize {
var tries: usize = 0;
while (tries < route.backends.len) : (tries += 1) {
const idx = route.cursor.fetchAdd(1, .monotonic) % route.backends.len;
if (isEligible(&route.backends[idx])) return idx;
}
return null; // pool fully drained -- nobody healthy to serve
}
The other strategies from last episode (least-connections, weighted) get the identical treatment -- filter to eligible backends first, then apply the policy among the survivors. I'll spare you the near-duplicate listings; the pattern is what matters, and the pattern is "health is a gate that sits in front of whatever balancing policy you chose".
Passive checks: trip the breaker instantly
Active checks bound our worst case to one probe interval. Passive checks slam that window shut. When a real request fails to even connect to a backend, we shouldn't wait up to two seconds for the next scheduled probe to notice the obvious -- we should react on the spot. This is the circuit breaker pattern, and there's a neat way to fit it in without breaking our single-writer discipline.
The trick is asymmetry: the request path is allowed to trip a backend down but never to bring it up. Tripping down is a single atomic store(.unhealthy), which is safe from any thread. Bringing a backend back up stays the exclusive job of the active checker's applyProbe, so the consecutive counters remain single-writer and untouched by request threads. Any thread can yank a dead box out instantly; only the deliberate, hysteresis-guarded checker can slide one back in.
/// Passive circuit-breaker trip. A live request just failed to connect to this backend,
/// so we short-circuit it to .unhealthy RIGHT NOW rather than waiting an interval for
/// the active checker. Crucially, the request path may only ever trip DOWN -- recovery
/// stays the checker's exclusive job via applyProbe. That asymmetry keeps the counters
/// single-writer while still letting ANY thread evict a corpse the instant it's found.
fn tripUnhealthy(b: *Backend) void {
b.health.store(.unhealthy, .monotonic);
}
I like this design because the concurrency argument is airtight and you can see it at a glance: writes to health come from two places (checker and request threads) but they're all plain atomic stores, so they can't tear; writes to the counters come from exactly one place (the checker), so they need no atomics at all. No mutex, no lock, no lost update, and every line of the reasoning fits in your head. That's the payoff of thinking about who-owns-what before reaching for a lock.
Wiring failover to health
Now last episode's serveWithFailover gets its health upgrade. It asks the pool only for eligible backends via pickHealthyRoundRobin; a connect failure both fails over to the next backend and trips the loser's breaker so the very next request skips it entirely. And when the pool is genuinely empty of healthy backends, we return 503 Service Unavailable -- which is the semantically correct code for "the service is temporarily unable to help", distinct from the 502 Bad Gateway you'd send for a backend that answered with garbage.
/// Failover, now health-aware. We only ever pick ELIGIBLE backends; a connect failure
/// both fails over AND trips that backend's breaker (passive check) so subsequent
/// requests skip it without paying the failed-connect tax. An empty healthy set is an
/// honest 503 -- "no healthy backend right now" -- not last episode's 502.
fn serveWithFailover(route: *Route, head: []const u8, client: std.net.Stream, scratch: []u8) void {
var attempts: usize = 0;
while (attempts < route.backends.len) : (attempts += 1) {
const idx = pickHealthyRoundRobin(route) orelse break; // nobody healthy left
const backend = &route.backends[idx];
_ = backend.inflight.fetchAdd(1, .monotonic);
defer _ = backend.inflight.fetchSub(1, .monotonic);
relay(backend.address, head, client, scratch) catch |err| {
std.log.warn("backend {d} failed ({s}); tripping + failing over", .{ idx, @errorName(err) });
tripUnhealthy(backend); // passive: yank it now, don't wait for the prober
continue;
};
return; // served cleanly
}
std.log.err("no healthy backend for {s}", .{route.prefix});
sendError(client, 503, "Service Unavailable");
}
Read the two halves together and you see the belt-and-suspenders in action. The active checker keeps health roughly current in the background; the request path both consumes that health (via pickHealthyRoundRobin) and feeds it (via tripUnhealthy). A backend that dies is out of rotation within one probe interval at worst, and within one request at best.
Testing the state machine without a socket
Because applyProbe and pickHealthyRoundRobin are pure logic over in-memory state, we can nail the tricky behaviours -- hysteresis and pool-drain -- in tests that run in microseconds and never open a socket. The hysteresis test is the one I'd defend hardest in review, because it pins down the exact anti-flap guarantee the whole episode rests on: one bad probe must not unseat a healthy backend.
test "hysteresis: a single failure does not unseat a healthy backend" {
const policy = HealthPolicy{ .rise = 2, .fall = 3 };
var b = Backend{ .address = undefined };
// Two good probes in a row bring an unknown backend up (rise = 2).
applyProbe(&b, true, policy);
applyProbe(&b, true, policy);
try std.testing.expectEqual(Health.healthy, b.health.load(.monotonic));
// ONE failure must NOT evict it -- this is the anti-flap promise, in one assertion.
applyProbe(&b, false, policy);
try std.testing.expectEqual(Health.healthy, b.health.load(.monotonic));
// But three failures in a row (fall = 3) do evict it.
applyProbe(&b, false, policy);
applyProbe(&b, false, policy);
try std.testing.expectEqual(Health.unhealthy, b.health.load(.monotonic));
}
test "a fully drained pool yields no backend" {
var backends = [_]Backend{
.{ .address = undefined },
.{ .address = undefined },
};
for (&backends) |*b| b.health.store(.unhealthy, .monotonic);
var route = Route{ .prefix = "/", .backends = &backends };
// Everyone down -> pick returns null -> caller serves 503, never dials a corpse.
try std.testing.expectEqual(@as(?usize, null), pickHealthyRoundRobin(&route));
}
Two tests, and between them they cover the two ways this feature can betray you: flapping (backend evicted on noise) and false confidence (dialing a backend we know is dead). A red bar on either would scream the moment a refactor broke the invariant -- which is exactly the safety net you want around code that decides whether real users get served.
Wiring it into main
The main from last episode barely changes. We build the same pool, then spawn the checker thread before the accept loop and join it on the way out via defer. The accept loop itself is unchanged in spirit -- route, then serveWithFailover -- it just quietly benefits from a health field that's being kept honest behind its back.
pub fn main() !void {
var gpa = std.heap.GeneralPurposeAllocator(.{}){};
defer _ = gpa.deinit();
const alloc = gpa.allocator();
var api_pool = [_]Backend{
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9001), .weight = 3 },
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9002), .weight = 1 },
.{ .address = try std.net.Address.parseIp("127.0.0.1", 9003), .weight = 1 },
};
var routes = [_]Route{
.{ .prefix = "/api", .backends = &api_pool },
};
var router = Router{ .routes = &routes };
const policy = HealthPolicy{};
var stop = std.atomic.Value(bool).init(false);
// The checker runs in the background so the accept loop never blocks on a probe.
const checker = try std.Thread.spawn(.{}, healthLoop, .{ &routes, policy, &stop });
defer {
stop.store(true, .release); // ask the loop to finish...
checker.join(); // ...and wait for it, so no thread outlives main.
}
const listen_addr = try std.net.Address.parseIp("127.0.0.1", 8080);
var server = try listen_addr.listen(.{ .reuse_address = true });
defer server.deinit();
std.log.info("health-checked proxy on {any}", .{listen_addr});
const buf = try alloc.alloc(u8, 64 * 1024);
defer alloc.free(buf);
const scratch = try alloc.alloc(u8, 64 * 1024);
defer alloc.free(scratch);
while (true) {
const conn = server.accept() catch continue;
// routing is unchanged from ep103; serveWithFailover now respects health.
const route = router.matchRoute("/api") orelse { conn.stream.close(); continue; };
serveWithFailover(route, "", conn.stream, buf);
conn.stream.close();
_ = scratch;
}
}
Fire up three backends on 9001-9003, point a loop of requests at the proxy, then kill one backend mid-loop and watch what's different from last episode:
$ for i in $(seq 1 9); do curl -s localhost:8080/api/ping; done
# all three answer in turn: 9001 9002 9003 9001 9002 9003 ...
# now kill 9002 and keep the requests coming:
$ kill %2
# ep104: every request that DREW 9002 first paid a failed-connect tax, then failed over.
# ep105: within one probe interval 9002 is marked unhealthy and SKIPPED entirely --
# no client ever draws it again until it comes back and passes `rise` probes.
That's the whole upgrade in one observable behaviour: the dead box stops costing anybody anything, almost immediately, and re-admits itself cautiously when it recovers. No 502s, no connect-timeout tax, no flapping.
How this compares to C, Rust, and Go
In C, this is bread-and-butter nginx (health_check in the plus build, plus the passive max_fails/fail_timeout in open-source nginx) and HAProxy, whose option httpchk and rise/fall/inter directives are literaly the knobs we just built -- I did not pick those field names by accident, they're HAProxy's own vocabulary. What Zig bought us over the C originals is, once again, the atomics carrying their memory-ordering contract in the type: std.atomic.Value(Health) versus a naked _Atomic int and a lot of trusting the programmer to pass the right order to every access. The single-writer discipline that keeps our counters lock-free is achievable in C too, but nothing in C reminds you which fields are shared -- here the std.atomic.Value wrapper is a standing sign that says "many threads, tread carefully", and the plain u32 next to it says "one thread, relax".
In Rust, you'd reach for tokio and probably the tower ecosystem again, where health and load-shedding live as composable Service layers, and the borrow checker would force the shared/private split we arrived at by discipline -- a health behind an AtomicU8 in an Arc, the private counters owned by the checker task and simply not shared, so a stray access from a request task wouldn't compile. That's the recurring trade: Rust makes the mistake impossible at the cost of ceremony; our Zig version leaves the door unlocked but makes the correct arrangement small enough to hold in your head and verify by eye.
In Go, you'd write almost exactly this by hand (the standard library gives you httputil.ReverseProxy for the relay but no health checking at all), a background goroutine ranging over backends on a time.Ticker, flipping an atomic.Bool or a mutex-guarded map. It'd be a dozen tidy lines -- and having built ours from the probe and the atomic up, you now know precisely why that goroutine needs the atomic, why one bad tick shouldn't flip the flag, and why the recovery path deserves more consecutive successes than the eviction path deserves failures ;-)
Where this is heading
Step back and look at what the reverse proxy became over three episodes: a routing switchboard (ep103), a load balancer across a pool (ep104), and now a self-healing front door that quietly evicts and re-admits backends without a client ever noticing. The genuinely new idea today was small and portable -- health is a gate that sits in front of your balancing policy, kept honest by a background heartbeat and a circuit breaker, tuned by a couple of hysteresis counters that stop the whole thing from flapping on noise. That exact pattern is a Kubernetes readiness probe, an AWS target-group health check, an Envoy outlier detector. Build it once from the sockets up and you'll recognise it in every piece of infrastructure you ever touch.
And with that, our little proxy is genuinely done -- not toy-done, but "I understand every atomic in it" done, which is a far better place to stand than "I trust the framework". So the next stretch of the series turns a corner. All through this networking arc we've leaned on slices, hash maps, and pools as if they were furniture -- always just there, provided by the standard library, used without a second thought. But a systems programmer who can only use the container the stdlib happened to ship is at its mercy the day its trade-offs don't fit. Pointers (episode 8) and allocators (episode 26) were the warm-up; from here we go back to first principles and build the fundamental data structures ourselves, from the simplest node-and-pointer arrangement upward, and learn exactly why each one makes the machines we've been writing fast. That's next.
Thanks for reading -- catch you in the next episode!