Learn Zig Series (#90) - Protocol Buffers Serialization

avatar

Learn Zig Series (#90) - Protocol Buffers Serialization

zig.png

What will I learn?

  • Why a binary serialization format exists at all, and what protobuf buys you over hand-rolled byte layouts or JSON;
  • How protobuf's wire format packs a field number and a type into a single varint tag;
  • How the base-128 varint works (the exact trick we just wrote for MQTT, reused);
  • Why signed integers get the ZigZag treatment before they hit the wire;
  • How length-delimited fields carry strings, bytes and nested messages with one uniform rule;
  • How to decode an unknown message and skip fields you've never heard of -- the property that makes protobuf forwards-compatible;
  • How Zig's comptime reflection (@typeInfo) lets us generate an encoder for any struct, with no codegen step;
  • How to test a binary format properly, and where the performance actually goes.

Requirements

  • A working modern computer running macOS, Windows or Ubuntu;
  • An installed Zig 0.14+ distribution (download from ziglang.org);
  • The ambition to learn Zig programming.

Difficulty

  • Intermediate

Curriculum (of the Learn Zig Series):

Learn Zig Series (#90) - Protocol Buffers Serialization

Solutions to Episode 89 Exercises

Last episode we built MQTT from the bytes up -- fixed header, varint length, PUBLISH parser, QoS levels, wildcard matching, and a tiny broker. The three exercises asked you to flesh that broker out into something a real client could talk to. Here are the solutions, each reusing the readString, QoS, PacketType and encodeRemainingLength helpers from episode 89.

Exercise 1: Parse a SUBSCRIBE packet (and answer with SUBACK)

A SUBSCRIBE body is a 2-byte packet id followed by one or more (length-prefixed filter, qos byte) entries. The trap is the QoS byte: only values 0, 1, 2 are legal, the upper bits are reserved and must be zero:

const std = @import("std");

pub const SubEntry = struct { filter: []const u8, qos: QoS };

pub fn parseSubscribe(alloc: std.mem.Allocator, body: []const u8) !struct {
    packet_id: u16,
    entries: []SubEntry,
} {
    if (body.len < 2) return error.MalformedSubscribe;
    const packet_id = std.mem.readInt(u16, body[0..2], .big);

    var off: usize = 2;
    var list: std.ArrayListUnmanaged(SubEntry) = .{};
    errdefer list.deinit(alloc); // free partial work if a later entry is bad
    while (off < body.len) {
        const filter = try readString(body, &off); // reused from episode 89
        if (off >= body.len) return error.MalformedSubscribe;
        const qos_byte = body[off];
        off += 1;
        if (qos_byte > 2) return error.MalformedSubscribe; // reserved bits / value 3
        try list.append(alloc, .{ .filter = filter, .qos = @enumFromInt(@as(u2, @intCast(qos_byte))) });
    }
    if (list.items.len == 0) return error.MalformedSubscribe; // SUBSCRIBE must carry >=1 filter
    return .{ .packet_id = packet_id, .entries = try list.toOwnedSlice(alloc) };
}

pub fn encodeSubAck(out: []u8, packet_id: u16, granted: []const QoS) !usize {
    const remaining = 2 + granted.len; // 2-byte id + one return code per filter
    if (out.len < 2 + remaining) return error.BufferTooSmall;
    out[0] = @as(u8, @intFromEnum(PacketType.suback)) << 4;
    var i: usize = 1;
    i += try encodeRemainingLength(@intCast(remaining), out[i..]);
    std.mem.writeInt(u16, out[i..][0..2], packet_id, .big);
    i += 2;
    for (granted) |q| {
        out[i] = @intFromEnum(q); // granted QoS == requested, in the simple case
        i += 1;
    }
    return i;
}

The errdefer list.deinit(alloc) is the detail that separates correct from leaky: if the third entry is malformed we've already allocated the first two, and that errdefer reclaims them on the way out. This is episode 7's allocator discipline showing up exactly where it matters.

Exercise 2: Build the CONNECT/CONNACK handshake

A CONNECT opens with the protocol name ("MQTT"), then a version byte, a flags byte, two keep-alive bytes, and the client identifier. We parse far enough to validate the version and reply accordingly:

pub const Connect = struct {
    protocol_level: u8,
    keep_alive: u16,
    client_id: []const u8,
};

pub fn parseConnect(body: []const u8) !Connect {
    var off: usize = 0;
    const proto = try readString(body, &off);
    if (!std.mem.eql(u8, proto, "MQTT")) return error.BadProtocolName;
    if (body.len < off + 4) return error.Malformed; // level + flags + 2 keep-alive bytes
    const level = body[off];
    off += 1;
    off += 1; // connect flags -- ignored in this minimal parser
    const keep_alive = std.mem.readInt(u16, body[off..][0..2], .big);
    off += 2;
    const client_id = try readString(body, &off);
    return .{ .protocol_level = level, .keep_alive = keep_alive, .client_id = client_id };
}

pub fn encodeConnAck(out: []u8, session_present: bool, return_code: u8) !usize {
    if (out.len < 4) return error.BufferTooSmall;
    out[0] = @as(u8, @intFromEnum(PacketType.connack)) << 4;
    out[1] = 2; // remaining length is always 2
    out[2] = if (session_present) 1 else 0; // session-present flag
    out[3] = return_code; // 0 = accepted
    return 4;
}

test "reject CONNECT with protocol level 3" {
    // "MQTT" string + level 3 + flags + keepalive + empty client id
    const body = [_]u8{ 0, 4, 'M', 'Q', 'T', 'T', 3, 0, 0, 60, 0, 0 };
    const c = try parseConnect(&body);
    const rc: u8 = if (c.protocol_level == 4) 0 else 0x01; // 0x01 = unacceptable version
    try std.testing.expectEqual(@as(u8, 0x01), rc);
}

Notice parseConnect does not decide the return code -- it just reports what the client claimed. The policy (level 4 good, anything else 0x01) lives in the handler, which keeps the parser pure and testable, exactly as it kept us honest in episode 12.

Exercise 3: Wire the broker into last episode's event loop

The whole point is one dispatch function over the drain loop. Parse a fixed header, make sure the full packet is buffered, hand the body to a handler that switches on the packet type:

fn handlePacket(broker: *Broker, conn: *Connection, alloc: std.mem.Allocator, hdr: FixedHeader, body: []const u8) !void {
    var scratch: [4096]u8 = undefined;
    switch (hdr.packet_type) {
        .connect => {
            const c = try parseConnect(body);
            const rc: u8 = if (c.protocol_level == 4) 0 else 0x01;
            const n = try encodeConnAck(&scratch, false, rc);
            try writeAll(conn.fd, scratch[0..n]);
        },
        .subscribe => {
            const sub = try parseSubscribe(alloc, body);
            defer alloc.free(sub.entries);
            var granted: [64]QoS = undefined;
            for (sub.entries, 0..) |e, idx| {
                try broker.subscribe(alloc, conn.fd, e.filter, e.qos);
                granted[idx] = e.qos;
            }
            const n = try encodeSubAck(&scratch, sub.packet_id, granted[0..sub.entries.len]);
            try writeAll(conn.fd, scratch[0..n]);
        },
        .publish => {
            const p = try parsePublish(hdr, body);
            broker.route(p.topic, p.payload);
            if (p.qos == .at_least_once) {
                const n = try encodePubAck(&scratch, p.packet_id.?);
                try writeAll(conn.fd, scratch[0..n]);
            }
        },
        .pingreq => {
            scratch[0] = @as(u8, @intFromEnum(PacketType.pingresp)) << 4;
            scratch[1] = 0; // PINGRESP has zero remaining length
            try writeAll(conn.fd, scratch[0..2]);
        },
        .disconnect => conn.state = .closed,
        else => {}, // packets a minimal broker doesn't originate
    }
}

And the drain loop is the same slide-the-buffer pattern from episode 88, just with the MQTT header parser plugged in where parseFrame used to sit:

fn drain(broker: *Broker, conn: *Connection, alloc: std.mem.Allocator) !void {
    var consumed: usize = 0;
    while (true) {
        const rest = conn.in.items[consumed..];
        const hdr = (try parseFixedHeader(rest)) orelse break; // need more bytes
        const total = hdr.header_len + hdr.remaining_length;
        if (rest.len < total) break; // packet not fully arrived yet
        const body = rest[hdr.header_len..total];
        try handlePacket(broker, conn, alloc, hdr, body);
        consumed += total;
    }
    // discard the bytes we fully processed; keep any partial tail
    std.mem.copyForwards(u8, conn.in.items[0..], conn.in.items[consumed..]);
    conn.in.shrinkRetainingCapacity(conn.in.items.len - consumed);
}

Connect two clients, subscribe one to test/#, publish from the other, watch it land. Three exercises and the broker went from "parses packets" to "holds a conversation". Now on to the format I promised you at the end of last episode.

Learn Zig Series (#90) - Protocol Buffers Serialization

Here we go ;-) At the close of episode 89 I pointed forward to "the formats whose entire job is taking arbitrary structured messages and packing them into the fewest honest bytes". Protocol Buffers -- protobuf, to its friends -- is the most widely deployed of exactly that breed. Google built it internally, open-sourced it in 2008, and today it is the wire format underneath gRPC, half the infrastructure at every large tech company, and a startling amount of the traffic flowing through your phone right now.

The pitch is simple. You describe your data once, in a .proto schema. From that schema, both ends agree on a compact binary layout where every field is identified by a small number rather than its name. The result is dramatically smaller than JSON (no repeated field names, no quotes, no whitespace), faster to parse, and -- the genuinely clever part -- forwards and backwards compatible, so a new client and an old server can keep talking even after the schema grows. We are not going to write a .proto compiler today (that's a parser project of its own). We are going to do the more illuminating thing: implement the wire format by hand, so that when you do reach for a generated library, you know precisely what those generated bytes are.

Why not just memcpy the struct?

The naive idea -- "I'll just write the struct's bytes straight to the socket" -- falls apart the instant two machines disagree. Endianness differs, struct padding differs between compilers, pointer fields hold addresses meaningless to the other process, and the moment you add a field every previously-written message becomes unreadable. We saw a gentler version of this back in episode 8 when we looked at memory layout: the in-memory shape of a struct is an implementation detail, not a contract.

JSON fixes the portability problem but overcorrects -- it ships the field name as a string with every single value, re-parses text into numbers on every read, and has no notion of a schema the two sides agreed on in advance. We wrote a JSON codec back in episode 20, so you've felt that cost. Protobuf threads the needle: a fixed, documented binary layout (portable like JSON) where fields are tiny integers (compact like a raw struct) and unknown fields are skippable (so the schema can evolve). That last property is why it won.

The wire format: tag, then value

Every field on the wire is a tag followed by a value. The tag is a single varint that packs two things: the field's number (the integer you assigned in the schema) in the upper bits, and a 3-bit wire type in the lowest three bits, telling the decoder how to read the value that follows.

const std = @import("std");

pub const WireType = enum(u3) {
    varint = 0, // int32, int64, uint*, bool, enum -- a base-128 varint
    i64 = 1, // fixed 8 bytes -- double, fixed64, sfixed64
    len = 2, // length-delimited -- string, bytes, embedded message, packed repeated
    sgroup = 3, // start group (deprecated, never emit)
    egroup = 4, // end group (deprecated)
    i32 = 5, // fixed 4 bytes -- float, fixed32, sfixed32
};

/// tag = (field_number << 3) | wire_type
pub fn encodeTag(field: u32, wt: WireType) u64 {
    return (@as(u64, field) << 3) | @intFromEnum(wt);
}

pub fn decodeTag(tag: u64) struct { field: u32, wire_type: WireType } {
    const wt: u3 = @intCast(tag & 0x7); // low 3 bits
    return .{ .field = @intCast(tag >> 3), .wire_type = @enumFromInt(wt) };
}

Backing the enum with a u3 is the same trick the MQTT PacketType enum used last episode -- the type physically cannot hold an out-of-range wire type, so the dispatch logic never has to second-guess a stray value. The wire type is the whole reason a decoder can skip a field it doesn't recognise: even without knowing what field 47 means, it knows from the wire type how many bytes to step over. Hold that thought, it's the punchline.

The varint, again (and why it's everywhere)

I told you in episode 89 that the seven-bits-plus-continuation-bit varint was "one of those ideas you recognise instantly in five other protocols". Welcome to protocol number two. Protobuf's varint is the same scheme -- seven payload bits per byte, top bit set means "more follows" -- with one difference from MQTT's Remaining Length: protobuf varints carry full 64-bit values, so they run up to ten bytes, and the groups are emitted least-significant first.

/// Append a base-128 varint of `value` to `list`. 1..10 bytes for a u64.
pub fn writeVarint(list: *std.ArrayListUnmanaged(u8), alloc: std.mem.Allocator, value: u64) !void {
    var x = value;
    while (x >= 0x80) {
        try list.append(alloc, @as(u8, @intCast(x & 0x7F)) | 0x80); // low 7 bits + continuation
        x >>= 7;
    }
    try list.append(alloc, @intCast(x)); // final byte, top bit clear
}

/// Read a varint from `buf` at `off`, advancing it. Hard-stops at 10 bytes.
pub fn readVarint(buf: []const u8, off: *usize) !u64 {
    var result: u64 = 0;
    var shift: u6 = 0;
    var count: usize = 0;
    while (count < 10) : (count += 1) {
        if (off.* >= buf.len) return error.Truncated;
        const byte = buf[off.*];
        off.* += 1;
        result |= @as(u64, byte & 0x7F) << shift;
        if (byte & 0x80 == 0) return result; // continuation bit clear -> done
        shift += 7;
    }
    return error.VarintTooLong; // 10 bytes and still going: malformed, reject
}

That count < 10 guard is not decoration -- without it a hostile peer hands you an endless run of 0x80 bytes and your decoder either loops forever or overflows shift (and a u6 shift past 63 is illegal in Zig, which would panic in safe builds). This is episode 6's lesson once more: a wire parser must treat every length and every count as adversarial, because on a public socket, eventually it is.

ZigZag: making signed integers behave

Here's a subtlety. A plain varint encodes small unsigned numbers compactly, but a small negative number like -1, stored as two's complement, has all its high bits set, so it would waste the full ten bytes every time. Protobuf's answer for its sint32/sint64 types is ZigZag encoding: map signed integers to unsigned ones so that numbers near zero -- positive or negative -- stay small. 0 becomes 0, -1 becomes 1, 1 becomes 2, -2 becomes 3, and so on, zig-zagging across the number line.

pub fn zigzagEncode(n: i64) u64 {
    // arithmetic shift fills with the sign bit; XOR folds negatives next to positives
    return @bitCast((n << 1) ^ (n >> 63));
}

pub fn zigzagDecode(u: u64) i64 {
    const x: i64 = @bitCast(u >> 1);
    return x ^ -(@as(i64, @intCast(u & 1))); // restore sign from the low bit
}

test "zigzag keeps small magnitudes small" {
    try std.testing.expectEqual(@as(u64, 0), zigzagEncode(0));
    try std.testing.expectEqual(@as(u64, 1), zigzagEncode(-1));
    try std.testing.expectEqual(@as(u64, 2), zigzagEncode(1));
    try std.testing.expectEqual(@as(u64, 3), zigzagEncode(-2));
    try std.testing.expectEqual(@as(i64, -1), zigzagDecode(zigzagEncode(-1)));
}

The @bitCast is doing real work here -- we want the raw bit pattern reinterpreted, not a numeric conversion that would trip an overflow check. This is exactly the kind of bit-level reinterpretation we leaned on in episode 17, and Zig makes the intent explicit: @bitCast says "same bits, different type", @intCast says "same value, different type, and panic if it doesn't fit". Two different operations, two different builtins, no silent C-style ambiguity.

Encoding a message

Time to put a real message on the wire. Say we're modelling a Person -- field 1 is an id (int32, varint), field 2 is a name (string, length-delimited), field 3 is an email. Encoding is "for each field, write its tag, then its value". Length-delimited values write a varint length, then the raw bytes:

const Person = struct {
    id: i64,
    name: []const u8,
    email: []const u8,
};

/// Write a length-delimited field (string / bytes / sub-message).
fn writeLenField(list: *std.ArrayListUnmanaged(u8), alloc: std.mem.Allocator, field: u32, bytes: []const u8) !void {
    try writeVarint(list, alloc, encodeTag(field, .len));
    try writeVarint(list, alloc, bytes.len); // length prefix
    try list.appendSlice(alloc, bytes); // then the payload, verbatim
}

pub fn encodePerson(alloc: std.mem.Allocator, p: Person) ![]u8 {
    var out: std.ArrayListUnmanaged(u8) = .{};
    errdefer out.deinit(alloc);

    // field 1: id as a varint
    try writeVarint(&out, alloc, encodeTag(1, .varint));
    try writeVarint(&out, alloc, @bitCast(p.id));

    // fields 2 and 3: length-delimited strings
    try writeLenField(&out, alloc, 2, p.name);
    try writeLenField(&out, alloc, 3, p.email);

    return out.toOwnedSlice(alloc);
}

Notice what is not in those bytes: the words "id", "name", "email" appear nowhere. The decoder learns field 1 is the id because both ends share the schema, and that is the entire compactness story in one observation. A JSON encoding of the same person ships every key name on every message; protobuf ships a one-byte tag.

Decoding -- and skipping the unknown

Decoding walks the buffer reading (tag, value) pairs until the bytes run out. For each tag we pull the field number and wire type; if it's a field we care about, we read it into our struct; if it's a field we've never heard of, we skip it using only the wire type. This skip is the mechanism behind protobuf's forwards compatibility -- an old decoder gracefully steps over fields a newer schema added.

/// Step over a field we don't recognise, using only its wire type.
fn skipField(buf: []const u8, off: *usize, wt: WireType) !void {
    switch (wt) {
        .varint => _ = try readVarint(buf, off),
        .i64 => off.* += 8,
        .i32 => off.* += 4,
        .len => {
            const n = try readVarint(buf, off);
            off.* += @intCast(n);
        },
        .sgroup, .egroup => return error.DeprecatedGroup,
    }
    if (off.* > buf.len) return error.Truncated; // a bogus length must not run us past the end
}

pub fn decodePerson(buf: []const u8) !Person {
    var p: Person = .{ .id = 0, .name = "", .email = "" };
    var off: usize = 0;
    while (off < buf.len) {
        const tag = try readVarint(buf, &off);
        const d = decodeTag(tag);
        switch (d.field) {
            1 => p.id = @bitCast(try readVarint(buf, &off)),
            2 => p.name = try readLen(buf, &off),
            3 => p.email = try readLen(buf, &off),
            else => try skipField(buf, &off, d.wire_type), // unknown field -> step over it
        }
    }
    return p;
}

/// Read a length-delimited byte slice, advancing `off`. Borrows from `buf`.
fn readLen(buf: []const u8, off: *usize) ![]const u8 {
    const n: usize = @intCast(try readVarint(buf, off));
    if (off.* + n > buf.len) return error.Truncated;
    const s = buf[off.*..][0..n];
    off.* += n;
    return s;
}

Two design choices worth calling out. First, readLen returns a slice that borrows from the input buffer rather than copying -- the decoded Person.name points straight into buf, so the caller must keep buf alive as long as the Person. That is a deliberate zero-copy decision, the same lifetime contract our MQTT readString had, and Zig forces you to be conscious of it because slices carry no ownership of their own. Second, the else => skipField arm is the whole forwards-compatibility story in one line: ship a Person with a brand-new field 4 to this old decoder and it reads id, name, email, then steps cleanly over field 4 without so much as a hiccup.

Comptime: generating the encoder for free

So far we hand-wrote encodePerson. That's fine for one struct, tedious for fifty, and tedious-plus-error-prone is exactly the smell that should make you reach for Zig's comptime. Back in episode 32 we used @typeInfo to walk a struct's fields at compile time; we can do the same here to generate a generic encoder that assigns field numbers by declaration order and picks the wire type from each field's Zig type. No code generation step, no build plugin -- the compiler does it while compiling.

/// Encode any struct: field N is the Nth declared field (1-based).
pub fn encodeStruct(alloc: std.mem.Allocator, value: anytype) ![]u8 {
    var out: std.ArrayListUnmanaged(u8) = .{};
    errdefer out.deinit(alloc);

    const fields = @typeInfo(@TypeOf(value)).@"struct".fields;
    inline for (fields, 1..) |field, field_num| {
        const fv = @field(value, field.name);
        switch (@typeInfo(field.type)) {
            .int => {
                try writeVarint(&out, alloc, encodeTag(field_num, .varint));
                try writeVarint(&out, alloc, @intCast(fv));
            },
            .pointer => |ptr| {
                // []const u8 -> a length-delimited bytes field
                if (ptr.size == .slice and ptr.child == u8) {
                    try writeVarint(&out, alloc, encodeTag(field_num, .len));
                    try writeVarint(&out, alloc, fv.len);
                    try out.appendSlice(alloc, fv);
                } else @compileError("unsupported pointer field: " ++ field.name);
            },
            else => @compileError("unsupported field type: " ++ field.name),
        }
    }
    return out.toOwnedSlice(alloc);
}

The inline for unrolls at compile time, so field_num is a comptime-known constant in each iteration and the whole switch on field.type collapses to straight-line code per field -- there is no runtime reflection cost, the dispatch happened during compilation. The @compileError arms are the lovely part: hand this function a struct with an unsupported field type and you get a clear message at build time, not a mystery at runtime. That is the same "make illegal states a compile error where you can" philosophy from episode 6, lifted up to whole-type level. A real protobuf library marries this reflection to a .proto-derived field-number map, but the engine is precisely what you see here.

Testing a binary format

As with MQTT, the protocol code is mostly pure functions, and the single highest-value test for any serializer is the round-trip: encode, decode, assert you got the original back. If encode and decode ever disagree, the round-trip fails loudly without you having to eyeball hex dumps.

test "person round-trips" {
    const alloc = std.testing.allocator;
    const original = Person{ .id = 1986, .name = "scipio", .email = "[email protected]" };

    const bytes = try encodePerson(alloc, original);
    defer alloc.free(bytes);

    const back = try decodePerson(bytes);
    try std.testing.expectEqual(original.id, back.id);
    try std.testing.expectEqualStrings(original.name, back.name);
    try std.testing.expectEqualStrings(original.email, back.email);
}

test "unknown field is skipped, known fields survive" {
    const alloc = std.testing.allocator;
    var buf: std.ArrayListUnmanaged(u8) = .{};
    defer buf.deinit(alloc);
    // field 1 = id 7
    try writeVarint(&buf, alloc, encodeTag(1, .varint));
    try writeVarint(&buf, alloc, 7);
    // field 99 = some future varint the old decoder never heard of
    try writeVarint(&buf, alloc, encodeTag(99, .varint));
    try writeVarint(&buf, alloc, 123456);

    const p = try decodePerson(buf.items);
    try std.testing.expectEqual(@as(i64, 7), p.id); // known field intact
}

That second test is the one I'd be proudest of -- it proves the forwards-compatibility claim rather than just asserting it. Beyond round-trips, point a fuzzer (episode 12's habit) at decodePerson with random bytes and confirm it only ever returns a value or a clean error, never reads out of bounds. The error.Truncated guards in readLen and skipField are exactly what a fuzzer hammers on, and they are exactly where a sloppy C decoder grows a buffer-overread CVE.

Performance considerations

Protobuf is already lean, so the wins are in how you drive it. First, the encoder above grows an ArrayListUnmanaged and reallocs as it goes -- fine for occasional messages, wasteful in a hot loop. A serious encoder either computes the exact size first and allocates once, or keeps a reusable buffer warm with clearRetainingCapacity (the trick from the allocator episodes) so a firehose of small messages allocates zero times after warm-up. Second, the zero-copy decode we built is the right default: borrowing slices out of the input buffer beats memcpy-ing every string into fresh allocations, as long as you respect the lifetime. Third, the fixed-width wire types (i32, i64) parse faster than varints because there's no per-byte branch -- if a field is always large, fixed64 can actually beat int64 on both size and speed.

Having said that -- do not optimise what you have not measured. Episode 34's profiler will tell you whether your time goes to the varint loop, the allocations, or (as ever) the syscalls moving bytes in and out. My money, as last time, is on the syscalls.

How this compares to C, Rust, and Go

In C, you'd reach for protobuf-c or nanopb (the latter aimed squarely at microcontrollers). Both generate structs and hand-written-style encode/decode functions, and both make you manage every buffer by hand -- the varint-and-length dance we wrote is exactly the code those libraries generate, bounds checks and all, which is sobering when you remember a forgotten check there is a remote read primitive.

In Go, google.golang.org/protobuf generates a struct per message and uses reflection at runtime to encode -- ergonomic, but the reflection and the garbage collector both cost you on a million-message-per-second path, which is why Google also ships a faster codegen mode.

In Rust, prost generates plain structs and derives the codec, with the borrow checker enforcing the zero-copy lifetimes that we had to remember by hand. It's arguably the most foolproof of the three, at the usual cost of fighting lifetimes when a decoded value needs to outlive its buffer.

Zig sits where it likes to sit: we wrote the entire varint, ZigZag, tag and message codec in a couple hundred lines, every allocation is visible, every bounds check is ours, and comptime gave us the reflection-driven generic encoder with no runtime reflection cost and no separate codegen tool. For production you'd still likely run a generated library against the real .proto schema -- but now you know exactly which bytes it's writing, and why ;-)

Where this is heading

Stand back and look at the kit we've assembled: a tag that fuses field number and type, a 64-bit varint, ZigZag for signed values, length-delimited framing for strings and nested messages, a decoder that skips what it doesn't understand, and a comptime engine that builds an encoder from a struct definition. That's a real serialization format, schema-driven and forwards-compatible, sitting on the same byte-handling instincts we've sharpened across the whole networking arc.

Protobuf needs a schema both ends agreed on in advance -- that's its strength and its constraint. But there's a whole other family of binary formats that throw out the pre-shared schema and instead make the bytes self-describing, carrying their own type information inline so a decoder can reconstruct the data with nothing agreed up front. The varint you've now written twice is going to make a third appearance, and the tag-then-value rhythm will feel awfully familiar -- but the tradeoff between "schema-driven and tiny" versus "self-describing and flexible" is the design tension worth chewing on before the next instalment.

The pieces were never separate tricks. The varint, the length prefix, the skip-the-unknown rule, the comptime reflection -- they're the vocabulary of every serialization system you'll ever touch, and you've now written each one yourself with your eyes wide open.

Exercises

  1. Add a repeated field. Extend Person with phones: []const []const u8 and teach encodePerson/decodePerson to handle field 4 appearing multiple times on the wire (protobuf encodes a repeated string as the same tag repeated). On decode, collect every field-4 occurrence into an ArrayListUnmanaged([]const u8). Write a round-trip test with three phone numbers.

  2. Encode a nested message. Give Person an address: Address field (where Address has a street and a city). Since a nested message is just a length-delimited field whose payload is another encoded message, encode the Address into its own buffer first, then write it as field 5 with wire type .len. Decode it back by slicing out those bytes and calling decodeAddress on them. This is recursion at the wire level -- prove it round-trips.

  3. Write a generic decoder with comptime. You wrote encodeStruct using @typeInfo; now write the matching decodeStruct(comptime T: type, buf: []const u8) !T that walks fields by number and fills the struct, using the same inline for over @typeInfo(T).@"struct".fields. Handle the .int and []const u8 cases, and make sure unknown field numbers still hit skipField. Test it against the Person produced by encodeStruct.

Thanks for reading -- de groeten, and see you in the next one!

@scipio

stem stemsocial steemstem zig programming
0
0
0.000
0 comments