Rex Liu

A log nobody read

SydraDB again: the write-ahead log I promised on day one, and the half of it I hadn't written.

SydraDB is the time-series database in Zig I started in September (the first post is about how it passed for Postgres). One of the things I wanted from it on the first day was a crash-safe write-ahead log in front of columnar segments.

A write-ahead log is a promise made of two halves. On the way in, every write is appended to the log before anyone is told it succeeded, so it's on disk before it's anywhere else. On the way back up after a crash, the database reads the log and replays whatever hadn't yet made it into its real storage. The first half costs you a little on every write, and the second half is the reason you pay it.

The appending is what I'd written. Each record is a length, a type, the series id, a timestamp and a value, with a CRC32 at the end:

// WAL v0: record = [u32 len][u8 type][u64 series_id][i64 ts][f64 value][u32 crc32]

I gave it a policy for when to fsync: after every record, on an interval, or never. Points went to the log and into an in-memory table, and every two seconds, or whenever the table passed 8 MiB, a flush wrote it out as segment files, one per series per hour, and recorded them in a manifest. On startup the engine loaded the manifest, so flushed segments survived a restart.

And this is how I opened the log on startup:

const f = try data_dir.createFile("wal/current.wal", .{ .read = true });

In Zig, createFile truncates by default. The standard library says so in the comment above the flag: "If the file already exists, and is a regular file, and the access mode allows writing, it will be truncated to length 0." So every start of SydraDB began by emptying the log, and nothing would have read it anyway, because there was no replay. The second half didn't exist. I'd planned an append-only log writer and reader with checksums, and built the writer and the checksums.

What was at risk was small, whatever had arrived since the last flush, a couple of seconds' worth with the defaults. That was also exactly the data the log was for. Anything older was already in a segment and didn't need the log at all.

When I came back to SydraDB a month later, in October, the first thing I did was make it read its own log. I changed the open so it stopped truncating:

var f = data_dir.openFile("wal/current.wal", open_flags) catch |err| switch (err) {
    error.FileNotFound => try data_dir.createFile("wal/current.wal", .{ .read = true }),
    else => return err,
};
try f.seekFromEnd(0);

Then I wrote the replay. The question it has to answer is which records in the log are already in a segment. A flush doesn't clear the log, which only rotates every 64 MiB, so most of what's in it has been flushed already, and replaying those points would store them twice. I answered it with a high-water mark per series. Before reading the log, the engine walks the manifest and remembers, for each series, the latest timestamp any segment holds. During replay, a record at or below its series' mark is skipped, and everything newer goes back into the in-memory table, which is flushed straight away.

The test I wrote with the fix checks the crash safety I'd wanted from the beginning. It appends two points to a log, closes the log without flushing anything, starts a fresh engine on the same directory, and asks for the series back. Both points are there.