Rex Liu

Everything is a message, including dying

libjzx, an Erlang-flavoured actor runtime in C, and supervisors built from nothing but mail.

I started libjzx in November to learn concurrency from the bottom up, and the part I most wanted to understand was Erlang's supervision. A process does one job and is allowed to crash. A supervisor above it notices the crash and restarts it, following a policy written down ahead of time, and the system stays up because failure always has somewhere to go. Joe Armstrong's thesis, Making reliable distributed systems in the presence of software errors (KTH, 2003), is the long explanation of why that works.

I wanted it in C, for the things I was already building: SydraDB, trading infrastructure, agents. It had to be infrastructure I'd actually use. So: one process, one scheduler thread, lots of small actors, and supervisors with Erlang/OTP's restart rules. one_for_one, one_for_all and rest_for_one; children that are permanent, transient or temporary; restart intensity limits; backoff.

I spent the first evening on plumbing: an actor table, a ring-buffer mailbox per actor, a run queue with fairness budgets so one chatty actor can't starve the rest, and a way for other threads to send messages in. Actor ids are a generation number in the high 32 bits and a table index in the low 32. When an actor dies, its slot's generation goes up, so any id still pointing at the old occupant stops resolving:

static jzx_actor* jzx_actor_table_lookup(jzx_actor_table* table, jzx_actor_id id) {
    uint32_t idx = jzx_id_index(id);
    if (idx >= table->capacity) {
        return NULL;
    }
    if (table->generations[idx] != jzx_id_generation(id)) {
        return NULL;
    }
    return table->slots[idx];
}

It looks like housekeeping. The supervision code I wrote the next day leans on it.

I wanted supervision without a second mechanism living next to the actors: no separate tree of structs for the runtime to walk, no callbacks. A supervisor in libjzx is an ordinary actor with some extra state, and everything it needs to know arrives as a message. When a child dies, its teardown sends the supervisor a JZX_TAG_SYS_CHILD_EXIT through the same mailbox as everything else. The supervisor decides whether that child should come back (permanent always does, transient only if it failed, temporary never), checks the intensity window, and applies its strategy. A restart delay is a message too. The supervisor mails itself, timed to arrive when the delay is up:

jzx_err err = jzx_send_after(loop,
                             sup_actor->id,
                             delay_ms,
                             payload,
                             sizeof(jzx_child_restart),
                             JZX_TAG_SYS_CHILD_RESTART,
                             NULL);

When JZX_TAG_SYS_CHILD_RESTART arrives, it spawns the child.

The first version of my exponential backoff doubled exactly once. The delay before the nth restart is meant to be the base delay times 2ⁿ, so the supervisor keeps a restart count for each child: bump the count, compute the delay, schedule the respawn. The respawn went through the same function that spawns a child the first time, and that function started by zeroing the child's bookkeeping, restart count included. So the count went 1 at the first failure, back to 0 when the child came up, 1 at the next failure, and every restart waited exactly twice the base. The other backoff mode, which adds a fixed step for every restart so far, was stuck the same way. I deleted the line that day.

There are two different things called "the child" here. One is the running actor, which lives until it crashes and whose id must never be reused. The other is the supervisor's record of that child, the spec it was started from and the count of times it has died, which has to outlive every copy of the actor. I'd been resetting the second whenever I made a new first.

The id rule is what keeps the exit handling short. Take one_for_all: one child fails, so the supervisor stops every sibling and schedules fresh copies. The stopped siblings' own exit messages are still on their way, and they can arrive after the replacements exist. The supervisor matches each exit against the exact ids of its current children, finds nothing, and drops it. A replacement can land in the slot its predecessor used, but it gets the next generation, so its id is new and a stale exit can't be mistaken for it.

All of this leans on a decision I made the first evening about who owns a message. The runtime never frees a payload, only the small message shell around it, and whoever receives the data frees it or passes it on. That's very C, and it makes every handler responsible for its own memory. It also means a supervisor reads top to bottom as one function that switches on message tags.

Making supervisors ordinary actors paid off once more, at the top of the tree. If children die faster than the intensity limit allows, the supervisor stops all of them and fails itself. A failed supervisor is torn down like any other actor, so the parent it was spawned with receives the same exit message a dying child would send. Reporting failure upward took no new code. Dying was already a message.