Rex Liu

Ten milliseconds of cover

libjzx again: putting the actor loop to sleep properly, and the wakeup a timeout had been hiding.

libjzx is the actor runtime in C I started in November, with Erlang-style supervisors (the first post is about those). From the start I'd claimed two things about it: that it had no hidden threads, so all the concurrency it introduces is explicit, and that it ran on libxev, Mitchell Hashimoto's event loop.

Neither was true. When I wrote the supervisors in November, I ran timers on a background pthread that pushed due messages into the cross-thread queue, and when the run queue was empty I had the loop sleep for a millisecond with nanosleep and look again. I'd used libxev exactly once, as a pointer in a struct that nothing touched. As a first version the nap is fine, since anything that arrives during it gets picked up on the next tick. It also means an idle loop wakes up a thousand times a second to find nothing to do.

I came back in December to make the loop sleep properly. The first attempt was a pipe. jzx_send_async writes a byte to it after enqueueing a message, and the idle loop blocks in poll on the read end, with a 10 ms timeout as a backstop. It's the self-pipe trick moved across threads. D. J. Bernstein recalls coming up with the original around 1990, for signal handlers and a closely related problem: a signal that arrives while select() is starting, "too early to interrupt it, too late to change its timeout." Remember that sentence.

The same evening I moved the loop onto libxev for real. It has a proper cross-thread wakeup in xev.Async: any thread calls notify(), and a loop blocked in run wakes up. I turned the idle branch into one blocking call that returns when some event fires, and since libxev could do the waking, I dropped the pipe and its timeout.

Then the loop could hang with a message sitting in its queue. Each tick starts by draining the cross-thread queue, then makes one non-blocking pass over I/O, and that pass also consumes any pending wakeup. So:

loop thread                          sender thread
-----------                          -------------
drain async queue (empty)
                                     enqueue message
                                     notify()
run xev, no wait   <- eats the notification
run actors (nothing to do)
idle: run xev, block   <- message is queued, nobody left to wake us

The notification arrives too late for the drain and is used up before the block. The message waits until something unrelated wakes the loop, which on a quiet system might be never. It's Bernstein's sentence with my own loop in place of select().

And it had been there since the pipe. The non-blocking pass read the pipe too, so a byte written in that same gap got consumed early, and the loop went to sleep on an empty pipe with a message in the queue. With the pipe, though, the sleep could only last 10 ms before the timeout came round and the next tick found the message. The race cost me at most ten milliseconds, so I never saw it. Once I took the timeout away, the same race became a hang.

The fix is one check before blocking:

if (jzx_async_has_pending(loop)) {
    continue;
}
jzx_xev_run(loop->xev, 1);

If anything is queued, go round again. A send that lands after the check still calls notify(), and that notification can only be consumed by the blocking run, which it wakes. The rule underneath is old. Lampson and Redell's paper on Mesa's monitors (1980) calls a notification "a hint to a waiting process", and has the waiter test the condition itself when it resumes. My loop did test it, by draining the queue at the top of every tick. It tested too early. The check has to come after the last thing that can consume the wakeup, right before you block.

So the timeout covered for me for exactly as long as the pipe existed, which was one evening.

The timer thread is still there, so "no hidden threads" still isn't true. Getting rid of it means moving timers onto the libxev loop itself, and I haven't done that yet.