Rex Liu

Money in, tickets out

Building ticketing for my chapter's events, where the only number that really matters is how many tickets each payment produces.

Most ticketing sites hold your seat while you type in your card number. Ours doesn't, and that was the first real decision I made when I started building ticketing for my chapter's events.

I was building for a few hundred to a couple thousand tickets per event, and at that size a reservation system costs more than it saves. Holds need expiry, expiry needs a sweeper, the sweeper needs to be right about time, and every one of those can strand a seat that nobody is actually buying. So the checkout page shows how many tickets are left, the buyer pays through Stripe Checkout, and the tickets are issued after the money arrives. If two people race for the last ticket, one of them gets it and the other gets an automatic refund. I'd rather refund one person occasionally than run a hold system every day.

That moves the whole correctness problem into one place: the moment after payment, when the server turns an order into tickets. Everything else on the site can be a little wrong for a while. That moment has to produce exactly the right number of tickets, once.

My first version, on the second day, checked capacity at issuance with a conditional update. Bump the ticket type's sold count only if the new count fits, and if no row comes back, the type is sold out. That's atomic and I was happy with it. It took me about a week to see what it didn't cover. An event has its own capacity on top of each ticket type's, a room only holds so many people whether they bought general or early-bird, and my event-level check was a plain count of sold seats with nothing stopping two transactions from counting at the same time. Each ticket type could be individually fine while the event as a whole oversold.

A per-row conditional update can't guard a limit that spans rows, so I switched to locks. Issuance now takes the ticket type row and then the event row FOR UPDATE, counts with both held, and either issues or fails the order and refunds. In spirit, with names that aren't the real ones:

begin;
select total, sold from ticket_type where id = $1 for update;
select capacity from event where id = $2 for update;
select coalesce(sum(seats), 0) from ticket where event_id = $2;
-- fits?  insert tickets, bump sold          : commit
-- full?  mark the order failed, queue refund : commit

I wrote regression tests for the capacity paths alongside the locks.

What Stripe actually promises#

Stripe tells you a payment happened by calling your webhook. It promises to deliver every event at least once, which means sometimes twice, sometimes out of order, and sometimes again an hour later because your server said 500 the first time. The handler has to turn "at least once" into "exactly once" by itself.

My first version did the textbook thing. Record each event id in an idempotency table, and skip any event you've already seen. I didn't sit down and ask what that table actually guaranteed until Christmas, and the answer was less than I'd assumed, in three different ways.

The check was check-then-insert: look for the event id, and if it isn't there, insert it and process. Two deliveries of the same event arriving together could both look, both find nothing, and both process. I replaced it with a single INSERT ... ON CONFLICT DO NOTHING. If no row comes back, someone else already claimed the event, and this delivery stops.

I wrote the record before the handler ran, and left it there if the handler then failed. Stripe would retry, see the event already recorded, and skip it forever. A database blip at the wrong second meant a paid order that never got its tickets, and no retry would ever fix it. So if the handler fails, it now deletes its own record, and the retry gets a real second chance.

And nothing locked the order itself. Two different events about the same payment (Checkout says complete, the payment intent says succeeded) could each check that the order had no tickets yet and each issue a set. Now the order row is locked before that check, so the second one waits, sees the tickets, and does nothing.

While I was in there I added the checks I should have had from the start. The payment status has to actually be paid, and the amount and currency have to match the order in the database. I also extended the stuck-payment job so it sweeps for orders marked paid with zero tickets and issues them. It's the belt to go with the suspenders. If everything above works, it never finds anything.

I did most of that on Christmas night, and I found all of it by reading the handler slowly, with Stripe's delivery guarantees open in another tab, and asking at every line what happens if this runs twice at the same time. I was quite pleased with myself. That lasted about two weeks.

One of the Christmas guards was the obvious one. Some order states look final. A refunded order is finished, and so, I decided, is a failed one. If a success event shows up for an order that already failed, it must be late or replayed, so ignore it. I even wrote tests saying a failed order must never be fulfilled by a late webhook, which means the mistake was deliberate, specified and tested.

What I'd gotten wrong is what Stripe means by failed. payment_intent.payment_failed reports that one attempt failed: a declined card, a 3-D Secure challenge that didn't go through. The payment itself isn't over. The buyer can try another card on the same payment, and if that works, Stripe sends the success events for the same order. The sequence I'd missed was the ordinary one:

card declined          ->  payment_failed     ->  order marked failed
buyer tries a new card ->  payment succeeded  ->  "order is failed, skip"

Charged, marked failed, no tickets. That needs nothing more than a declined card and a second try, which is one of the most ordinary things that happens at a checkout. The sweep I'd added at Christmas would have walked straight past it too. It looks for orders still pending and for orders marked paid with no tickets, and this order was marked failed, so the belt and the suspenders had the same blind spot.

In January I turned it around. A success event for a failed order now recovers it: log a warning, mark it paid, issue the tickets. Then I inverted the three tests that pinned the old behaviour, which is a strange feeling. They were careful, thorough tests of exactly the wrong idea.