Crash recovery
What happens when the process dies in the middle of a capture.
The question a skeptical reader should ask: if the server dies at the worst moment, can the winner be charged twice, or not at all?
Crash recovery: charged exactly once
Scroll the figure sideways, or open it full size.
Text version of this diagram
The capture sequence is: read the truth of the hold, save the intent (capture, amount, request id), call PayPal, save capture_ok.
- Crash A, after the intent is saved and before the call: on restart the truth shows no capture, and the saved intent is sent again with the same request id.
- Crash B, after PayPal took the capture and before the answer was saved: on restart the truth shows the capture, which is adopted. If the truth cannot be read, or PayPal does not know the hold, the settlement waits with the same request id for ten minutes and then goes to FAILED_NEEDS_REVIEW. It never tries the next bidder.
- Crash C, after capture_ok was saved: the state is already CAPTURED, so the restart goes on to void the other holds without a capture call.
Result: the winner is charged once.
The rule behind all three cases
On every run the saga reads the truth at PayPal first, then acts. A capture that is already there is adopted. Only when PayPal shows no capture is the saved intent sent again, with the same request id.
| Crash point | What the journal holds | What the restart does |
|---|---|---|
| A: after the intent, before the call reached PayPal | Intent saved, state CAPTURING | Truth shows no capture, so the same intent is sent again with the same request id |
| B: after PayPal took the capture, before the answer was saved | The same intent, no outcome | Truth shows the capture (invoice id and amount match), so it is adopted. No second call |
C: after capture_ok was saved | State CAPTURED, winner recorded | Goes on to void the other holds. No capture call |
If the truth cannot be read, or PayPal does not know the hold, that is not an answer: the intent stays, no other candidate is tried, the unknown clock runs, and after ten minutes the settlement goes to FAILED_NEEDS_REVIEW for a person.
The restart, minute by minute
The first diagram shows what each crash leaves behind. This one follows the restart that comes after it. Pick a crash point, then step through what the server does.
Restart timeline: from a crash back to one charge
Scroll the figure sideways, or open it full size.
Finish. The other holds are voided, three at a time, and the settlement ends SETTLED. Each resume pass also applies stored webhook events that have no result and sweeps abandoned holds.
docs/evidence/SIM/SIM-MULTIDAY.md): 10 variants, every restart ends with one charge. The restart stages are read from packages/server/src/runtime/auction-service.ts, boot/compose.ts, packages/payments/src/saga/create.ts, saga/trail.ts and saga/capture.ts. One in-process crash at the same place (R10) passed on the real sandbox; killing a real server process on a deployed host has not been run.Text version of this diagram
Crash points injected by the simulation, in order: journaled:begin, journaled:intent, before:capture, after:capture, journaled:capture_ok.
- journaled:begin: journal holds state CAPTURING, no intent; PayPal holds no capture. The journal only says the capture phase began. The restart does the whole phase, intent first.
- journaled:intent: journal holds intent saved: capture, amount, request id; PayPal holds no capture (call not sent). The intent was saved before the call, so the restart sends exactly that call. Nothing is rebuilt.
- before:capture: journal holds intent saved: capture, amount, request id; PayPal holds no capture (died just before the call). From the journal this looks the same as the previous point. The truth read tells them apart from point 4.
- after:capture: journal holds intent saved, no outcome; PayPal holds capture taken, the answer was lost. The worst moment. The truth read finds the capture, so the winner is charged once. This holds even if PayPal forgot the request id (simulated S6 variants).
- journaled:capture_ok: journal holds state CAPTURED, winner set, intent cleared; PayPal holds capture taken. The capture is finished and saved. The restart goes straight on to releasing the other holds.
Restart stages: resume() runs at boot and every 60 seconds; a lease of a killed process holds until it expires (320 seconds by default); the journal is read and missing audit rows are appended; the truth of the hold is read at PayPal; the decision follows from the crash point; the other holds are voided and the settlement is settled, and stored webhook events and abandoned holds are swept.
The server calls resume()
compose.ts calls service.resume() once when the host boots. The service then calls it again on a timer, every 60 seconds by default.
The lease may still be held
A settlement run takes a lease so that two runners never work on one auction. A process that was killed cannot give it back, so the new process is told "busy" until the lease expires. The default lifetime is 320 seconds: twice the longest a retried money call can take (four attempts of ten seconds plus the Retry-After waits), plus a minute. The next 60 second pass tries again. In the simulation an exception releases the lease on the way out, so those restarts do not show this wait.
The journal and its audit trail are caught up
The journal is read from disk. A transition is saved first and its audit row is written after, so a crash between the two could lose the row. The journal history is the source: the run starts by appending the rows above auditedSeq, before any other effect.
The truth is read, then the decision follows
PayPal is asked about the hold before anything is sent. Depending on where the crash was, the saved intent is sent again, a capture that is already there is adopted, or no capture call is made at all.
The settlement finishes and the sweeps run
The other holds are voided, three at a time, and the settlement ends SETTLED. Every resume pass also applies webhook events that were stored but have no result, and sweeps abandoned holds and orphan voids.
The server side
The host calls resume() on start and then on a timer (every 60 seconds by default). It loads every known room, continues unfinished settlements, applies stored webhook events and retries orphan voids. The audit trail rows of a transition are recovered from the journal history, so a crash between a save and its audit row loses nothing.
What has been tested
Where this lives in the code
- packages/payments/src/saga/capture.tstruth first, write-ahead intent, crash boundaries
- packages/payments/src/saga/trail.tsaudit rows recovered from the journal
- packages/payments/src/saga/create.tsthe settlement lease and its lifetime
- packages/server/src/boot/compose.tsservice.resume() at boot
- packages/server/src/runtime/auction-service.tsresume() and the resume timer
- packages/server/src/boot/checkpoint-kill.tstest hook that kills the process at a boundary
- docs/evidence/SIM/SIM-MULTIDAY.mdscenario S6
Last updated on