Operator runbook
Deploy, set secrets by name, check the result, and what to do when something goes wrong. Names only, never values.
This page is for the person who runs the server. It is built from docs/DEPLOY.md, docs/RUNBOOK-GATES.md, docs/RUNBOOK-BACKUP.md and infra/. Everything points at the PayPal sandbox.
Roles
| Role | Can do |
|---|---|
| Owner | Change repository secrets and the sandbox-nightly, staging and production environments, use the PayPal developer dashboard, run anything against the real sandbox from a laptop |
| Contributor | Read evidence and workflow logs, run the fake PayPal and the text-level voice eval, open pull requests. Never receives secrets |
Deploy
Apply the blueprint
In Render choose New, Blueprint, pick the repository and apply render.yaml. It describes one web service, gavel-host, with one instance and a 1 GB disk at /var/data. Health check path /healthz. Auto deploy is off. The plan must allow a persistent disk.
Fill the variables the blueprint leaves empty
Names only, set in the service Environment tab (see the table below).
Wire GitHub to Render
Copy the Render deploy hook. In GitHub, environment staging: secret RENDER_DEPLOY_HOOK_URL, optional secret ADMIN_KEY (same value as on Render, enables the create-auction smoke check), variable STAGING_URL.
Register the webhook
In the PayPal developer dashboard (sandbox app, Webhooks) add https://<api-host>/webhooks/paypal for PAYMENT.AUTHORIZATION.VOIDED, PAYMENT.AUTHORIZATION.DENIED, PAYMENT.CAPTURE.PENDING, PAYMENT.CAPTURE.COMPLETED, PAYMENT.CAPTURE.DENIED and PAYMENT.CAPTURE.DECLINED. Other types are acknowledged and ignored. Copy the webhook id into PAYPAL_WEBHOOK_ID on Render only.
Run the deploy workflow and read the smoke output
Run deploy-staging by hand. It calls the deploy hook, waits 60 seconds, then runs scripts/smoke.sh.
Deploy the web app
Create a Cloudflare Pages project, build with npm ci && npm run build -w @gavel/web and pass VITE_API_BASE and VITE_WS_BASE. Add the Pages origin to CORS_ORIGINS.
Production uses the deploy-prod workflow: tag the commit or run it by hand, approval on the production environment (needs a required reviewer), deploy only if the commit is the head of main, poll GET /version until it serves the commit, then run the smoke test. If the commit never appears, the previous deploy is restored. Details: infra/rollback.md.
Secrets and variables, by name
Prop
Type
The full list the server reads is .env.example in the repository root. Never paste a value into an issue, a log or a pull request.
Post-deploy checks
| Check | Expected |
|---|---|
GET /healthz | 200 while the process runs |
GET /readyz | 200 when migrations are applied and the store answers, else 503 |
GET /warmup | 200 when the warm steps finish within 3000 ms, else 503 with a per-step report |
GET /version | {"sha":"<commit>"} of the deployed commit |
unsigned POST /webhooks/paypal | 400 |
POST /api/auctions without a key | 401 (201 with the admin key) |
By hand: ADMIN_KEY=... scripts/smoke.sh https://<api-host>. The smoke can pass against the old instance if it answers before the new build is live.
The uptime workflow probes the three health routes every 5 minutes. It stays off until the variables UPTIME_ENABLED (true) and PRODUCTION_URL are set. Two failed rounds in a row raise an alert.
What to do when
- Check the webhook list in the PayPal developer dashboard: the URL, the subscribed events and recent deliveries.
- Confirm
PAYPAL_WEBHOOK_IDon Render is the id of that webhook. A mismatch makes every verification fail. - An unsigned post must answer 400. A real event that fails verification is rejected, so a missing id shows up as rejected deliveries.
- Event ids are deduplicated, so a redelivery is safe. How the server treats events: webhooks.
- The retry behaviour of PayPal is UNVERIFIED (G4).
Review means the saga stopped on something it will not decide alone (an invariant break, an unknown capture that outlived ten minutes, a mismatch, or a void that failed five resumes). See saga states.
- Read what PayPal knows. List the authorizations of the sandbox account and compare with the journal export (
GET /admin/export?kind=payments&auction=<id>, admin key). - To release holds the review is keeping, call
POST /ops/auctions/<auctionId>/release-reviewwith the admin key. To void every hold of a hammered auction that is not captured, callPOST /ops/auctions/<auctionId>/void-all. Both are audited with the actoradmin, and answer 409busywhile a run holds the settlement lease. - Do not edit the journal by hand.
A timeout or 5xx on capture is an unknown answer, not a failure. The saga keeps the settlement in CAPTURING, never moves to the next bidder, and sends the same request id on every retry. After unknownDeadlineMs (600000, ten minutes) it goes to review. Wait for the deadline, then handle it as a review. See crash recovery.
Restart it on the same data directory (DATA_DIR, the disk at /var/data). The journal is saved before each effect, so the runner resumes. A resume that was cut off while holding the settlement lease waits for the lease time instead of losing it. Check /readyz, then the journal export.
Voice bids always go through the confirm prompt (G5). Without DEEPGRAM_API_KEY or GROQ_API_KEY the /audio socket falls back to cached transcripts (VOICE_STT unset means replay). For a demo use /replay/:recordingId, which makes no PayPal call. See voice and confirm.
Two failed rounds raise the alert. Check /healthz first (process alive), then /readyz (store), then /warmup (it reports which step is cold). On the free tier the host sleeps; the keep-alive choice is decision D-0008 in infra/uptime.md. Warmup under 3 s after 15 minutes idle is not measured yet.
Run deploy-prod by hand with rollback set to true (or pass rollback_deploy_id). The job restores the newest earlier deploy built from a different commit and waits until /version serves it. The disk is not rolled back: a release that changed the stored data format also needs a data restore.
- Stop the server. Download the newest
ledger-snapshot.tgz.gpgartifact from thebackupworkflow and decrypt it with the private key kept off GitHub:gpg --decrypt ledger-snapshot.tgz.gpg | tar -xz. - The snapshot holds ledgers (
<auctionId>.ndjson, chain-checkable),payments/<id>.json(journal and holds),roomlog/<id>.ndjsonandpayments-global.json(processed webhook ids, kill switch state). It is a JSON view, not a byte copy: rebuildjournals.json,holds.jsonandwebhooks.jsonby hand or script. There is no import route. - The snapshot can be a day old. After a restore, reconcile against PayPal and void every authorization no restored record accounts for. Keep the kill switch engaged until then.
- This procedure is documented, not exercised. See backup.
Socket upgrades and HTTP joins share one per-address bucket. Raise RATE_JOIN_BURST (default 40; refill RATE_JOIN_PER_MIN, default 10). Each address may hold at most 20 open sockets.
- docs/DEPLOY.mdstaging deploy
- infra/rollback.mdproduction deploy and rollback
- infra/uptime.mdprobes and alert rule
- render.yamlthe Render blueprint
- .env.exampleevery variable name the project reads
- packages/server/src/http/ops.tsadmin ops routes
Last updated on