Harold KimandPaperclip
98df214917
When reusing a Codex sandbox, the workspace-restore step now performs an auth-merge check before adopting the sandbox's auth.json. The merge fails closed when: - host or sandbox Codex auth is unusable (missing token or account) - the subscription payload shapes differ between host and sandbox - subscription account identities do not match Additional hardening: node absence on PATH during the embedded decision script now exits non-zero instead of silently falling through. Co-change comments link parseAuth (the embedded script) to hasUsableAuthPayload (codex-home.ts) so a future auth-format change cannot silently diverge between the two implementations. Co-Authored-By: Paperclip <noreply@paperclip.ing>
3.2 KiB
Dev-Plane Deploy & Restart Hygiene
Operational runbook for restarting a dev/shared Paperclip control plane without killing in-flight agent work. Written after the 2026-07-06/07 failure spike, where a restart-heavy deploy window was the single largest source of failed tasks.
Why this matters
Every control-plane restart hard-kills any heartbeat run in flight at that moment. The run is finalized as failed with error_code = 'process_lost'. There is an automatic single retry (process_lost_retry), but it is best-effort: in the 07-06/07 incident window only 12 of 36 lost runs recovered. On the evening of 07-06, 9 restarts in 90 minutes killed 16 in-flight runs.
Rules of thumb
-
Batch your deploys. Stack up changes and restart once, instead of restart-per-change. A restart storm (several restarts within an hour) multiplies run loss for near-zero benefit.
-
Restart during low fleet activity. Check for active runs before restarting; prefer windows where the fleet is quiet. A quick check against the control-plane DB:
SELECT count(*) FROM heartbeat_runs WHERE status = 'running'; -
Drain before restart (upcoming). Graceful SIGTERM drain — stop accepting new runs, let in-flight runs finish or checkpoint, then exit — is being added in PAP-12930. Once it lands, send SIGTERM and wait for drain instead of hard-restarting. Until then, rule 2 is your drain.
-
After any restart, glance at the damage. See the detection queries below; confirm lost runs either retried successfully or get manual follow-up.
How to spot a restart burst
Two signals, cross-referenced:
1. Server start markers in the instance log. Each boot logs Server listening on <host>:<port>. Logs live at ~/.paperclip/instances/<instance>/server.log, rotated daily to server.log-YYYYMMDD.gz.
grep -h "Server listening" ~/.paperclip/instances/default/server.log
zgrep -h "Server listening" ~/.paperclip/instances/default/server.log-20260706.gz
Many markers minutes apart = restart burst.
2. Same-minute process_lost clusters in heartbeat_runs. Runs killed by a restart are finalized together, so they cluster on the same minute:
SELECT date_trunc('minute', finished_at) AS minute, count(*)
FROM heartbeat_runs
WHERE error_code = 'process_lost'
GROUP BY 1 HAVING count(*) > 1
ORDER BY 1 DESC;
If the cluster minutes line up with the Server listening timestamps, the failures are restart-inflicted, not a product regression.
Note: query the DB directly for this — the /heartbeat-runs list API ignores the status= filter and omits error fields.
Checking recovery after a burst
Each process_lost run should have triggered one retry wake (reason = 'process_lost_retry'). To find lost runs that never recovered, look for process_lost failures with no subsequent successful run for the same issue, and re-wake or reassign those issues manually.
Related failure modes (not restart-caused)
Seen in the same incident window; do not confuse them with restart damage:
workspace_validation_failed— deterministic workspace validation retry loops (self-heal: PAP-12931).- Provider quota exhaustion — dominant cause of
claude_transient_upstreamfailures (quota-aware handling: PAP-12932).