mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
b1e70ab6dd80a3ec45144f9c32b4f48ad874bbfd
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
500bc694b5 |
fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A release swap runs two servers for a short overlap. The old server drains, and the new server takes over the runs. > - Each server proves that it is alive with a controller-instance lease. Recovery uses that lease to decide whether a run's controller is gone. > - In local swap cells, a healthy old server exited during the overlap, because its lease renewal waited for a connection on the busy main pool. Every run it drove was interrupted. > - When a controller crashed, its runs waited for another server boot, or for a timer that restarted on every look. Tasks stood still for 21 to 28 minutes. > - This pull request renews the lease on its own connection, and lets the live server end a crashed controller's runs within a bounded time. > - The benefit is that a swap does not kill a healthy server, and a crashed server's runs end in about 2 minutes, so their tasks continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. During a two-server release swap (local cells, load 14 to 34), the outgoing server exited with "controller instance lease lost while holding work". Five lease renewals in a row timed out after 10 s. The 10 s timeout included the wait for a connection on the shared main pool, which the traffic switch and the replacement's startup pass kept busy. 2. When the old server was killed in the middle of the overlap, its runs stayed in progress for 21 to 28 minutes. The live server did not re-check a crashed controller's runs: only a new boot did. The unverified-controller wait started when a server first looked at the run, not when the controller left, and the successor's own wait record counted as the dead owner's sign of life. **Expected behavior** A healthy server keeps its lease during a swap. When a controller is gone (its lease lapsed and it is fenced or stopped), the live server ends that controller's runs within a bounded time, without another server boot, so restart recovery continues the tasks. **Steps to reproduce** 1. Run two servers against one database with a release swap and a 200 s overlap. 2. Kill the old server 40 s into the overlap (SIGKILL). 3. See that its runs stay in progress for more than 20 minutes, until another server starts. ## What Changed - Lease renewal on its own connection (`packages/db/src/client.ts`, `packages/db/src/index.ts`, `server/src/services/controller-instance-lease.ts`): - New `controllerInstanceLeaseDb(db)`: one private 1-connection pool (`application_name` `paperclip-controller-instance-lease`), made the same way as the existing `controllerLeaseDb` and closed by `closeLivenessDb`. - The renewal is one statement on that connection, as before. - The connection sends `statement_timeout` = 8 s (below the 10 s client timeout) as a startup parameter (new `DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The server cancels a slow renewal, so it never holds the single connection that the next renewal needs, and there is no extra round trip per renewal. (An earlier revision set it per transaction; that added three round trips to every renewal and broke the idle-server receipt test.) - If a connection pooler refuses the startup parameter ("unsupported startup parameter"), the lease logs a warning and renews on a replacement connection without it. - A crashed controller's runs (`server/src/services/recovery/ownership-registry.ts`, `server/src/index.ts`, `server/src/services/native-runtime/native-restart-recovery.ts`): - The ownership reconciler on the live server hands runs whose controller is proven gone to restart recovery. It does not wait for another server boot. - The unverified-controller wait is bounded from the moment the controller's lease lapsed, not from the first look. - When the controller boot is `stopped` or `fenced`, the run ends at once. - A successor's wait record no longer refreshes the dead owner's sign of life. Safety of ending a run when its controller boot is fenced or stopped: - Only `stopped` (written by the boot's own graceful stop) and `fenced` (a newer epoch fenced it after its lease expired plus 30 s grace) count. `live` and `grace` never count, so a controller that is alive and renewing is never treated as gone. `expired` alone does not count either. - A boot that cannot renew stops itself at its self-fence window (about its lease expiry), and a boot that finds itself fenced exits at once. So when another server can fence it, the old process has already stopped driving runs. The lease change above removes the main cause of healthy boots failing to renew. - If the old process is still alive (partition or stall) and its Runner continues: the old run is fenced. Its agent token is refused for writes (`agent_run_superseded`), its remote workspace epoch is abandoned or the continuation gets a new workspace, and the native coordinator records `replacementDenied`. There are never two writers in one workspace. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/controller-instance-lease.test.ts` "own connection": red (10004 ms timeout while the main pool is held) to green. Full file green (26). - `server/src/__tests__/controller-instance-lease.test.ts` "has the server cancel a renewal blocked on a row lock, and renews on the same connection next time": red without the connection-level timeout (the renewal runs to the 10 s client timeout and stays active), green with it (cancelled at about 8 s, no active lease statement left, the next renewal succeeds in under 2 s on the same backend). - `packages/db/src/client-options.test.ts`: the statement timeout is sent as a startup parameter. - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`: "lapsed longer ago" / "end of the bound" (wait counted from the lease lapse), the live-server re-check of a crashed controller, the fenced/stopped boot ending the run at once, and the successor's wait record. Full file green. Changed existing expectation (it asserted the old wait for a stopped or fenced controller): - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` "keeps foreign local Runner and provider ownership after its controller is stopped/fenced": the disposition is now `blocked` (`controller_process_space_unverified`) instead of `awaiting_evidence`, the run is `failed` with `native_restart_recovery_blocked`, and the finalization is `terminal_failure` with `executorStopEvidence: "unverified"`. The safety checks stay: no local PID probe, no claim, no stamp, no local stop proof, the run keeps its process identity and session checkpoint, and no result is written. Local commands, on this branch based on current master: - `cd server && npx vitest run src/__tests__/controller-instance-lease.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140 tests passed. - `npx vitest run src/__tests__/controller-instance-lease.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 225 passed, 14 skipped (real-process cases that need CI). - `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts -t "releases an ordinary held task Retry"`: passed (idle receipt: one lease statement after the work ends). Local swap cells (two servers, embedded Postgres, real agents, old server SIGKILLed 40 s into a 200 s overlap): - Before: runs of the killed server stayed in progress 21 to 28 minutes. - N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the successor's wait write still moved the timer; fixed by the last commit). - R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4 hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run end 3 min 39 s. - N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min 01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the lease TTL (90 s) plus the fence grace (30 s). - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds this fix and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lease and crashed-controller fixes were visibly working in it. ## Risks - A pooled database URL whose pooler refuses `statement_timeout` as a startup parameter: the first renewal fails over to a connection without it (logged), so the lease keeps working without the server-side cancel. - One extra database connection per server, held while the server runs. It is idle except for one renewal every few seconds. - A run whose controller boot is fenced or stopped now ends at once instead of after the bound. See the safety notes above: only a boot that is not alive can be in these states. - No migration. No schema change. Nothing changes before `turn.start` on the normal path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
251c6d5c69 |
fix(recovery): settle a remote workspace save whose process died mid-save (#20326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.
**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.
**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.
**Paperclip version or commit**
`master` at
|
||
|
|
9b71d03215 |
fix: recover interrupted native preparations and unblock upgrades (#20177)
## Thinking Path > - Paperclip manages AI-agent companies and their work. > - A native run prepares its workspace before a runner PID or checkpoint exists. > - A server crash can interrupt that stage and leave its controller on another host. > - Recovery needs a durable owner proof, a delivered recovery wake, and the saved execution contract. > - The old paths missed controller lease registration and boot proof, skipped checkpoint-free preparations, and counted retained work as live upgrade blockers. > - This PR connects preparation to the existing controller lease and recovery paths, and separates retained work from the upgrade census. > - The result is bounded recovery without a second executor or loss of saved work. ## Linked Issues or Issue Description **What happened?** A native preparation interrupted by a server crash could remain running indefinitely with no runner PID or checkpoint. Native controller stamps did not wake the shared instance lease. That lease was disabled on volume-backed Railway services, and native takeover omitted the boot binding needed for its cross-host proof. Scoped recovery selected checkpoint-backed runs but skipped these preparations. The upgrade probe then counted the retained row as current execution and could block every later upgrade. **Expected behavior** Track the preparation's controller before a runner starts. Recover it through the normal classifier after proving its owner stopped or fencing that boot. Keep live or unprovable owners held. Deliver recovery wakes to preparations without checkpoints. Preserve unresolved historical work and report it separately from current execution during upgrade observations. **Steps to reproduce** 1. Save an observed native preparation with no runner process, result, provider-start evidence or checkpoint. 2. Replace its controller with a process in another PID space. Let its owner lease expire and send a scoped recovery wake. 3. Before this fix, recovery skips the checkpoint-free preparation or cannot prove its foreign controller stopped. Upgrade readiness continues to count that old row as a live blocker. Related work: #18037, #19651, #19868, and #20168. The foreign-PID guard from #19868 remains. The queued first-turn input fix from #20168 is included through the current base. ## What Changed - Register the existing instance lease when native execution and restart claims are written. Keep passive identity reads free of ownership effects. Keep that lease enabled on volume-backed Railway services unless explicitly disabled. - Preserve the existing PID/start-time proof within the same readable PID space. Bind foreign or unknown-space takeover to its saved boot and last write. A live lease or grace period keeps the run held with a scheduled check. Repeated evidence waits preserve their timestamp. - Include explicitly fenced, checkpoint-free preparations in the existing finite recovery candidate inventory. Normal launch, checkpoint, provider evidence and admission checks still decide whether work may resume. - Expose the existing board stop-confirmation operation for historical foreign-host owners. Bind confirmation to the exact run, boot, PID, process start and host. Reject changed ownership. - Report narrowly defined old, unowned preparations as `retainedBlockedRuns` outside the live upgrade census. Local executions, pending admission, drain state, truncation and positive checkpoint proof retain their barriers. - Update the execution contract and add PostgreSQL, real-route, race and preservation regressions. ## Verification - Genuine red/green regressions cover the old upgrade census, missing native lease registration, the volume-backed default, foreign-host confirmation, and scoped recovery skipping checkpoint-free preparation. Test bodies ran against real PostgreSQL before their corresponding production changes. - Combined verification before the final idle correction (head `055be6c84d9`): **459 passed, 22 skipped, 11 files passed**. Covers controller lease/identity, native restart classification and integration, host confirmation, heartbeat recovery, instance route authorization, upgrade readiness, and queued first-turn input. The focused command used `pnpm exec vitest run` with those 11 suites and `--maxWorkers=1 --no-file-parallelism`. The 22 skipped cases require the Rust debug fake-Codex fixture, which was not built locally; the real-process CI coverage must pass. - `pnpm --filter @paperclipai/server typecheck`: passed on head `055be6c84d9b058064daad0af71bb5d893ab9529`. - Actual route proof uses the real heartbeat service and PostgreSQL. Unauthorized access returns 403. The retained run and owner records remain unchanged. Recovery integration admits one bootstrap claim across two replacement controllers, keeps a live owner held, and preserves local execution. - `pnpm verify`: exit 2, requests broad CI because the execution contract changed. Full local suite not run. Current-head CI must supply broad coverage. - Local source checkout on macOS ARM64, Node v25.6.1, pnpm 9.15.4. Tests use isolated PostgreSQL. No live provider restart is claimed by those fixtures. - CI found an idle-query regression in the first version of the lease wiring. Two focused PostgreSQL regressions reproduced the misplaced identity-read notification and the missing claim notification. The fix moves registration to actual ownership writes. `env -u npm_execpath pnpm exec vitest run server/src/__tests__/maintenance-server-routes.test.ts --maxWorkers=1 --no-file-parallelism`: **17 passed**, including 90 real seconds and two qualified hourly windows with no extra idle queries. An initial local setup attempt inherited npm metadata and did not run tests; clearing that metadata selected the correct pnpm helper. Three focused controller/native suites also passed **168 tests, 22 fixture-gated skips**. Server typecheck passed after the correction on head `4a527326dcd5b3a9e286f0e4c3e761d3056af4d9`. - The older CI server shard also exposed a same-space graceful-restart regression. The new boot fallback had overridden local process-exit proof. Seven suspension cases and a focused unit regression reproduced it; the fix limits boot fallback to foreign or unknown PID spaces. Fresh verification on `acf1065d98150210c1e227c7f3b05a1eb489730b`: **251 passed, 22 fixture-gated skips, 4 files passed**, including all **40** graceful-restart suspension tests. Server typecheck passed. The local regression failed before this correction, and the same-space and foreign-space paths pass together after it. - Current-head CI completed successfully on `acf1065d98150210c1e227c7f3b05a1eb489730b` (run 42568109784). Greptile reviewed that exact head at 5/5 with no unresolved review threads. - Secret and private-data scans passed. No workflow, lockfile or migration changes. ## Risks - The instance lease now applies to volume-backed Railway services. Its existing 30-second renewal, quiescence, grace and self-fence behavior applies while work is held. Idle servers retain the existing no-query behavior. An explicit `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` remains available. - A controller lease proves only the server owner's state. It does not prove an agent process stopped or authorize replay, resource release or volume removal. The existing classifier and provider/runner checks remain responsible for those decisions. - Previous-release records keep their shape and saved state. A missing boot lease stays unprovable and requires the scoped stop-confirmation path. A changed confirmation target is rejected. - The upgrade observation does not recover a held task by itself. Source tests do not qualify a hosted replacement. Deployment and the affected task's useful result must be verified separately. - This repairs recovery after a crash; it does not claim to identify or fix every source of heap exhaustion. ## Model Used OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3 Code, with repository, shell, and GitHub tools. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
e8a3f1f6f2 |
feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, the control plane settles in-progress records (leases, claims, intents) only once it proves their owner process is gone > - The proof compares host and PID space, so a stamp from a replaced container or a rebooted machine stays unprovable forever, and its records never settle > - Self-hosted users on Docker, Kubernetes and Fly hit this on every redeploy; a Railway deployment with an exclusive volume is covered by `deployment_replaced` (#20026) > - This pull request adds a fenced controller epoch: each boot holds a database lease, and an expired boot is fenced so its records can be proven gone > - The benefit is that work stranded by a replaced container is recovered, without keeping an idle database awake ## Linked Issues or Issue Description Follow-up to #20026 and the Phase 2 durable-ownership data. Related: #19965 (owner stamps), #19883 (process space). No duplicate PR was found. **What happened?** A record stamped by a server process in another container reports `different_process_space`. A dead PID elsewhere is never proof, so the record stays `unprovable` and is never settled. Every redeploy of a self-hosted container leaves such records behind. **Expected behavior** A record owned by a server boot that was replaced (stopped cleanly, or lost its lease and was fenced) is proven gone and settled exactly once. A live boot's records stay with it. An idle server issues no recurring queries. **Steps to reproduce** 1. Run Paperclip in a container and start work, so an environment lease is stamped by that boot. 2. Replace the container. The new container has a new PID namespace. 3. The ownership reconciler reports `different_process_space` (unprovable) for the old lease, forever. **Paperclip version or commit** `master` before this PR. ## What Changed - New `controller_instances` table (migration 0449): boot id, epoch (bigserial), process space, pid, a lease on the database clock, `fenced_at` / `fenced_by_epoch`, `stopped_at`. - `controller-instance-lease.ts`, registered with the shared delivery-work coordinator (`controller-instance-lease` queue, runs during drain): - **Holds work → renews; idle → silent.** One upsert creates the row on first use and extends the lease only if the boot holds work: a stamp within the last 90 s, or a stamp whose ownership write may still be pending (held until the server's tracked work settles, capped at 10 minutes; a renewal that sees some other held record does not clear it, so a slow ownership write cannot be outlived), or an in-progress record held by its boot id. Those records are active environment leases, in-flight cleanup claims, sandbox creation intents, workspace finalize attempts, agent lifecycle steps, native finalizer and cleanup leases (`lease_holder` bound to the token), open native coordinator claims, and running legacy runs. Renewal runs every 30 s while the boot holds work. When it finds none, the lease lapses and the worker sleeps until the next stamp (`onControllerStamp`). - **No idle trailing statements.** When the server's last tracked work settles, the worker re-checks holdings at once, at most every 5 s (the rate limit also absorbs the worker's own run). So the last check happens with the work, not 30 s later. - **Quiesce, don't crash.** Each renewal has a 10 s bound. A failed or timed-out renewal applies an internal task drain, so no new work is dispatched. A recovered renewal resumes. The boot exits only after 60 s of failure (counted from the first failed attempt's start) while it still holds work, or at once when it finds itself fenced. - **Fence on demand, by a newer epoch.** For a stamp from another host or PID space, the prover reads that boot's row: - `stopped` or `fenced`: gone. - live lease: alive. - otherwise, the boot's last sign of life is the later of its lease expiry and the record's last write plus the TTL. Within 30 s of that, the owner stays `alive` (`controller_lease_grace`), so the settle watch keeps rechecking. After that, the prover fences the boot, but only with a lease row of its own (created on demand) and only an older epoch. - A graceful shutdown records `stopped_at`. - A reconciliation pass that settles nothing (`record: false` or `settle: false`) never fences and never creates the boot's row. - `owner-proof.ts`: these proofs apply only to server-held records. A stamp with a recorded process group stays unprovable, because agent processes can outlive their server. The reconciler passes each record's `updatedAt` to the proof. - `task-admission.ts`: - the quiesce drain never replaces an operator's drain, and has no idle owner, so HTTP stays open; - a `subscribeIdleWorkSettled` hook fires when the last tracked work finishes. - Off on a Railway deployment with an exclusive volume (it has its own replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on|off` overrides this. A stateless Railway service keeps the lease. A server without its own lease row never fences. - `maintenance-server-routes.test.ts`: in the post-run quiet window, the lease may check once more that the run's records were released. The test now excludes those statements, then asserts that the lease issues nothing in a further window. - Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the two known gaps. ## Verification - `controller-instance-lease.test.ts` (15 cases): - a replaced container with all Railway variables empty is fenced and settled exactly once over two passes; - a live lease is alive; - an owner in grace, and work written after an idle lapse, stay alive and watched; - an older epoch never fences a newer one; - no fencing without its own lease row (off or unregistered), and an idle prover creates its row on demand; - a stopped boot is gone; - process-group stamps stay unprovable; - idle issues zero statements, and one upsert renews only while records are held; - **a 45 s outage:** quiesced, no exit, resumes; - **a renewal that never answers:** bounded, and the boot exits on time; - **a 70 s outage:** exits at 60–65 s, and the successor recovers the work exactly once; - fenced: stops; - the settle re-check; - graceful stop; off on a volume-exclusive Railway deployment, on for a stateless one, and the on/off override; - a running legacy run keeps the lease live. - Red, with the proof's instance check removed: the replaced-container, live, stopped and successor cases fail. - Wider: task-drain, idle-sleep, idle-local-work, owner-proof and ownership suites, plus the maintenance quiet-window tests. Server low-memory typecheck is clean. ## Risks - **Clock skew.** The proof compares a record's `updatedAt` (often the writer's clock) with the lease (database clock). The 30 s grace absorbs ordinary skew. - **A new exit path exists.** A server whose database is unreachable for more than 60 s while it holds work now exits, after first quiescing. This is the intended safety backstop. - **Bounded trailing activity.** A lease that holds work issues one statement every 30 s, plus at most one check when its work settles. An idle server issues none. - **Boots from before this release** have no row, and their records stay unprovable. The board "mark owner gone" action for those follows in a separate PR. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |