PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-14 08:53:19 +01:00

Commit Graph

Author	SHA1	Message	Date
Dotta	a3e7e63330	Guard Native recovery with process-space proof and explicit containment (#19813 ) ## Thinking Path > - Paperclip manages agents and their work. > - Recovery must preserve execution ownership and accepted results. > - A saved PID proves nothing outside its recorded process namespace. > - Uncertain unfinished work must stay held until its original owners stop. > - Operators need a source-bound containment action and a separate ordinary Retry. > - Accepted results must use the existing finalizer without another provider turn. > - This PR keeps those paths safe, recoverable, and quiet between real obligations. ## Linked Issues or Issue Description Refs: #19713 Refs: #19591 Recovery could use a local PID to release or replace a controller from another process namespace. Connected-chat Retry also rejected successful containment on a task in `todo`. The new containment action used a broader permission than the existing owner-stop action. A later review found that result-less guards also rejected the existing accepted-result confirmation action. ## What Changed - Record and verify the controller's kernel/PID namespace. Keep unknown and foreign unfinished ownership held. Preserve independent durable stop evidence and the existing remote lease fences. - Use the shared accepted-result ownership reconciler. Bind settlement to the observed lease token, generation, boot, PID, namespace and process birth. Retain finalizer, cleanup, current-controller, live-owner and remote exclusions. - Share the remote-owner reader across recovery consumers. Scope runner leases to the same company, task and run. Do not make local PID probes for remote accepted-result ownership. - Gate original-host containment on `runtime:confirm_owner_stop`. Bind it to the current task, source, assignee, original expiry, lease and Stop state. Audit separate controller/provider containment and reviewed action outcomes. - Keep containment separate from ordinary Retry. Use one canonical committed receipt and the normal admission, budget, workspace and lineage checks. Preserve the recorded external-action outcomes. - Keep the controller-only confirmation actionable for both result-less work and accepted results. Result-less work gets its next source-bound attention in the same transaction. Accepted results keep their original result and use normal shared settlement/finalization. - Keep identity inspection quiet. Preserve the original foreign-owner expiry and use the existing scoped finalizer in accepted-result recovery controls. - Update the recovery card, validators, execution projection, docs and access maps. Reuse existing canonical columns; add no migration or ownership backfill. ## Verification Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`. The current fix has a causal red/green check. On the test-only revision, both foreign-local stopped/fenced controller cases failed because recovery probed saved runner, process-group and provider PIDs in this host's namespace. This measured the wrong probes; it did not run duplicate providers. After the fix, the same two cases passed every no-probe, no-claim, no-controller-stamp, no-stop-receipt and unchanged-authority assertion. Three existing remote preparation, grace and fenced-controller progress cases also passed. Total: 5 passed, 0 failed;170 filtered rows excluded. All three normal affected App TypeScript programs passed on this head (660,681 and428 roots). The fix separates a controller wait from execution stop proof. Independent live/grace evidence still schedules recovery. A stopped/fenced foreign controller can pass this namespace guard only for remote execution, which keeps its later checkpoint and never-launched checks. Foreign local execution stays held before runner or provider PID inspection. Earlier source-scoped results remain evidence, not fresh executions on the current head: - On `c86dd741`, the complete failed-owner file passed 25/25, with no skips or filters. Empty scans wrote no controller stamps. Foreign ownership kept its original expiry. Normal returned-claim settlement and attention at the exact deadline passed. - On `c86dd741`, 15 selected accepted-result/restart controls passed. The81 filtered rows are excluded. Both completion cases used the existing scoped finalizer and kept their accepted result without another provider turn. - Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334` plus `c86dd741`) passed the complete 17-case maintenance-server file. This includes the original90-real-second quiet observation and two qualified hourly windows. The quiet case took118,918ms. The four jobs that failed on the earlier publication all passed on that synthetic merge. - On `6243b32c`, the complete ownership-report file passed 29/29. Native NULL namespace stayed unprovable and generic compatibility remained. Three selected recovery controls passed;170 filtered rows are excluded. - On `f81f2c3c`,356 business cases passed: whole ownership-route221, recoveryUI70, owner-proof38 and controller-confirmation8; selected accepted-result authority18; and one real connected-chat containment, separate authorized Retry, useful Native result and downstream reply case. The2,024 filtered rows are excluded. A separate earlier accepted-result causal case kept the original result and completed after restart without replay. Normal Shared exports, App programs, UI typecheck, token gates and map checks passed on their recorded revisions. Unchanged DB, adapter, Eval and Runner inputs/outputs have explicit carry proofs. The clean normal merge with fetched master `8fe524f7` preserves all31 exclusive owned paths and14,827 unrelated master entries. The nine overlapping paths were reviewed, including the upstream unidentified-run stop action. Fresh CI must validate that actual merge composition. PG cases used existing external PG17, fresh owned databases, pools at most4, one worker and no SQL observer during cases. Registered clients and recorded process groups closed. Independent final SQL audits found zero campaign sessions before normal database cleanup. Original failing fixtures remain retained. Original failures are preserved. One new test first failed compilation due to a missing getter import; an import-only correction passed its full owning programs before the causal red/green check. A local quiet attempt stopped in the normal prerequisite hook before any business case and its collector rejected two normal build markers. The later17-case CI pass establishes quiet recovery; it does not turn that failed local collector into a pass. An earlier useful case passed before its private cleanup collector failed on an owned read-only directory; normal cleanup completed after all owners closed. Earlier runner-shutdown cancellation and a separate private whole-suite failure remain unexplained. Current-head checks are complete:57 success,1 neutral and3 skipped, including the successful aggregate CI gate. The legacy status passed. Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this head. All seven review threads are resolved. Full recursive typecheck, the full test suite and release build were not repeated locally. No result here proves deployed suspension, a live provider or termination on another host. ## Risks - Unknown physical ownership can require an owner/admin action. A false containment statement can be unsafe. - New work, reassignment, Stop, renewed authority, accepted results or pending cleanup can invalidate an action. The operator must refresh. - Controller confirmation does not prove provider/tool containment. Completed external actions must not be replayed. - Generic legacy and child-process compatibility remains separate from the strict Native controller policy. - Existing saved state and canonical schema are preserved. Permanent safety refusals remain held with their normal named recovery path. ## Model Used OpenAI `gpt-6.1-sol` agents implemented and tested the changes with reasoning, repository tools and code execution. A GPT-6-based Codex managing agent reviewed and integrated them. The host does not expose the manager's exact build or the context-window sizes. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge	2026-11-09 12:06:11 -06:00
Devin FoleyandPaperclip	9b71d03215	fix: recover interrupted native preparations and unblock upgrades (#20177 ) ## Thinking Path > - Paperclip manages AI-agent companies and their work. > - A native run prepares its workspace before a runner PID or checkpoint exists. > - A server crash can interrupt that stage and leave its controller on another host. > - Recovery needs a durable owner proof, a delivered recovery wake, and the saved execution contract. > - The old paths missed controller lease registration and boot proof, skipped checkpoint-free preparations, and counted retained work as live upgrade blockers. > - This PR connects preparation to the existing controller lease and recovery paths, and separates retained work from the upgrade census. > - The result is bounded recovery without a second executor or loss of saved work. ## Linked Issues or Issue Description What happened? A native preparation interrupted by a server crash could remain running indefinitely with no runner PID or checkpoint. Native controller stamps did not wake the shared instance lease. That lease was disabled on volume-backed Railway services, and native takeover omitted the boot binding needed for its cross-host proof. Scoped recovery selected checkpoint-backed runs but skipped these preparations. The upgrade probe then counted the retained row as current execution and could block every later upgrade. Expected behavior Track the preparation's controller before a runner starts. Recover it through the normal classifier after proving its owner stopped or fencing that boot. Keep live or unprovable owners held. Deliver recovery wakes to preparations without checkpoints. Preserve unresolved historical work and report it separately from current execution during upgrade observations. Steps to reproduce 1. Save an observed native preparation with no runner process, result, provider-start evidence or checkpoint. 2. Replace its controller with a process in another PID space. Let its owner lease expire and send a scoped recovery wake. 3. Before this fix, recovery skips the checkpoint-free preparation or cannot prove its foreign controller stopped. Upgrade readiness continues to count that old row as a live blocker. Related work: #18037, #19651, #19868, and #20168. The foreign-PID guard from #19868 remains. The queued first-turn input fix from #20168 is included through the current base. ## What Changed - Register the existing instance lease when native execution and restart claims are written. Keep passive identity reads free of ownership effects. Keep that lease enabled on volume-backed Railway services unless explicitly disabled. - Preserve the existing PID/start-time proof within the same readable PID space. Bind foreign or unknown-space takeover to its saved boot and last write. A live lease or grace period keeps the run held with a scheduled check. Repeated evidence waits preserve their timestamp. - Include explicitly fenced, checkpoint-free preparations in the existing finite recovery candidate inventory. Normal launch, checkpoint, provider evidence and admission checks still decide whether work may resume. - Expose the existing board stop-confirmation operation for historical foreign-host owners. Bind confirmation to the exact run, boot, PID, process start and host. Reject changed ownership. - Report narrowly defined old, unowned preparations as `retainedBlockedRuns` outside the live upgrade census. Local executions, pending admission, drain state, truncation and positive checkpoint proof retain their barriers. - Update the execution contract and add PostgreSQL, real-route, race and preservation regressions. ## Verification - Genuine red/green regressions cover the old upgrade census, missing native lease registration, the volume-backed default, foreign-host confirmation, and scoped recovery skipping checkpoint-free preparation. Test bodies ran against real PostgreSQL before their corresponding production changes. - Combined verification before the final idle correction (head `055be6c84d9`): 459 passed, 22 skipped, 11 files passed. Covers controller lease/identity, native restart classification and integration, host confirmation, heartbeat recovery, instance route authorization, upgrade readiness, and queued first-turn input. The focused command used `pnpm exec vitest run` with those 11 suites and `--maxWorkers=1 --no-file-parallelism`. The 22 skipped cases require the Rust debug fake-Codex fixture, which was not built locally; the real-process CI coverage must pass. - `pnpm --filter @paperclipai/server typecheck`: passed on head `055be6c84d9b058064daad0af71bb5d893ab9529`. - Actual route proof uses the real heartbeat service and PostgreSQL. Unauthorized access returns 403. The retained run and owner records remain unchanged. Recovery integration admits one bootstrap claim across two replacement controllers, keeps a live owner held, and preserves local execution. - `pnpm verify`: exit 2, requests broad CI because the execution contract changed. Full local suite not run. Current-head CI must supply broad coverage. - Local source checkout on macOS ARM64, Node v25.6.1, pnpm 9.15.4. Tests use isolated PostgreSQL. No live provider restart is claimed by those fixtures. - CI found an idle-query regression in the first version of the lease wiring. Two focused PostgreSQL regressions reproduced the misplaced identity-read notification and the missing claim notification. The fix moves registration to actual ownership writes. `env -u npm_execpath pnpm exec vitest run server/src/__tests__/maintenance-server-routes.test.ts --maxWorkers=1 --no-file-parallelism`: 17 passed, including 90 real seconds and two qualified hourly windows with no extra idle queries. An initial local setup attempt inherited npm metadata and did not run tests; clearing that metadata selected the correct pnpm helper. Three focused controller/native suites also passed 168 tests, 22 fixture-gated skips. Server typecheck passed after the correction on head `4a527326dcd5b3a9e286f0e4c3e761d3056af4d9`. - The older CI server shard also exposed a same-space graceful-restart regression. The new boot fallback had overridden local process-exit proof. Seven suspension cases and a focused unit regression reproduced it; the fix limits boot fallback to foreign or unknown PID spaces. Fresh verification on `acf1065d98150210c1e227c7f3b05a1eb489730b`: 251 passed, 22 fixture-gated skips, 4 files passed, including all 40 graceful-restart suspension tests. Server typecheck passed. The local regression failed before this correction, and the same-space and foreign-space paths pass together after it. - Current-head CI completed successfully on `acf1065d98150210c1e227c7f3b05a1eb489730b` (run 42568109784). Greptile reviewed that exact head at 5/5 with no unresolved review threads. - Secret and private-data scans passed. No workflow, lockfile or migration changes. ## Risks - The instance lease now applies to volume-backed Railway services. Its existing 30-second renewal, quiescence, grace and self-fence behavior applies while work is held. Idle servers retain the existing no-query behavior. An explicit `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` remains available. - A controller lease proves only the server owner's state. It does not prove an agent process stopped or authorize replay, resource release or volume removal. The existing classifier and provider/runner checks remain responsible for those decisions. - Previous-release records keep their shape and saved state. A missing boot lease stays unprovable and requires the scoped stop-confirmation path. A changed confirmation target is rejected. - The upgrade observation does not recover a held task by itself. Source tests do not qualify a hosted replacement. Deployment and the affected task's useful result must be verified separately. - This repairs recovery after a crash; it does not claim to identify or fix every source of heap exhaustion. ## Model Used OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3 Code, with repository, shell, and GitHub tools. Context window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-08 21:53:13 -08:00
DottaandClaude Opus 5.5	f439150cb3	fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - #19868 and #19871 made restart recovery judge a recorded pid only in the PID space that recorded it > - Rows saved before those host columns existed have no recorded host. They kept the previous local dead-pid proof, so in-flight native runs would not be stranded at upgrade > - On an ephemeral (multi-host) server that fallback is unsafe: the row may have been written on another host > - This pull request limits the fallback to single-host servers, records every use, and gives ephemeral servers a working board action > - The benefit is that single-host upgrades still recover on their own, multi-host servers never accept a cross-host proof, and the next release can remove the fallback with evidence ## Linked Issues or Issue Description Bug description After #19868 and #19871, a `native_run_finalizations` row with `controller_host_hash = null`, or a run with `process_host_hash = null`, still accepted "the pid is dead here" as proof on every server. That includes an ephemeral server, where the row may come from another host. Nothing recorded how often the fallback was used, so it had no safe removal path. Expected behavior - Single-host server: the fallback applies for one release, and each use writes a `legacy_host_unknown_dead_pid_proof` activity-log entry. - Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is unprovable, and the board has an action to confirm the previous server stopped. - A mismatched host is always unprovable. Refs #19868, #19871, #19846 ## What Changed - `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: - A coordinator row with `controllerHostHash: null` returns `controller_host_unrecorded` when `ephemeralServerEnabled()` is true. - Otherwise a pid proof from that row carries `legacyHostFallback: true`. - `operator_attested` is accepted as proof (`operator_attested_controller_stopped`). - Receipt-built owners, which have no host key, are unchanged. - New `recordLegacyHostFallbackProof` writes the activity-log entry. It is not a run-log event: run events are progress evidence for restart recovery's crash-loop credit, and the record must not change any recovery decision. It is called from restart recovery, the dead-controller result-lease release, native retirement, and remote warm-session recovery. - New `native-controller-host-recovery.ts`: - `requestNativeControllerStopConfirmation` (board-owned `native_controller_host_unverified` recovery action, idempotent, never displaces another active action). - `confirmNativeControllerStopped`. It verifies the exact recorded controller and stamps `operator_attested` with compare-and-set. It then resolves the action, wakes live-run recovery, and logs `issue.native_controller_stop_confirmed`. - Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`. It is board-only and requires `runtime:confirm_owner_stop`. Generic `restored`/`false_positive` resolution of this cause is refused. A shared `confirmNativeControllerStoppedSchema` validates the body. - UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server stopped" control for this cause, and the generic recovery card hides it. Adds the `issuesApi.confirmControllerStopped` client call. - `recovery/service.ts` orphan reaper: a null `process_host_hash` is judged only when not ephemeral. A successful use records the entry best-effort, after the terminal write. On an ephemeral server the run is left to its existing Stop control. - `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts` (re-exported by `ephemeral-server-contract.ts`). Importing the contract module from recovery code created an import cycle in the runner child process. - The new route is registered in OpenAPI. - `doc/execution-semantics.md`: documents the host rule and the one-release sunset. ## Verification - `native-restart-recovery.test.ts`: null host on a single-host server is released with the flag. Null host on an ephemeral server is unprovable. A mismatched host is unprovable in both modes. An attested controller is accepted. Receipt owners are unchanged. - New `native-controller-host-recovery.test.ts` (embedded Postgres): - The fallback writes the activity-log entry and no run event. - Ephemeral mode creates exactly one board action. - A wrong run id is refused. - Confirmation stamps the attestation, after which takeover is allowed. The action resolves and the activity is logged. A second confirmation is refused. - `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a null host is terminalized and records one event. On an ephemeral server it is left alone with no event. The existing audit-failure case still clears the lock. - `WorkspaceExportRecovery.control.test.tsx`: the controller confirmation posts to `confirmControllerStopped` and never to the copyback endpoint. - Locally: these suites plus `remote-warm-session-recovery`, `native-controller-recovery.integration` and `native-startup-cleanup-recovery` (136 passed), and after the CI fixes `native-runner-restart-preparation` and `native-runner-restart-recovery` integration, `openapi-routes` and the suites above (256 passed); the UI recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no errors in the touched files; `pnpm check:token-gates` clean. ## Risks - Previous-release story: no schema change. Null-host rows saved by earlier releases recover as before on single-host servers, now with an event, and wait for the board on ephemeral servers. The next release removes the fallback; the event count shows whether any rows remain. - On an ephemeral server, a null-host orphaned run process is not reaped automatically. The operator uses Stop. That is the intended "unprovable" outcome. - A board confirmation is an attestation. It applies only to the exact recorded controller (boot id + pid) and only while the host is still unrecorded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-07 13:25:47 -06:00

Author

SHA1

Message

Date

Dotta

a3e7e63330

Guard Native recovery with process-space proof and explicit containment (#19813 )

## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

2026-11-09 12:06:11 -06:00

Devin FoleyandPaperclip

9b71d03215

fix: recover interrupted native preparations and unblock upgrades (#20177 )

## Thinking Path

> - Paperclip manages AI-agent companies and their work.
> - A native run prepares its workspace before a runner PID or
checkpoint exists.
> - A server crash can interrupt that stage and leave its controller on
another host.
> - Recovery needs a durable owner proof, a delivered recovery wake, and
the saved execution contract.
> - The old paths missed controller lease registration and boot proof,
skipped checkpoint-free preparations, and counted retained work as live
upgrade blockers.
> - This PR connects preparation to the existing controller lease and
recovery paths, and separates retained work from the upgrade census.
> - The result is bounded recovery without a second executor or loss of
saved work.

## Linked Issues or Issue Description

**What happened?**

A native preparation interrupted by a server crash could remain running
indefinitely with no runner PID or checkpoint. Native controller stamps
did not wake the shared instance lease. That lease was disabled on
volume-backed Railway services, and native takeover omitted the boot
binding needed for its cross-host proof. Scoped recovery selected
checkpoint-backed runs but skipped these preparations. The upgrade probe
then counted the retained row as current execution and could block every
later upgrade.

**Expected behavior**

Track the preparation's controller before a runner starts. Recover it
through the normal classifier after proving its owner stopped or fencing
that boot. Keep live or unprovable owners held. Deliver recovery wakes
to preparations without checkpoints. Preserve unresolved historical work
and report it separately from current execution during upgrade
observations.

**Steps to reproduce**

1. Save an observed native preparation with no runner process, result,
provider-start evidence or checkpoint.
2. Replace its controller with a process in another PID space. Let its
owner lease expire and send a scoped recovery wake.
3. Before this fix, recovery skips the checkpoint-free preparation or
cannot prove its foreign controller stopped. Upgrade readiness continues
to count that old row as a live blocker.

Related work: #18037, #19651, #19868, and #20168. The foreign-PID guard
from #19868 remains. The queued first-turn input fix from #20168 is
included through the current base.

## What Changed

- Register the existing instance lease when native execution and restart
claims are written. Keep passive identity reads free of ownership
effects. Keep that lease enabled on volume-backed Railway services
unless explicitly disabled.
- Preserve the existing PID/start-time proof within the same readable
PID space. Bind foreign or unknown-space takeover to its saved boot and
last write. A live lease or grace period keeps the run held with a
scheduled check. Repeated evidence waits preserve their timestamp.
- Include explicitly fenced, checkpoint-free preparations in the
existing finite recovery candidate inventory. Normal launch, checkpoint,
provider evidence and admission checks still decide whether work may
resume.
- Expose the existing board stop-confirmation operation for historical
foreign-host owners. Bind confirmation to the exact run, boot, PID,
process start and host. Reject changed ownership.
- Report narrowly defined old, unowned preparations as
`retainedBlockedRuns` outside the live upgrade census. Local executions,
pending admission, drain state, truncation and positive checkpoint proof
retain their barriers.
- Update the execution contract and add PostgreSQL, real-route, race and
preservation regressions.

## Verification

- Genuine red/green regressions cover the old upgrade census, missing
native lease registration, the volume-backed default, foreign-host
confirmation, and scoped recovery skipping checkpoint-free preparation.
Test bodies ran against real PostgreSQL before their corresponding
production changes.
- Combined verification before the final idle correction (head
`055be6c84d9`): **459 passed, 22 skipped, 11 files passed**. Covers
controller lease/identity, native restart classification and
integration, host confirmation, heartbeat recovery, instance route
authorization, upgrade readiness, and queued first-turn input. The
focused command used `pnpm exec vitest run` with those 11 suites and
`--maxWorkers=1 --no-file-parallelism`. The 22 skipped cases require the
Rust debug fake-Codex fixture, which was not built locally; the
real-process CI coverage must pass.
- `pnpm --filter @paperclipai/server typecheck`: passed on head
`055be6c84d9b058064daad0af71bb5d893ab9529`.
- Actual route proof uses the real heartbeat service and PostgreSQL.
Unauthorized access returns 403. The retained run and owner records
remain unchanged. Recovery integration admits one bootstrap claim across
two replacement controllers, keeps a live owner held, and preserves
local execution.
- `pnpm verify`: exit 2, requests broad CI because the execution
contract changed. Full local suite not run. Current-head CI must supply
broad coverage.
- Local source checkout on macOS ARM64, Node v25.6.1, pnpm 9.15.4. Tests
use isolated PostgreSQL. No live provider restart is claimed by those
fixtures.
- CI found an idle-query regression in the first version of the lease
wiring. Two focused PostgreSQL regressions reproduced the misplaced
identity-read notification and the missing claim notification. The fix
moves registration to actual ownership writes. `env -u npm_execpath pnpm
exec vitest run server/src/__tests__/maintenance-server-routes.test.ts
--maxWorkers=1 --no-file-parallelism`: **17 passed**, including 90 real
seconds and two qualified hourly windows with no extra idle queries. An
initial local setup attempt inherited npm metadata and did not run
tests; clearing that metadata selected the correct pnpm helper. Three
focused controller/native suites also passed **168 tests, 22
fixture-gated skips**. Server typecheck passed after the correction on
head `4a527326dcd5b3a9e286f0e4c3e761d3056af4d9`.
- The older CI server shard also exposed a same-space graceful-restart
regression. The new boot fallback had overridden local process-exit
proof. Seven suspension cases and a focused unit regression reproduced
it; the fix limits boot fallback to foreign or unknown PID spaces. Fresh
verification on `acf1065d98150210c1e227c7f3b05a1eb489730b`: **251
passed, 22 fixture-gated skips, 4 files passed**, including all **40**
graceful-restart suspension tests. Server typecheck passed. The local
regression failed before this correction, and the same-space and
foreign-space paths pass together after it.
- Current-head CI completed successfully on
`acf1065d98150210c1e227c7f3b05a1eb489730b` (run 42568109784). Greptile
reviewed that exact head at 5/5 with no unresolved review threads.
- Secret and private-data scans passed. No workflow, lockfile or
migration changes.

## Risks

- The instance lease now applies to volume-backed Railway services. Its
existing 30-second renewal, quiescence, grace and self-fence behavior
applies while work is held. Idle servers retain the existing no-query
behavior. An explicit `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` remains
available.
- A controller lease proves only the server owner's state. It does not
prove an agent process stopped or authorize replay, resource release or
volume removal. The existing classifier and provider/runner checks
remain responsible for those decisions.
- Previous-release records keep their shape and saved state. A missing
boot lease stays unprovable and requires the scoped stop-confirmation
path. A changed confirmation target is rejected.
- The upgrade observation does not recover a held task by itself. Source
tests do not qualify a hosted replacement. Deployment and the affected
task's useful result must be verified separately.
- This repairs recovery after a crash; it does not claim to identify or
fix every source of heap exhaustion.

## Model Used

OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3 Code,
with repository, shell, and GitHub tools. Context window size is not
exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-08 21:53:13 -08:00

DottaandClaude Opus 5.5

f439150cb3

fix(recovery): limit the null-host owner fallback to single-host servers and record each use (#19883 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #19868 and #19871 made restart recovery judge a recorded pid only in
the PID space that recorded it
> - Rows saved before those host columns existed have no recorded host.
They kept the previous local dead-pid proof, so in-flight native runs
would not be stranded at upgrade
> - On an ephemeral (multi-host) server that fallback is unsafe: the row
may have been written on another host
> - This pull request limits the fallback to single-host servers,
records every use, and gives ephemeral servers a working board action
> - The benefit is that single-host upgrades still recover on their own,
multi-host servers never accept a cross-host proof, and the next release
can remove the fallback with evidence

## Linked Issues or Issue Description

**Bug description**
After #19868 and #19871, a `native_run_finalizations` row with
`controller_host_hash = null`, or a run with `process_host_hash = null`,
still accepted "the pid is dead here" as proof on every server. That
includes an ephemeral server, where the row may come from another host.
Nothing recorded how often the fallback was used, so it had no safe
removal path.

**Expected behavior**
- Single-host server: the fallback applies for one release, and each use
writes a `legacy_host_unknown_dead_pid_proof` activity-log entry.
- Ephemeral server (`PAPERCLIP_EPHEMERAL_SERVER=true`): the row is
unprovable, and the board has an action to confirm the previous server
stopped.
- A mismatched host is always unprovable.

Refs #19868, #19871, #19846

## What Changed

- `native-restart-recovery.ts` `evaluateNativeControllerTakeover`:
- A coordinator row with `controllerHostHash: null` returns
`controller_host_unrecorded` when `ephemeralServerEnabled()` is true.
- Otherwise a pid proof from that row carries `legacyHostFallback:
true`.
- `operator_attested` is accepted as proof
(`operator_attested_controller_stopped`).
  - Receipt-built owners, which have no host key, are unchanged.
- New `recordLegacyHostFallbackProof` writes the activity-log entry. It
is not a run-log event: run events are progress evidence for restart
recovery's crash-loop credit, and the record must not change any
recovery decision. It is called from restart recovery, the
dead-controller result-lease release, native retirement, and remote
warm-session recovery.
- New `native-controller-host-recovery.ts`:
- `requestNativeControllerStopConfirmation` (board-owned
`native_controller_host_unverified` recovery action, idempotent, never
displaces another active action).
- `confirmNativeControllerStopped`. It verifies the exact recorded
controller and stamps `operator_attested` with compare-and-set. It then
resolves the action, wakes live-run recovery, and logs
`issue.native_controller_stop_confirmed`.
- Route `POST /issues/:id/recovery-actions/confirm-controller-stopped`.
It is board-only and requires `runtime:confirm_owner_stop`. Generic
`restored`/`false_positive` resolution of this cause is refused. A
shared `confirmNativeControllerStoppedSchema` validates the body.
- UI: `WorkspaceExportRecovery` reuses the "Confirm the previous server
stopped" control for this cause, and the generic recovery card hides it.
Adds the `issuesApi.confirmControllerStopped` client call.
- `recovery/service.ts` orphan reaper: a null `process_host_hash` is
judged only when not ephemeral. A successful use records the entry
best-effort, after the terminal write. On an ephemeral server the run is
left to its existing Stop control.
- `ephemeralServerEnabled` moves to a leaf `ephemeral-server-mode.ts`
(re-exported by `ephemeral-server-contract.ts`). Importing the contract
module from recovery code created an import cycle in the runner child
process.
- The new route is registered in OpenAPI.
- `doc/execution-semantics.md`: documents the host rule and the
one-release sunset.

## Verification

- `native-restart-recovery.test.ts`: null host on a single-host server
is released with the flag. Null host on an ephemeral server is
unprovable. A mismatched host is unprovable in both modes. An attested
controller is accepted. Receipt owners are unchanged.
- New `native-controller-host-recovery.test.ts` (embedded Postgres):
  - The fallback writes the activity-log entry and no run event.
  - Ephemeral mode creates exactly one board action.
  - A wrong run id is refused.
- Confirmation stamps the attestation, after which takeover is allowed.
The action resolves and the activity is logged. A second confirmation is
refused.
- `recovery-stale-issue-lock-sweep.test.ts`: on a single-host server a
null host is terminalized and records one event. On an ephemeral server
it is left alone with no event. The existing audit-failure case still
clears the lock.
- `WorkspaceExportRecovery.control.test.tsx`: the controller
confirmation posts to `confirmControllerStopped` and never to the
copyback endpoint.
- Locally: these suites plus `remote-warm-session-recovery`,
`native-controller-recovery.integration` and
`native-startup-cleanup-recovery` (136 passed), and after the CI fixes
`native-runner-restart-preparation` and `native-runner-restart-recovery`
integration, `openapi-routes` and the suites above (256 passed); the UI
recovery suites (52 passed); `node scripts/typecheck-layers.mjs` with no
errors in the touched files; `pnpm check:token-gates` clean.

## Risks

- Previous-release story: no schema change. Null-host rows saved by
earlier releases recover as before on single-host servers, now with an
event, and wait for the board on ephemeral servers. The next release
removes the fallback; the event count shows whether any rows remain.
- On an ephemeral server, a null-host orphaned run process is not reaped
automatically. The operator uses Stop. That is the intended "unprovable"
outcome.
- A board confirmation is an attestation. It applies only to the exact
recorded controller (boot id + pid) and only while the host is still
unrecorded.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-07 13:25:47 -06:00