Commit Graph
4 Commits
Author SHA1 Message Date
Dotta a3e7e63330 Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 12:06:11 -06:00
DottaandClaude Opus 5.5 f5400e2594 feat(recovery): settle the accepted-result lease on the shared ownership path (#19933)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A control-plane restart can kill the controller after an agent's
result is accepted. The result then stays leased to a dead process.
> - Today a private sweep (`native-dead-controller-result-lease.ts`,
#19713) proves that controller dead with its own liveness rules.
> - The durable-ownership plan (#19831) replaces private proofs with one
prover (`proveOwnerGone`, #19853) and one registry (#19861).
> - This pull request moves the accepted-result executor lease to that
shared path and deletes the private sweep. It is Phase 2's first kind.
> - The benefit is one tested rule for when a dead owner's result lease
is released, and the reconciler now repairs a lease that restart
recovery missed.

## Linked Issues or Issue Description

Refs #19831 (plan), #19853 (prover), #19861 (registry), #19713 (old
sweep), #19883 (legacy host fallback), #19927 (Phase 2 data).

**What happened?**
The accepted-result lease had its own liveness proof, separate from the
shared prover. The reconciler could only report a dead owner, not settle
it. When restart recovery missed a lease, it waited out the 20-minute
lease.

**What did you expect?**
One prover decides, and both restart recovery and the periodic
reconciler pass settle a lease whose controller is proven gone.

**Steps to reproduce**
1. Accept a native run result.
2. Kill the controller (SIGKILL) and start a new server.
3. If the restart sweep does not see the row, the result stays leased
until the lease expires.

## What Changed

- `ownership-registry.ts`: a kind may register `settle` and
`onUnprovable`. `native_accepted_result_lease` registers both. Other
kinds stay report-only.
- New `native-accepted-result-lease-settlement.ts`: a compare-and-swap
settle that expires exactly the proven owner's lease. It keeps the old
guards: the run must still be `running`, remote copyback stays fenced,
and a lease taken over since the proof is left alone.
- `ownership-invariant-report.ts`: the pass calls `settle` on `gone` and
`onUnprovable` on `unprovable`. A read-only pass (`record: false`) never
settles. New `settleOwnableKind()` for restart recovery.
- `heartbeat/recovery.ts`: restart recovery calls
`settleOwnableKind("native_accepted_result_lease")` instead of the old
sweep.
- Deleted `native-dead-controller-result-lease.ts`.
- `owner-proof.ts`: accepts a board attestation (`operator_attested`).
It also keeps the one-release local fallback for rows saved before their
PID space was recorded, on single-host servers only, and flags each use
so it is logged (`legacy_host_unknown_dead_pid_proof`, #19883). An
ephemeral server returns `unprovable` (`process_space_unrecorded`) and
the existing "confirm the previous server stopped" board action is
raised.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 27 pass, 7
of them new.
- `npx vitest run src/services/owner-proof.test.ts
src/__tests__/ownership-reconciliation-report.test.ts`: pass.
- Red/green (tests written after the code): 5 new behaviour tests fail
on master and pass here. They cover the ephemeral board action, the
reconciler-pass settle, the live-owner counter-case in that pass, the
legacy marker, and operator attestation. The stamped-dead-controller and
foreign-PID-space cases pass on both, as the old path behaved the same.
- Server typecheck passes (`npx tsc --noEmit -p server/tsconfig.json`).
- Not run locally: the full server suite (`pnpm verify` asks for broad
coverage because a file was deleted). Required CI covers it.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): NRF rows
saved before #19868 have no `controller_host_hash`. They keep the same
one-release local proof as before on single-host servers, and each use
is logged. On ephemeral servers they stay unprovable and ask the board,
as before. Rows on another PID space keep their lease (the fence), as
before.
- Fail-open risk: settle is compare-and-swap on the lease token, boot,
PID, host hash and start time of the proven owner. An `unprovable` owner
is never released. Live-owner and foreign-space cases are tested.
- No change adds work before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:58:04 -06:00
DottaandClaude Opus 5.5 cbf811cd3b fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer

## Linked Issues or Issue Description

**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.

**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.

**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.

**Paperclip version or commit**
Observed on e2d490c61. Reproduced on master before this change. The
owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883
does not cover this shape: `proveOwnerGone` returns `unprovable`
(`different_host`) for an owner on another container.

## What Changed

- `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker`
(deployment identity). `ProcessProbe` can carry this process's
deployment and whether the platform gives it an exclusive volume. New
verdict reason `deployment_replaced`.
- New rule in `proveOwnerGone`, after the current-boot check: the owner
is gone when this server runs with a Railway volume
(`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names
the same project, environment and service with a different deployment
ID. Railway stops the previous deployment before it mounts the volume
into the new one, and a service with a volume has no replicas. So every
process of the old deployment, including orphaned copyback children, has
exited.
- `native-workspace-finalization-ownership.ts`: evaluate that proof
before the cloud harness request. On `deployment_replaced`, claim the
exact owner receipt under the existing advisory lock and row-lock
revalidation, log `issue.workspace_owner_stop_verified` with
`deploymentProof`, mark the recovery `resuming`, and continue the save
with no provider turn. Other cases keep the harness request and the
existing fences.
- New owner receipts also record the mounted volume ID (`volumeId`).
When a receipt has it, the proof requires this server to mount the same
volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be
set. The finalizer reaches this check only while it holds the run's
copyback advisory lock, so the old owner's lock session has also ended.
- Copy: the hold's next action, the route conflict and the two Retry
wait messages now describe the wait. They no longer ask a person to
verify a server or controller. The hold's text still names the Resume
saved result action for platforms where neither this rule nor the
harness can confirm the stop.
- `doc/native-workspace-finalization-recovery.md`: describe the rule.

Previous-release story: receipts written by earlier releases already
record `worker.deploymentId`, so they carry forward and take this path
after the upgrade. No schema change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts
src/services/owner-proof.test.ts`
- Before the fix: "resumes the interrupted save automatically, with no
board action" fails (`acquired: false`).
  - After the fix: all pass.
- The restart test seeds the staging receipt shape (old deployment,
other hostname, PID 3, existing `operator_required` hold). It checks
that the save resumes once, the hold becomes `resuming`, the task
presentation no longer waits for a user, the receipt clears, and no
agent wake is created.
- Negative cases stay fenced: same deployment, a service without a
volume, a half-configured volume, another service, another environment,
and a receipt that names a different volume. Unit tests cover the same
rule in `proveOwnerGone`, including that the current boot stays alive.
- Related suites pass: finalizer, queued-wake recovery, route ownership,
workspace recovery gates and dispatch, explicit continuation, recovery
services (1005 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- The rule depends on Railway volume exclusivity. It applies only when
this server has a volume mounted and the receipt names the same service
and environment. Without a volume, the existing harness and board paths
still apply.
- A receipt from a deployment that ran before the volume was attached
could, in theory, overlap during that one redeploy. Volumes are attached
at stack creation, so this is not a practical case.
- Copy changes are text only. Error codes are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:12:23 -06:00
DottaandClaude Opus 5.5 ff53eb84db feat(recovery): add OwnerStamp and proveOwnerGone (gone | alive | unprovable) (#19853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Restart recovery must decide when a dead process's in-progress
record may be released
> - About 31 sites decide "owner gone" today, with different rules for
EPERM, PID reuse and other hosts (Phase 0 inventory, finding 1)
> - The durable ownership plan (section A) asks for one owner stamp and
one prover that returns `gone | alive | unprovable`
> - This pull request adds that stamp and prover. Nothing calls it yet;
the next PR uses it in a report-only reconciler
> - The benefit is one tested definition of "owner gone" that later
phases move each record kind onto

## Linked Issues or Issue Description

Refs #19831 (plan, section A and Phase 1). Refs #19846 (Phase 0
inventory). Refs #19848 (Track 1: 1 s start-time gap and EPERM as
alive). Stacked on #19851.

**Problem**
Each subsystem proves "owner gone" with its own rules. Some accept a
dead PID on another host as proof. None returns "unprovable".

**Expected behavior**
One prover with one set of rules. A different host and an expired fence
alone are never proof. "Unprovable" never releases a record.

**Impact**
This is the primitive that Phase 1's report-only reconciler and Phase
2's migrations use.

## What Changed

- New `server/src/services/owner-proof.ts`:
- `OwnerStamp`: a local stamp (`host`, `processSpace`, `bootId`, `pid`,
`processStartedAt`, `processGroupId`, `generation`, `scope`) or a remote
stamp (`fenceToken`, `generation`, `scope`, `provider`).
- `proveOwnerGone(stamp, …)` returns `{ verdict: "gone" | "alive" |
"unprovable", reason }`.
- Local rules: this boot is alive; the stamp must name this host and PID
space (host, kernel boot, PID namespace); the PID must be dead or reused
(start-time gap of at least 1 s, via Track 1's
`processStartsDefinitelyDiffer`); EPERM is alive; the process group must
be empty or led by a newer process.
- Remote rules: a superseded fence token or generation is gone; a
provider stop proof is gone; a held fence is alive; a matching fence
that only expired is unprovable. Process liveness is never consulted for
a remote owner.
- `currentLocalOwnerStamp()` reads only cached identity, so writing a
stamp adds no OS call to the turn path.
- `controller-process-identity.ts` exports `observedProcessStartedAt`
and `UNKNOWN_PROCESS_SPACE` for the prover.
- New `owner-proof.test.ts`: the proof matrix (EPERM, ESRCH, PID reuse
inside and at 1 s, unreadable start time, a different host, a different
or unknown PID space, process groups, a held, superseded, expired or
provider-proved remote fence).

## Verification

- `npx vitest run src/services/owner-proof.test.ts` (in `server/`): 20
tests pass.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- Low risk. No caller uses the prover yet, so no behavior changes.
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Stamps from older rows that lack fields return
`unprovable` (`missing_identity`), which never releases a record.
- Track 1 is still fixing proofs in place. The prover reuses Track 1's
helpers (`processExists`, `processStartsDefinitelyDiffer`), so a later
Track 1 rule change applies here too.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:04:55 -06:00