Commit Graph
7 Commits
Author SHA1 Message Date
Dotta a3e7e63330 Guard Native recovery with process-space proof and explicit containment (#19813)
## Thinking Path

> - Paperclip manages agents and their work.
> - Recovery must preserve execution ownership and accepted results.
> - A saved PID proves nothing outside its recorded process namespace.
> - Uncertain unfinished work must stay held until its original owners
stop.
> - Operators need a source-bound containment action and a separate
ordinary Retry.
> - Accepted results must use the existing finalizer without another
provider turn.
> - This PR keeps those paths safe, recoverable, and quiet between real
obligations.

## Linked Issues or Issue Description

Refs: #19713

Refs: #19591

Recovery could use a local PID to release or replace a controller from
another process namespace. Connected-chat Retry also rejected successful
containment on a task in `todo`. The new containment action used a
broader permission than the existing owner-stop action. A later review
found that result-less guards also rejected the existing accepted-result
confirmation action.

## What Changed

- Record and verify the controller's kernel/PID namespace. Keep unknown
and foreign unfinished ownership held. Preserve independent durable stop
evidence and the existing remote lease fences.
- Use the shared accepted-result ownership reconciler. Bind settlement
to the observed lease token, generation, boot, PID, namespace and
process birth. Retain finalizer, cleanup, current-controller, live-owner
and remote exclusions.
- Share the remote-owner reader across recovery consumers. Scope runner
leases to the same company, task and run. Do not make local PID probes
for remote accepted-result ownership.
- Gate original-host containment on `runtime:confirm_owner_stop`. Bind
it to the current task, source, assignee, original expiry, lease and
Stop state. Audit separate controller/provider containment and reviewed
action outcomes.
- Keep containment separate from ordinary Retry. Use one canonical
committed receipt and the normal admission, budget, workspace and
lineage checks. Preserve the recorded external-action outcomes.
- Keep the controller-only confirmation actionable for both result-less
work and accepted results. Result-less work gets its next source-bound
attention in the same transaction. Accepted results keep their original
result and use normal shared settlement/finalization.
- Keep identity inspection quiet. Preserve the original foreign-owner
expiry and use the existing scoped finalizer in accepted-result recovery
controls.
- Update the recovery card, validators, execution projection, docs and
access maps. Reuse existing canonical columns; add no migration or
ownership backfill.

## Verification

Current publication head: `71ea76b568de09d6313d8992e803131d9d5c54f3`.

**The current fix has a causal red/green check.** On the test-only
revision, both foreign-local stopped/fenced controller cases failed
because recovery probed saved runner, process-group and provider PIDs in
this host's namespace. This measured the wrong probes; it did not run
duplicate providers. After the fix, the same two cases passed every
no-probe, no-claim, no-controller-stamp, no-stop-receipt and
unchanged-authority assertion. Three existing remote preparation, grace
and fenced-controller progress cases also passed. Total: **5 passed, 0
failed;170 filtered rows excluded**. All three normal affected App
TypeScript programs passed on this head (660,681 and428 roots).

The fix separates a controller wait from execution stop proof.
Independent live/grace evidence still schedules recovery. A
stopped/fenced foreign controller can pass this namespace guard only for
remote execution, which keeps its later checkpoint and never-launched
checks. Foreign local execution stays held before runner or provider PID
inspection.

Earlier source-scoped results remain evidence, not fresh executions on
the current head:

- On `c86dd741`, the complete failed-owner file passed **25/25**, with
no skips or filters. Empty scans wrote no controller stamps. Foreign
ownership kept its original expiry. Normal returned-claim settlement and
attention at the exact deadline passed.
- On `c86dd741`, **15 selected accepted-result/restart controls
passed**. The81 filtered rows are excluded. Both completion cases used
the existing scoped finalizer and kept their accepted result without
another provider turn.
- Trusted CI at synthetic merge `50d072a5` (actual master `1b7fe334`
plus `c86dd741`) passed the complete **17-case maintenance-server
file**. This includes the original90-real-second quiet observation and
two qualified hourly windows. The quiet case took118,918ms. The four
jobs that failed on the earlier publication all passed on that synthetic
merge.
- On `6243b32c`, the complete ownership-report file passed **29/29**.
Native NULL namespace stayed unprovable and generic compatibility
remained. Three selected recovery controls passed;170 filtered rows are
excluded.
- On `f81f2c3c`,356 business cases passed: whole ownership-route221,
recoveryUI70, owner-proof38 and controller-confirmation8; selected
accepted-result authority18; and one real connected-chat containment,
separate authorized Retry, useful Native result and downstream reply
case. The2,024 filtered rows are excluded. A separate earlier
accepted-result causal case kept the original result and completed after
restart without replay.

Normal Shared exports, App programs, UI typecheck, token gates and map
checks passed on their recorded revisions. Unchanged DB, adapter, Eval
and Runner inputs/outputs have explicit carry proofs. The clean normal
merge with fetched master `8fe524f7` preserves all31 exclusive owned
paths and14,827 unrelated master entries. The nine overlapping paths
were reviewed, including the upstream unidentified-run stop action.
Fresh CI must validate that actual merge composition.

PG cases used existing external PG17, fresh owned databases, pools at
most4, one worker and no SQL observer during cases. Registered clients
and recorded process groups closed. Independent final SQL audits found
zero campaign sessions before normal database cleanup. Original failing
fixtures remain retained.

Original failures are preserved. One new test first failed compilation
due to a missing getter import; an import-only correction passed its
full owning programs before the causal red/green check. A local quiet
attempt stopped in the normal prerequisite hook before any business case
and its collector rejected two normal build markers. The later17-case CI
pass establishes quiet recovery; it does not turn that failed local
collector into a pass. An earlier useful case passed before its private
cleanup collector failed on an owned read-only directory; normal cleanup
completed after all owners closed. Earlier runner-shutdown cancellation
and a separate private whole-suite failure remain unexplained.

Current-head checks are complete:57 success,1 neutral and3 skipped,
including the successful aggregate CI gate. The legacy status passed.
Greptile is5/5 and Ponytail gives an explicit5/5 Ship verdict for this
head. All seven review threads are resolved. Full recursive typecheck,
the full test suite and release build were not repeated locally. No
result here proves deployed suspension, a live provider or termination
on another host.

## Risks

- Unknown physical ownership can require an owner/admin action. A false
containment statement can be unsafe.
- New work, reassignment, Stop, renewed authority, accepted results or
pending cleanup can invalidate an action. The operator must refresh.
- Controller confirmation does not prove provider/tool containment.
Completed external actions must not be replayed.
- Generic legacy and child-process compatibility remains separate from
the strict Native controller policy.
- Existing saved state and canonical schema are preserved. Permanent
safety refusals remain held with their normal named recovery path.

## Model Used

OpenAI `gpt-6.1-sol` agents implemented and tested the changes with
reasoning, repository tools and code execution. A GPT-6-based Codex
managing agent reviewed and integrated them. The host does not expose
the manager's exact build or the context-window sizes.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 12:06:11 -06:00
DottaandClaude Opus 5.5 e8a3f1f6f2 feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, the control plane settles in-progress records
(leases, claims, intents) only once it proves their owner process is
gone
> - The proof compares host and PID space, so a stamp from a replaced
container or a rebooted machine stays unprovable forever, and its
records never settle
> - Self-hosted users on Docker, Kubernetes and Fly hit this on every
redeploy; a Railway deployment with an exclusive volume is covered by
`deployment_replaced` (#20026)
> - This pull request adds a fenced controller epoch: each boot holds a
database lease, and an expired boot is fenced so its records can be
proven gone
> - The benefit is that work stranded by a replaced container is
recovered, without keeping an idle database awake

## Linked Issues or Issue Description

Follow-up to #20026 and the Phase 2 durable-ownership data. Related:
#19965 (owner stamps), #19883 (process space). No duplicate PR was
found.

**What happened?**

A record stamped by a server process in another container reports
`different_process_space`. A dead PID elsewhere is never proof, so the
record stays `unprovable` and is never settled. Every redeploy of a
self-hosted container leaves such records behind.

**Expected behavior**

A record owned by a server boot that was replaced (stopped cleanly, or
lost its lease and was fenced) is proven gone and settled exactly once.
A live boot's records stay with it. An idle server issues no recurring
queries.

**Steps to reproduce**

1. Run Paperclip in a container and start work, so an environment lease
is stamped by that boot.
2. Replace the container. The new container has a new PID namespace.
3. The ownership reconciler reports `different_process_space`
(unprovable) for the old lease, forever.

**Paperclip version or commit**

`master` before this PR.

## What Changed

- New `controller_instances` table (migration 0449): boot id, epoch
(bigserial), process space, pid, a lease on the database clock,
`fenced_at` / `fenced_by_epoch`, `stopped_at`.
- `controller-instance-lease.ts`, registered with the shared
delivery-work coordinator (`controller-instance-lease` queue, runs
during drain):
- **Holds work → renews; idle → silent.** One upsert creates the row on
first use and extends the lease only if the boot holds work: a stamp
within the last 90 s, or a stamp whose ownership write may still be
pending (held until the server's tracked work settles, capped at 10
minutes; a renewal that sees some other held record does not clear it,
so a slow ownership write cannot be outlived), or an in-progress record
held by its boot id. Those records are active environment leases,
in-flight cleanup claims, sandbox creation intents, workspace finalize
attempts, agent lifecycle steps, native finalizer and cleanup leases
(`lease_holder` bound to the token), open native coordinator claims, and
running legacy runs. Renewal runs every 30 s while the boot holds work.
When it finds none, the lease lapses and the worker sleeps until the
next stamp (`onControllerStamp`).
- **No idle trailing statements.** When the server's last tracked work
settles, the worker re-checks holdings at once, at most every 5 s (the
rate limit also absorbs the worker's own run). So the last check happens
with the work, not 30 s later.
- **Quiesce, don't crash.** Each renewal has a 10 s bound. A failed or
timed-out renewal applies an internal task drain, so no new work is
dispatched. A recovered renewal resumes. The boot exits only after 60 s
of failure (counted from the first failed attempt's start) while it
still holds work, or at once when it finds itself fenced.
- **Fence on demand, by a newer epoch.** For a stamp from another host
or PID space, the prover reads that boot's row:
    - `stopped` or `fenced`: gone.
    - live lease: alive.
- otherwise, the boot's last sign of life is the later of its lease
expiry and the record's last write plus the TTL. Within 30 s of that,
the owner stays `alive` (`controller_lease_grace`), so the settle watch
keeps rechecking. After that, the prover fences the boot, but only with
a lease row of its own (created on demand) and only an older epoch.
  - A graceful shutdown records `stopped_at`.
- A reconciliation pass that settles nothing (`record: false` or
`settle: false`) never fences and never creates the boot's row.
- `owner-proof.ts`: these proofs apply only to server-held records. A
stamp with a recorded process group stays unprovable, because agent
processes can outlive their server. The reconciler passes each record's
`updatedAt` to the proof.
- `task-admission.ts`:
- the quiesce drain never replaces an operator's drain, and has no idle
owner, so HTTP stays open;
- a `subscribeIdleWorkSettled` hook fires when the last tracked work
finishes.
- Off on a Railway deployment with an exclusive volume (it has its own
replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on|off`
overrides this. A stateless Railway service keeps the lease. A server
without its own lease row never fences.
- `maintenance-server-routes.test.ts`: in the post-run quiet window, the
lease may check once more that the run's records were released. The test
now excludes those statements, then asserts that the lease issues
nothing in a further window.
- Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the
two known gaps.

## Verification

- `controller-instance-lease.test.ts` (15 cases):
- a replaced container with all Railway variables empty is fenced and
settled exactly once over two passes;
  - a live lease is alive;
- an owner in grace, and work written after an idle lapse, stay alive
and watched;
  - an older epoch never fences a newer one;
- no fencing without its own lease row (off or unregistered), and an
idle prover creates its row on demand;
  - a stopped boot is gone;
  - process-group stamps stay unprovable;
- idle issues zero statements, and one upsert renews only while records
are held;
  - **a 45 s outage:** quiesced, no exit, resumes;
- **a renewal that never answers:** bounded, and the boot exits on time;
- **a 70 s outage:** exits at 60–65 s, and the successor recovers the
work exactly once;
  - fenced: stops;
  - the settle re-check;
- graceful stop; off on a volume-exclusive Railway deployment, on for a
stateless one, and the on/off override;
  - a running legacy run keeps the lease live.
- Red, with the proof's instance check removed: the replaced-container,
live, stopped and successor cases fail.
- Wider: task-drain, idle-sleep, idle-local-work, owner-proof and
ownership suites, plus the maintenance quiet-window tests. Server
low-memory typecheck is clean.

## Risks

- **Clock skew.** The proof compares a record's `updatedAt` (often the
writer's clock) with the lease (database clock). The 30 s grace absorbs
ordinary skew.
- **A new exit path exists.** A server whose database is unreachable for
more than 60 s while it holds work now exits, after first quiescing.
This is the intended safety backstop.
- **Bounded trailing activity.** A lease that holds work issues one
statement every 30 s, plus at most one check when its work settles. An
idle server issues none.
- **Boots from before this release** have no row, and their records stay
unprovable. The board "mark owner gone" action for those follows in a
separate PR.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:06:32 -06:00
DottaandClaude Opus 5.5 c966334ec3 fix(recovery): settle a run-less lease once its stamped holder is proven gone (#20026)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane recovers in-progress records after a server
restart through a shared ownership path
> - Run-less sandbox leases (login, probe, harness verification) were
released only by a five-minute idle sweep
> - That sweep releases by age: it leaves a dead holder's sandbox for
minutes and takes a live login session's sandbox
> - This pull request settles a run-less lease only when its stamped
holder is proven gone
> - The benefit is no stuck sandbox after a restart, and no live session
loses its sandbox

## Linked Issues or Issue Description

Related: #19965 (named the owner of login/probe leases), #19964 (the
run-lease settle this mirrors). No duplicate PR was found.

**What happened?**

The final Daytona chaos wave `w-20261108T083036-s9401` recorded I2
`environment_login_probe_lease` (`pid_dead`) on 2 records. These were
the codex agent's harness-verification leases. A restart killed their
holder. The leases stayed `active` until the five-minute idle sweep
released them. The same sweep releases any run-less lease after five
idle minutes, also when its holder is alive.

**Expected behavior**

A run-less lease is released as soon as its stamped holder is proven
gone. A lease whose holder is alive is never released, however long it
is idle.

**Steps to reproduce**

1. Start a Daytona wave: `node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/main.ts --env daytona --daytona-image <image>`
(K1/K4 faults).
2. Let a fault kill the server while an agent's harness verification
holds its sandbox lease.
3. See the reconciler report I2 `environment_login_probe_lease` until
the idle sweep releases it.

**Paperclip version or commit**

`master` at `8475826d2d`.

## What Changed

- New `settleEnvironmentLoginProbeLease`, registered on
`environment_login_probe_lease`. It runs only for a stamped holder that
is proven gone, and compares and swaps on that exact stamp.
- It moves the lease to `pending_cleanup` with the recovered-orphan
reason, so the cleanup sweep destroys the sandbox in its next tick.
- It leaves the lease alone when a legacy shutdown fence applies, or
when another live lease holds the same provider resource.
- The idle sweep for run-less leases now covers only legacy rows saved
before holders were stamped (no `leaseController` key). Stamped rows are
never released by age.
- `holderUnchanged` is now exported from the run-lease settle so both
settles share it.
- The controller stamp (`leaseController`, and every other
`ControllerProcess` stamp) now records the platform deployment when the
platform names one (Railway: project, environment, service, deployment,
volume). `controllerStamp` passes it to `proveOwnerGone`. A container
redeploy changes the PID space, and the old stamp would read
`different_process_space` (unprovable) forever. Now the existing
`deployment_replaced` proof shows the old holder gone. This also covers
`environment_run_lease_active` and the other kinds built on
`controllerStamp`.
- `currentDeploymentIdentity` in `owner-proof.ts` now delegates to
`currentControllerDeployment`, so there is one parser.
- Data doc: added the final Daytona wave, the six Daytona-only crash
points (4 pass, 2 not reached, no violations), and the cause of the
roughly 450 s codex TTFT. Runs waited behind the agent's harness
verification, which restarted after each fault; boot itself took about
32 s.
- Restart-chaos README: corrected which Daytona crash points a plain
task reaches.

## Verification

- Red, before the change:
`environment-login-probe-lease-ownership.test.ts` failed 3 of 6. No
settle happened, and the idle sweep released 3 live holders' leases.
- Green: `pnpm exec vitest run` passes for
`environment-login-probe-lease-ownership`,
`environment-run-lease-ownership` and
`heartbeat-orphaned-active-lease-sweep` (45/45), and for
`environment-lease-owner-stamps`,
`heartbeat-observed-workspace-preservation`,
`ownership-reconciliation-report`, `heartbeat/recovery` and
`workspace-recovery-warm-handoff`.
- Redeploy: a stamp from a replaced Railway deployment settles with
`deployment_replaced`. A stamp from the same deployment in another PID
space stays `active`. This was red when `controllerStamp` dropped the
deployment.
- Live-owner counter-cases: the current process, another live PID,
another PID space, and a lease re-stamped after the proof all stay
`active`. A read-only pass never settles.
- Not re-run live: the Daytona wave and crash points ran before this
fix. The next Daytona wave should show no
`environment_login_probe_lease` I2.

## Risks

- A stamped run-less lease whose live holder never releases it, because
of a bug in that process, is no longer released after five idle minutes.
It is released once that process exits. This is the intended trade:
never take a live holder's sandbox.
- On an ephemeral server without a platform deployment identity (not
Railway), a stamp from a replaced container is still unprovable. Such a
run-less lease stays `active` until the provider expires the sandbox,
and the reconciler reports it as unprovable. Before this PR, the
five-minute sweep released it by age. This is the same open gap as
unstamped legacy leases on an ephemeral server: there is no operator
action yet. It is listed as a Phase 2 follow-up.
- Legacy unstamped run-less leases keep the age fallback. These are rows
written before #19965, and no current code writes them.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 06:43:45 -06:00
DottaandClaude Opus 5.5 cc072e7c4e feat(recovery): settle orphaned active environment leases on the shared ownership path (#19964)
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs hold environment leases (local workspaces and remote
sandboxes) while they execute
> - A crash between the terminal run write and the lease release leaves
the lease `active` with no holder
> - A sweep released such a lease when the run was terminal and the
lease was idle for five minutes. The age was the only guard against a
race with the live executor, which releases the lease itself with the
run's resource disposition
> - Phase 2 of the durable ownership plan moves each in-progress record
kind onto one shared owner proof (#19933 added the settle hook)
> - This pull request moves `environment_run_lease_active` onto that
path: the lease names its holder, the shared prover judges the holder,
and the settle never destroys a sandbox
> - The benefit is that a dead holder's lease settles on the next pass,
a live holder is never raced, and the five-minute age gate is gone

## Linked Issues or Issue Description

Refs #19933

**What happened?**
An orphaned active environment lease was released by age: "run terminal
plus five minutes" (the reaper in `heartbeat/recovery.ts` and the
live-recovery `orphanLease` worker). The ownership data showed one such
lease settle 119 s after it was ready. Without the age gate the sweep
could flip a lease to `pending_cleanup` (destroy) while the executor or
a recovery path was about to retain it.

**Expected behavior**
A run lease settles when its holder is proven gone, by identity and not
by age. A live holder is never raced. A settle never destroys a sandbox
whose files may still be needed.

**Steps to reproduce**
1. Start a run that holds an environment lease.
2. Kill the server after the run status becomes terminal and before the
lease release.
3. Restart. The lease stays `active` until the five-minute age passes,
then the sweep destroys the sandbox.

## What Changed

- `environments.ts`: the write that makes a lease active (insert and
same-run reacquire) stamps the holder process in
`metadata.leaseController`. The stamp does no I/O
(`environment-lease-controller.ts`).
- `recovery/ownership-registry.ts`: `environment_run_lease_active` lists
leases with their run status. The owner is the run while it is active,
then the stamped holder. It registers `settle` and `listForRuns`.
- `environment-run-lease-settlement.ts` (new): compare-and-swap settle
on the exact row (same run, status `active`, same metadata, terminal
run). A sandbox lease gets the stop-and-retain intent (stop and keep,
never destroy). Other leases move to `pending_cleanup` as before. It
keeps a lease that a pending native export, a legacy shutdown fence, a
shared provider resource, or an accepted native result still needs.
- `recovery/ownership-invariant-report.ts`: `settleOwnableKind` accepts
`currentProcessIdle`. A caller that passes every run this process
executes in `skipRunIds` may treat a current-process owner of another
run as gone (`current_process_idle`). The periodic pass never does this.
- `heartbeat/recovery.ts`: `sweepOrphanedActiveLeases` no longer
releases run leases by age. It runs native workspace preservation first
(a separate obligation), then calls `settleOwnableKind` with this
process's executions. Login and probe leases (no run) keep the idle
sweep until their own change.
- `heartbeat/live-recovery-work.ts`: a run lease is due now; only
run-less leases keep the five-minute idle deadline.
- `sandbox-stop-and-retain.ts`: an ownership settle records a recovery
marker on the intent and keeps `updatedAt`, so the pending-cleanup sweep
stops the sandbox in the same tick.
- Deleted: the age gate for run leases and the page-order bump for run
leases.

## Verification

- `cd server && npx vitest run
src/__tests__/environment-run-lease-ownership.test.ts
src/__tests__/heartbeat-orphaned-active-lease-sweep.test.ts
src/__tests__/ownership-reconciliation-report.test.ts` (embedded
Postgres).
- Red on unchanged master sources, green on this branch (tests were
written after the code):
  - stamps the lease holder in the write that makes a lease active
- settles a lease whose holder died after the run went terminal, on the
next pass (crash-point state: terminal run, dead holder in this PID
space)
  - stops and keeps a dead holder's sandbox instead of destroying it
- leaves a current-process lease to this process: settled only when it
is not executing the run
  - keeps the run fact alone for one release on a single-host server
- names no owner on an ephemeral server
(`PAPERCLIP_EPHEMERAL_SERVER=true`)
- the three updated sweep tests (sandbox stopped and kept, not
destroyed)
- Live-owner counter-cases (green): run still active; another live PID;
another PID space; current process while executing; read-only pass
(`record: false`) never settles; a lease re-stamped after the proof is
not settled; a legacy lease whose run this boot owns is kept.
- Also run green: `recovery.test.ts`,
`heartbeat-observed-workspace-preservation.test.ts`,
`workspace-recovery-warm-handoff.test.ts`,
`environment-service.test.ts`,
`heartbeat-pending-cleanup-sweep.test.ts`,
`native-environment-settlement-work.test.ts`,
`environment-cleanup-work.test.ts`.
- Full typecheck is left to CI (local memory limit).

## Risks

- Behavior change: an orphaned sandbox lease is now stopped and kept,
not destroyed. Workspace preservation and provider auto-delete decide
when it goes away. This can keep stopped sandboxes longer than before.
- A recovery path on another process can mark a run terminal while the
dead executor's stamp is still on the lease. The settle can then win
before that path releases. Because the settle only stops and keeps the
sandbox, files are not lost; the later release continues from the
stop-only intent.
- Previous-release rows: a single-host server keeps the run fact for one
release (a run this boot owns is excluded). An ephemeral server never
releases an unstamped lease; it stays visible as an I1 gap with no
operator action yet.
- No migration. The stamp lives in existing lease metadata.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edit, embedded Postgres tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 22:43:57 -06:00
DottaandClaude Opus 5.5 f5400e2594 feat(recovery): settle the accepted-result lease on the shared ownership path (#19933)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A control-plane restart can kill the controller after an agent's
result is accepted. The result then stays leased to a dead process.
> - Today a private sweep (`native-dead-controller-result-lease.ts`,
#19713) proves that controller dead with its own liveness rules.
> - The durable-ownership plan (#19831) replaces private proofs with one
prover (`proveOwnerGone`, #19853) and one registry (#19861).
> - This pull request moves the accepted-result executor lease to that
shared path and deletes the private sweep. It is Phase 2's first kind.
> - The benefit is one tested rule for when a dead owner's result lease
is released, and the reconciler now repairs a lease that restart
recovery missed.

## Linked Issues or Issue Description

Refs #19831 (plan), #19853 (prover), #19861 (registry), #19713 (old
sweep), #19883 (legacy host fallback), #19927 (Phase 2 data).

**What happened?**
The accepted-result lease had its own liveness proof, separate from the
shared prover. The reconciler could only report a dead owner, not settle
it. When restart recovery missed a lease, it waited out the 20-minute
lease.

**What did you expect?**
One prover decides, and both restart recovery and the periodic
reconciler pass settle a lease whose controller is proven gone.

**Steps to reproduce**
1. Accept a native run result.
2. Kill the controller (SIGKILL) and start a new server.
3. If the restart sweep does not see the row, the result stays leased
until the lease expires.

## What Changed

- `ownership-registry.ts`: a kind may register `settle` and
`onUnprovable`. `native_accepted_result_lease` registers both. Other
kinds stay report-only.
- New `native-accepted-result-lease-settlement.ts`: a compare-and-swap
settle that expires exactly the proven owner's lease. It keeps the old
guards: the run must still be `running`, remote copyback stays fenced,
and a lease taken over since the proof is left alone.
- `ownership-invariant-report.ts`: the pass calls `settle` on `gone` and
`onUnprovable` on `unprovable`. A read-only pass (`record: false`) never
settles. New `settleOwnableKind()` for restart recovery.
- `heartbeat/recovery.ts`: restart recovery calls
`settleOwnableKind("native_accepted_result_lease")` instead of the old
sweep.
- Deleted `native-dead-controller-result-lease.ts`.
- `owner-proof.ts`: accepts a board attestation (`operator_attested`).
It also keeps the one-release local fallback for rows saved before their
PID space was recorded, on single-host servers only, and flags each use
so it is logged (`legacy_host_unknown_dead_pid_proof`, #19883). An
ephemeral server returns `unprovable` (`process_space_unrecorded`) and
the existing "confirm the previous server stopped" board action is
raised.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 27 pass, 7
of them new.
- `npx vitest run src/services/owner-proof.test.ts
src/__tests__/ownership-reconciliation-report.test.ts`: pass.
- Red/green (tests written after the code): 5 new behaviour tests fail
on master and pass here. They cover the ephemeral board action, the
reconciler-pass settle, the live-owner counter-case in that pass, the
legacy marker, and operator attestation. The stamped-dead-controller and
foreign-PID-space cases pass on both, as the old path behaved the same.
- Server typecheck passes (`npx tsc --noEmit -p server/tsconfig.json`).
- Not run locally: the full server suite (`pnpm verify` asks for broad
coverage because a file was deleted). Required CI covers it.

## Risks

- Previous-release story (rules/upgrade-safe-run-state.mdc): NRF rows
saved before #19868 have no `controller_host_hash`. They keep the same
one-release local proof as before on single-host servers, and each use
is logged. On ephemeral servers they stay unprovable and ask the board,
as before. Rows on another PID space keep their lease (the fence), as
before.
- Fail-open risk: settle is compare-and-swap on the lease token, boot,
PID, host hash and start time of the proven owner. An `unprovable` owner
is never released. Live-owner and foreign-space cases are tested.
- No change adds work before `turn.start`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 20:58:04 -06:00
DottaandClaude Opus 5.5 cbf811cd3b fix(recovery): prove a replaced deployment's copyback owner gone automatically (#19913)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs save their workspace after the result is accepted, under
a copyback owner receipt
> - A deploy can restart the server while that save is in progress
> - The new server runs in a new container, so it cannot see the old
process and fails closed
> - The run then waits on a board action that asks a person to verify
the previous controller
> - Recoveries must continue without a human step, and copy must not ask
anyone to inspect a server
> - This pull request adds a deployment-level rule to the shared
owner-liveness primitive and uses it in workspace finalization
> - The benefit is that a save interrupted by a deploy resumes on its
own, with no second writer

## Linked Issues or Issue Description

**What happened?**
A deploy restarted the server while three runs were saving their
accepted results. Each run kept a copyback owner receipt that names the
old Railway deployment on another container. The restarted server
reported `native_workspace_finalization_owner_unverified`, owned by the
board, with the next action "Verify the previous controller and its
workspace-copyback processes have stopped...". The cloud harness stop
receipt did not arrive, so the runs stayed in "Recovery needed" for
about an hour, until a person confirmed the stop by hand.

**Expected behavior**
The restarted server proves that the previous controller is gone,
resumes the save, and settles the run with no human step. No user-facing
text asks a person to verify a server or controller.

**Steps to reproduce**
1. Start a native run on a Railway service with a volume.
2. Let the result be accepted and the workspace save start.
3. Deploy, so that the server restarts during the save.
4. See that the run waits for a board confirmation.

**Paperclip version or commit**
Observed on e2d490c61. Reproduced on master before this change. The
owner-proof work in #19848, #19851, #19853, #19868, #19871 and #19883
does not cover this shape: `proveOwnerGone` returns `unprovable`
(`different_host`) for an owner on another container.

## What Changed

- `owner-proof.ts`: `LocalOwnerStamp` can carry the platform `worker`
(deployment identity). `ProcessProbe` can carry this process's
deployment and whether the platform gives it an exclusive volume. New
verdict reason `deployment_replaced`.
- New rule in `proveOwnerGone`, after the current-boot check: the owner
is gone when this server runs with a Railway volume
(`RAILWAY_VOLUME_ID`, `RAILWAY_VOLUME_MOUNT_PATH`) and the stamp names
the same project, environment and service with a different deployment
ID. Railway stops the previous deployment before it mounts the volume
into the new one, and a service with a volume has no replicas. So every
process of the old deployment, including orphaned copyback children, has
exited.
- `native-workspace-finalization-ownership.ts`: evaluate that proof
before the cloud harness request. On `deployment_replaced`, claim the
exact owner receipt under the existing advisory lock and row-lock
revalidation, log `issue.workspace_owner_stop_verified` with
`deploymentProof`, mark the recovery `resuming`, and continue the save
with no provider turn. Other cases keep the harness request and the
existing fences.
- New owner receipts also record the mounted volume ID (`volumeId`).
When a receipt has it, the proof requires this server to mount the same
volume. Both `RAILWAY_VOLUME_ID` and `RAILWAY_VOLUME_MOUNT_PATH` must be
set. The finalizer reaches this check only while it holds the run's
copyback advisory lock, so the old owner's lock session has also ended.
- Copy: the hold's next action, the route conflict and the two Retry
wait messages now describe the wait. They no longer ask a person to
verify a server or controller. The hold's text still names the Resume
saved result action for platforms where neither this rule nor the
harness can confirm the stop.
- `doc/native-workspace-finalization-recovery.md`: describe the rule.

Previous-release story: receipts written by earlier releases already
record `worker.deploymentId`, so they carry forward and take this path
after the upgrade. No schema change.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-workspace-owner-restart-recovery.test.ts
src/services/owner-proof.test.ts`
- Before the fix: "resumes the interrupted save automatically, with no
board action" fails (`acquired: false`).
  - After the fix: all pass.
- The restart test seeds the staging receipt shape (old deployment,
other hostname, PID 3, existing `operator_required` hold). It checks
that the save resumes once, the hold becomes `resuming`, the task
presentation no longer waits for a user, the receipt clears, and no
agent wake is created.
- Negative cases stay fenced: same deployment, a service without a
volume, a half-configured volume, another service, another environment,
and a receipt that names a different volume. Unit tests cover the same
rule in `proveOwnerGone`, including that the current boot stays alive.
- Related suites pass: finalizer, queued-wake recovery, route ownership,
workspace recovery gates and dispatch, explicit continuation, recovery
services (1005 tests).
- `pnpm --filter @paperclipai/server typecheck` passes.

## Risks

- The rule depends on Railway volume exclusivity. It applies only when
this server has a volume mounted and the receipt names the same service
and environment. Without a volume, the existing harness and board paths
still apply.
- A receipt from a deployment that ran before the volume was attached
could, in theory, overlap during that one redeploy. Volumes are attached
at stack creation, so this is not a practical case.
- Copy changes are text only. Error codes are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 16:12:23 -06:00
DottaandClaude Opus 5.5 ff53eb84db feat(recovery): add OwnerStamp and proveOwnerGone (gone | alive | unprovable) (#19853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Restart recovery must decide when a dead process's in-progress
record may be released
> - About 31 sites decide "owner gone" today, with different rules for
EPERM, PID reuse and other hosts (Phase 0 inventory, finding 1)
> - The durable ownership plan (section A) asks for one owner stamp and
one prover that returns `gone | alive | unprovable`
> - This pull request adds that stamp and prover. Nothing calls it yet;
the next PR uses it in a report-only reconciler
> - The benefit is one tested definition of "owner gone" that later
phases move each record kind onto

## Linked Issues or Issue Description

Refs #19831 (plan, section A and Phase 1). Refs #19846 (Phase 0
inventory). Refs #19848 (Track 1: 1 s start-time gap and EPERM as
alive). Stacked on #19851.

**Problem**
Each subsystem proves "owner gone" with its own rules. Some accept a
dead PID on another host as proof. None returns "unprovable".

**Expected behavior**
One prover with one set of rules. A different host and an expired fence
alone are never proof. "Unprovable" never releases a record.

**Impact**
This is the primitive that Phase 1's report-only reconciler and Phase
2's migrations use.

## What Changed

- New `server/src/services/owner-proof.ts`:
- `OwnerStamp`: a local stamp (`host`, `processSpace`, `bootId`, `pid`,
`processStartedAt`, `processGroupId`, `generation`, `scope`) or a remote
stamp (`fenceToken`, `generation`, `scope`, `provider`).
- `proveOwnerGone(stamp, …)` returns `{ verdict: "gone" | "alive" |
"unprovable", reason }`.
- Local rules: this boot is alive; the stamp must name this host and PID
space (host, kernel boot, PID namespace); the PID must be dead or reused
(start-time gap of at least 1 s, via Track 1's
`processStartsDefinitelyDiffer`); EPERM is alive; the process group must
be empty or led by a newer process.
- Remote rules: a superseded fence token or generation is gone; a
provider stop proof is gone; a held fence is alive; a matching fence
that only expired is unprovable. Process liveness is never consulted for
a remote owner.
- `currentLocalOwnerStamp()` reads only cached identity, so writing a
stamp adds no OS call to the turn path.
- `controller-process-identity.ts` exports `observedProcessStartedAt`
and `UNKNOWN_PROCESS_SPACE` for the prover.
- New `owner-proof.test.ts`: the proof matrix (EPERM, ESRCH, PID reuse
inside and at 1 s, unreadable start time, a different host, a different
or unknown PID space, process groups, a held, superseded, expired or
provider-proved remote fence).

## Verification

- `npx vitest run src/services/owner-proof.test.ts` (in `server/`): 20
tests pass.
- `npx tsc --noEmit -p server`: no errors.

## Risks

- Low risk. No caller uses the prover yet, so no behavior changes.
- Previous-release story (`rules/upgrade-safe-run-state.mdc`): no
persisted shape changes. Stamps from older rows that lack fields return
`unprovable` (`missing_identity`), which never releases a record.
- Track 1 is still fixing proofs in place. The prover reuses Track 1's
helpers (`processExists`, `processStartsDefinitelyDiffer`), so a later
Track 1 rule change applies here too.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:04:55 -06:00