Commit Graph
4 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 500bc694b5 fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap runs two servers for a short overlap. The old server
drains, and the new server takes over the runs.
> - Each server proves that it is alive with a controller-instance
lease. Recovery uses that lease to decide whether a run's controller is
gone.
> - In local swap cells, a healthy old server exited during the overlap,
because its lease renewal waited for a connection on the busy main pool.
Every run it drove was interrupted.
> - When a controller crashed, its runs waited for another server boot,
or for a timer that restarted on every look. Tasks stood still for 21 to
28 minutes.
> - This pull request renews the lease on its own connection, and lets
the live server end a crashed controller's runs within a bounded time.
> - The benefit is that a swap does not kill a healthy server, and a
crashed server's runs end in about 2 minutes, so their tasks continue by
themselves.

## Linked Issues or Issue Description

Refs #21135, #21222

**What happened?**
1. During a two-server release swap (local cells, load 14 to 34), the
outgoing server exited with "controller instance lease lost while
holding work". Five lease renewals in a row timed out after 10 s. The 10
s timeout included the wait for a connection on the shared main pool,
which the traffic switch and the replacement's startup pass kept busy.
2. When the old server was killed in the middle of the overlap, its runs
stayed in progress for 21 to 28 minutes. The live server did not
re-check a crashed controller's runs: only a new boot did. The
unverified-controller wait started when a server first looked at the
run, not when the controller left, and the successor's own wait record
counted as the dead owner's sign of life.

**Expected behavior**
A healthy server keeps its lease during a swap. When a controller is
gone (its lease lapsed and it is fenced or stopped), the live server
ends that controller's runs within a bounded time, without another
server boot, so restart recovery continues the tasks.

**Steps to reproduce**
1. Run two servers against one database with a release swap and a 200 s
overlap.
2. Kill the old server 40 s into the overlap (SIGKILL).
3. See that its runs stay in progress for more than 20 minutes, until
another server starts.

## What Changed

- Lease renewal on its own connection (`packages/db/src/client.ts`,
`packages/db/src/index.ts`,
`server/src/services/controller-instance-lease.ts`):
- New `controllerInstanceLeaseDb(db)`: one private 1-connection pool
(`application_name` `paperclip-controller-instance-lease`), made the
same way as the existing `controllerLeaseDb` and closed by
`closeLivenessDb`.
  - The renewal is one statement on that connection, as before.
- The connection sends `statement_timeout` = 8 s (below the 10 s client
timeout) as a startup parameter (new
`DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The
server cancels a slow renewal, so it never holds the single connection
that the next renewal needs, and there is no extra round trip per
renewal. (An earlier revision set it per transaction; that added three
round trips to every renewal and broke the idle-server receipt test.)
- If a connection pooler refuses the startup parameter ("unsupported
startup parameter"), the lease logs a warning and renews on a
replacement connection without it.
- A crashed controller's runs
(`server/src/services/recovery/ownership-registry.ts`,
`server/src/index.ts`,
`server/src/services/native-runtime/native-restart-recovery.ts`):
- The ownership reconciler on the live server hands runs whose
controller is proven gone to restart recovery. It does not wait for
another server boot.
- The unverified-controller wait is bounded from the moment the
controller's lease lapsed, not from the first look.
- When the controller boot is `stopped` or `fenced`, the run ends at
once.
- A successor's wait record no longer refreshes the dead owner's sign of
life.

Safety of ending a run when its controller boot is fenced or stopped:
- Only `stopped` (written by the boot's own graceful stop) and `fenced`
(a newer epoch fenced it after its lease expired plus 30 s grace) count.
`live` and `grace` never count, so a controller that is alive and
renewing is never treated as gone. `expired` alone does not count
either.
- A boot that cannot renew stops itself at its self-fence window (about
its lease expiry), and a boot that finds itself fenced exits at once. So
when another server can fence it, the old process has already stopped
driving runs. The lease change above removes the main cause of healthy
boots failing to renew.
- If the old process is still alive (partition or stall) and its Runner
continues: the old run is fenced. Its agent token is refused for writes
(`agent_run_superseded`), its remote workspace epoch is abandoned or the
continuation gets a new workspace, and the native coordinator records
`replacementDenied`. There are never two writers in one workspace.

## Verification

Red-to-green tests (each failed before its fix and passes after it):
- `server/src/__tests__/controller-instance-lease.test.ts` "own
connection": red (10004 ms timeout while the main pool is held) to
green. Full file green (26).
- `server/src/__tests__/controller-instance-lease.test.ts` "has the
server cancel a renewal blocked on a row lock, and renews on the same
connection next time": red without the connection-level timeout (the
renewal runs to the 10 s client timeout and stays active), green with it
(cancelled at about 8 s, no active lease statement left, the next
renewal succeeds in under 2 s on the same backend).
- `packages/db/src/client-options.test.ts`: the statement timeout is
sent as a startup parameter.
- `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`:
"lapsed longer ago" / "end of the bound" (wait counted from the lease
lapse), the live-server re-check of a crashed controller, the
fenced/stopped boot ending the run at once, and the successor's wait
record. Full file green.

Changed existing expectation (it asserted the old wait for a stopped or
fenced controller):
-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
"keeps foreign local Runner and provider ownership after its controller
is stopped/fenced": the disposition is now `blocked`
(`controller_process_space_unverified`) instead of `awaiting_evidence`,
the run is `failed` with `native_restart_recovery_blocked`, and the
finalization is `terminal_failure` with `executorStopEvidence:
"unverified"`. The safety checks stay: no local PID probe, no claim, no
stamp, no local stop proof, the run keeps its process identity and
session checkpoint, and no result is written.

Local commands, on this branch based on current master:
- `cd server && npx vitest run
src/__tests__/controller-instance-lease.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140
tests passed.
- `npx vitest run src/__tests__/controller-instance-lease.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
225 passed, 14 skipped (real-process cases that need CI).
- `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts
-t "releases an ordinary held task Retry"`: passed (idle receipt: one
lease statement after the work ends).

Local swap cells (two servers, embedded Postgres, real agents, old
server SIGKILLed 40 s into a 200 s overlap):
- Before: runs of the killed server stayed in progress 21 to 28 minutes.
- N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the
successor's wait write still moved the timer; fixed by the last commit).
- R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4
hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run
end 3 min 39 s.
- N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min
01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the
lease TTL (90 s) plus the fence grace (30 s).
- Hour-long release-swap wave (w-20261112T233630-s7201) on the branch
that holds this fix and the related recovery fixes: 11/11 tasks finished
hands-off (master: 9/11). The lease and crashed-controller fixes were
visibly working in it.

## Risks

- A pooled database URL whose pooler refuses `statement_timeout` as a
startup parameter: the first renewal fails over to a connection without
it (logged), so the lease keeps working without the server-side cancel.
- One extra database connection per server, held while the server runs.
It is idle except for one renewal every few seconds.
- A run whose controller boot is fenced or stopped now ends at once
instead of after the bound. See the safety notes above: only a boot that
is not alive can be in these states.
- No migration. No schema change. Nothing changes before `turn.start` on
the normal path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
documented behavior or command changes)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 20:11:27 -06:00
DottaandClaude Opus 5.5 251c6d5c69 fix(recovery): settle a remote workspace save whose process died mid-save (#20326)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.

**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.

**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.

**Paperclip version or commit**
`master` at 1b7fe33432.

Refs #20291 (the quiesced case: a save with no active invocation) and
#20233 (the same ownership pattern for local workspace commands).

## What Changed

- `saveRemoteWorkspace` records the controller process stamp of each
invocation in `saveWork.owners` when it registers the invocation. It
removes that entry when the invocation settles. A module-level set marks
invocations that are still running in this process
(`isRemoteWorkspaceSaveInvocationLive`).
- The ownership registry owns a `remote_snapshot_save` with active
invocations by that stamp. A save whose active invocations have no
stamps was written before this change and is owned by its run, as with
finalize attempts. A mix of stamps, or more than one process, is
unprovable.
- `settleAbandonedRemoteSnapshotSave` settles a save with active
invocations once its owner is proven gone:
- Process-owned: the proven stamp must still match every active
invocation, and none may be live in this process.
  - Legacy run-owned: the run must be terminal.
- Either way it clears the active invocations, records them as
`abandonedInvocationIds`, and marks the save `failed` with `lastFailure:
remote_workspace_save_abandoned`. The #20291 re-arm of blocked fallbacks
still applies.
- The controller instance lease counts running saves stamped with this
boot as held work, so a long save keeps its owner provably alive.
- The ownership inventory row for this kind describes the stamp.

## Verification

- Red before the fix, green after: `settles a remote snapshot save whose
process died mid-save, and keeps live, foreign or unprovable ones`
(`ownership-reconciliation-report.test.ts`). On master it settles 0
rows. With the fix it settles exactly two:
  - a save stamped by a dead process (proof `pid_dead`);
- the wave's shape: an unstamped active invocation on a `process_lost`
run.
It keeps five kinds of save running: a save stamped by this live
process, a stamp from another PID space, a legacy save on a live run, a
save with mixed stamps, and (in the older #20291 test) a stamped active
invocation.
- `stamps each save invocation with the process that runs it and clears
the stamp when it settles` (`remote-workspace-save-concurrency.test.ts`)
pauses a real save mid-flight. It checks the owner stamp and the
in-process live mark, then checks that both are cleared after the save
finishes.
- The reconciler, save-concurrency, remote workspace lifecycle, snapshot
retention, retained-source fallback, preparation gates, and controller
instance lease suites pass (153 tests, 1 environment-gated file
skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Not yet verified: the full Daytona wave replay that shows the queued
retry starts after the save settles. I asked the chaos harness owner to
replay it on this build.

## Risks

- A legacy save (written before this change) with an active invocation
is settled when its run is terminal. The finalize-attempt kind uses the
same rule. During a rolling deploy, an old-release process could still
be running such a save for a terminal run. If that save later succeeds,
its publication still marks the operation `succeeded`.
- If the retry in the wave is also held by the dead writer's dirty
workspace authority, settling the save may not be enough to start it.
The replay will show whether a second change is needed.
- No schema or API change. `saveWork.owners` is new optional metadata.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:27:44 -06:00
Devin FoleyandPaperclip 9b71d03215 fix: recover interrupted native preparations and unblock upgrades (#20177)
## Thinking Path

> - Paperclip manages AI-agent companies and their work.
> - A native run prepares its workspace before a runner PID or
checkpoint exists.
> - A server crash can interrupt that stage and leave its controller on
another host.
> - Recovery needs a durable owner proof, a delivered recovery wake, and
the saved execution contract.
> - The old paths missed controller lease registration and boot proof,
skipped checkpoint-free preparations, and counted retained work as live
upgrade blockers.
> - This PR connects preparation to the existing controller lease and
recovery paths, and separates retained work from the upgrade census.
> - The result is bounded recovery without a second executor or loss of
saved work.

## Linked Issues or Issue Description

**What happened?**

A native preparation interrupted by a server crash could remain running
indefinitely with no runner PID or checkpoint. Native controller stamps
did not wake the shared instance lease. That lease was disabled on
volume-backed Railway services, and native takeover omitted the boot
binding needed for its cross-host proof. Scoped recovery selected
checkpoint-backed runs but skipped these preparations. The upgrade probe
then counted the retained row as current execution and could block every
later upgrade.

**Expected behavior**

Track the preparation's controller before a runner starts. Recover it
through the normal classifier after proving its owner stopped or fencing
that boot. Keep live or unprovable owners held. Deliver recovery wakes
to preparations without checkpoints. Preserve unresolved historical work
and report it separately from current execution during upgrade
observations.

**Steps to reproduce**

1. Save an observed native preparation with no runner process, result,
provider-start evidence or checkpoint.
2. Replace its controller with a process in another PID space. Let its
owner lease expire and send a scoped recovery wake.
3. Before this fix, recovery skips the checkpoint-free preparation or
cannot prove its foreign controller stopped. Upgrade readiness continues
to count that old row as a live blocker.

Related work: #18037, #19651, #19868, and #20168. The foreign-PID guard
from #19868 remains. The queued first-turn input fix from #20168 is
included through the current base.

## What Changed

- Register the existing instance lease when native execution and restart
claims are written. Keep passive identity reads free of ownership
effects. Keep that lease enabled on volume-backed Railway services
unless explicitly disabled.
- Preserve the existing PID/start-time proof within the same readable
PID space. Bind foreign or unknown-space takeover to its saved boot and
last write. A live lease or grace period keeps the run held with a
scheduled check. Repeated evidence waits preserve their timestamp.
- Include explicitly fenced, checkpoint-free preparations in the
existing finite recovery candidate inventory. Normal launch, checkpoint,
provider evidence and admission checks still decide whether work may
resume.
- Expose the existing board stop-confirmation operation for historical
foreign-host owners. Bind confirmation to the exact run, boot, PID,
process start and host. Reject changed ownership.
- Report narrowly defined old, unowned preparations as
`retainedBlockedRuns` outside the live upgrade census. Local executions,
pending admission, drain state, truncation and positive checkpoint proof
retain their barriers.
- Update the execution contract and add PostgreSQL, real-route, race and
preservation regressions.

## Verification

- Genuine red/green regressions cover the old upgrade census, missing
native lease registration, the volume-backed default, foreign-host
confirmation, and scoped recovery skipping checkpoint-free preparation.
Test bodies ran against real PostgreSQL before their corresponding
production changes.
- Combined verification before the final idle correction (head
`055be6c84d9`): **459 passed, 22 skipped, 11 files passed**. Covers
controller lease/identity, native restart classification and
integration, host confirmation, heartbeat recovery, instance route
authorization, upgrade readiness, and queued first-turn input. The
focused command used `pnpm exec vitest run` with those 11 suites and
`--maxWorkers=1 --no-file-parallelism`. The 22 skipped cases require the
Rust debug fake-Codex fixture, which was not built locally; the
real-process CI coverage must pass.
- `pnpm --filter @paperclipai/server typecheck`: passed on head
`055be6c84d9b058064daad0af71bb5d893ab9529`.
- Actual route proof uses the real heartbeat service and PostgreSQL.
Unauthorized access returns 403. The retained run and owner records
remain unchanged. Recovery integration admits one bootstrap claim across
two replacement controllers, keeps a live owner held, and preserves
local execution.
- `pnpm verify`: exit 2, requests broad CI because the execution
contract changed. Full local suite not run. Current-head CI must supply
broad coverage.
- Local source checkout on macOS ARM64, Node v25.6.1, pnpm 9.15.4. Tests
use isolated PostgreSQL. No live provider restart is claimed by those
fixtures.
- CI found an idle-query regression in the first version of the lease
wiring. Two focused PostgreSQL regressions reproduced the misplaced
identity-read notification and the missing claim notification. The fix
moves registration to actual ownership writes. `env -u npm_execpath pnpm
exec vitest run server/src/__tests__/maintenance-server-routes.test.ts
--maxWorkers=1 --no-file-parallelism`: **17 passed**, including 90 real
seconds and two qualified hourly windows with no extra idle queries. An
initial local setup attempt inherited npm metadata and did not run
tests; clearing that metadata selected the correct pnpm helper. Three
focused controller/native suites also passed **168 tests, 22
fixture-gated skips**. Server typecheck passed after the correction on
head `4a527326dcd5b3a9e286f0e4c3e761d3056af4d9`.
- The older CI server shard also exposed a same-space graceful-restart
regression. The new boot fallback had overridden local process-exit
proof. Seven suspension cases and a focused unit regression reproduced
it; the fix limits boot fallback to foreign or unknown PID spaces. Fresh
verification on `acf1065d98150210c1e227c7f3b05a1eb489730b`: **251
passed, 22 fixture-gated skips, 4 files passed**, including all **40**
graceful-restart suspension tests. Server typecheck passed. The local
regression failed before this correction, and the same-space and
foreign-space paths pass together after it.
- Current-head CI completed successfully on
`acf1065d98150210c1e227c7f3b05a1eb489730b` (run 42568109784). Greptile
reviewed that exact head at 5/5 with no unresolved review threads.
- Secret and private-data scans passed. No workflow, lockfile or
migration changes.

## Risks

- The instance lease now applies to volume-backed Railway services. Its
existing 30-second renewal, quiescence, grace and self-fence behavior
applies while work is held. Idle servers retain the existing no-query
behavior. An explicit `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` remains
available.
- A controller lease proves only the server owner's state. It does not
prove an agent process stopped or authorize replay, resource release or
volume removal. The existing classifier and provider/runner checks
remain responsible for those decisions.
- Previous-release records keep their shape and saved state. A missing
boot lease stays unprovable and requires the scoped stop-confirmation
path. A changed confirmation target is rejected.
- The upgrade observation does not recover a held task by itself. Source
tests do not qualify a hosted replacement. Deployment and the affected
task's useful result must be verified separately.
- This repairs recovery after a crash; it does not claim to identify or
fix every source of heap exhaustion.

## Model Used

OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3 Code,
with repository, shell, and GitHub tools. Context window size is not
exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-08 21:53:13 -08:00
DottaandClaude Opus 5.5 e8a3f1f6f2 feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, the control plane settles in-progress records
(leases, claims, intents) only once it proves their owner process is
gone
> - The proof compares host and PID space, so a stamp from a replaced
container or a rebooted machine stays unprovable forever, and its
records never settle
> - Self-hosted users on Docker, Kubernetes and Fly hit this on every
redeploy; a Railway deployment with an exclusive volume is covered by
`deployment_replaced` (#20026)
> - This pull request adds a fenced controller epoch: each boot holds a
database lease, and an expired boot is fenced so its records can be
proven gone
> - The benefit is that work stranded by a replaced container is
recovered, without keeping an idle database awake

## Linked Issues or Issue Description

Follow-up to #20026 and the Phase 2 durable-ownership data. Related:
#19965 (owner stamps), #19883 (process space). No duplicate PR was
found.

**What happened?**

A record stamped by a server process in another container reports
`different_process_space`. A dead PID elsewhere is never proof, so the
record stays `unprovable` and is never settled. Every redeploy of a
self-hosted container leaves such records behind.

**Expected behavior**

A record owned by a server boot that was replaced (stopped cleanly, or
lost its lease and was fenced) is proven gone and settled exactly once.
A live boot's records stay with it. An idle server issues no recurring
queries.

**Steps to reproduce**

1. Run Paperclip in a container and start work, so an environment lease
is stamped by that boot.
2. Replace the container. The new container has a new PID namespace.
3. The ownership reconciler reports `different_process_space`
(unprovable) for the old lease, forever.

**Paperclip version or commit**

`master` before this PR.

## What Changed

- New `controller_instances` table (migration 0449): boot id, epoch
(bigserial), process space, pid, a lease on the database clock,
`fenced_at` / `fenced_by_epoch`, `stopped_at`.
- `controller-instance-lease.ts`, registered with the shared
delivery-work coordinator (`controller-instance-lease` queue, runs
during drain):
- **Holds work → renews; idle → silent.** One upsert creates the row on
first use and extends the lease only if the boot holds work: a stamp
within the last 90 s, or a stamp whose ownership write may still be
pending (held until the server's tracked work settles, capped at 10
minutes; a renewal that sees some other held record does not clear it,
so a slow ownership write cannot be outlived), or an in-progress record
held by its boot id. Those records are active environment leases,
in-flight cleanup claims, sandbox creation intents, workspace finalize
attempts, agent lifecycle steps, native finalizer and cleanup leases
(`lease_holder` bound to the token), open native coordinator claims, and
running legacy runs. Renewal runs every 30 s while the boot holds work.
When it finds none, the lease lapses and the worker sleeps until the
next stamp (`onControllerStamp`).
- **No idle trailing statements.** When the server's last tracked work
settles, the worker re-checks holdings at once, at most every 5 s (the
rate limit also absorbs the worker's own run). So the last check happens
with the work, not 30 s later.
- **Quiesce, don't crash.** Each renewal has a 10 s bound. A failed or
timed-out renewal applies an internal task drain, so no new work is
dispatched. A recovered renewal resumes. The boot exits only after 60 s
of failure (counted from the first failed attempt's start) while it
still holds work, or at once when it finds itself fenced.
- **Fence on demand, by a newer epoch.** For a stamp from another host
or PID space, the prover reads that boot's row:
    - `stopped` or `fenced`: gone.
    - live lease: alive.
- otherwise, the boot's last sign of life is the later of its lease
expiry and the record's last write plus the TTL. Within 30 s of that,
the owner stays `alive` (`controller_lease_grace`), so the settle watch
keeps rechecking. After that, the prover fences the boot, but only with
a lease row of its own (created on demand) and only an older epoch.
  - A graceful shutdown records `stopped_at`.
- A reconciliation pass that settles nothing (`record: false` or
`settle: false`) never fences and never creates the boot's row.
- `owner-proof.ts`: these proofs apply only to server-held records. A
stamp with a recorded process group stays unprovable, because agent
processes can outlive their server. The reconciler passes each record's
`updatedAt` to the proof.
- `task-admission.ts`:
- the quiesce drain never replaces an operator's drain, and has no idle
owner, so HTTP stays open;
- a `subscribeIdleWorkSettled` hook fires when the last tracked work
finishes.
- Off on a Railway deployment with an exclusive volume (it has its own
replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on|off`
overrides this. A stateless Railway service keeps the lease. A server
without its own lease row never fences.
- `maintenance-server-routes.test.ts`: in the post-run quiet window, the
lease may check once more that the run's records were released. The test
now excludes those statements, then asserts that the lease issues
nothing in a further window.
- Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the
two known gaps.

## Verification

- `controller-instance-lease.test.ts` (15 cases):
- a replaced container with all Railway variables empty is fenced and
settled exactly once over two passes;
  - a live lease is alive;
- an owner in grace, and work written after an idle lapse, stay alive
and watched;
  - an older epoch never fences a newer one;
- no fencing without its own lease row (off or unregistered), and an
idle prover creates its row on demand;
  - a stopped boot is gone;
  - process-group stamps stay unprovable;
- idle issues zero statements, and one upsert renews only while records
are held;
  - **a 45 s outage:** quiesced, no exit, resumes;
- **a renewal that never answers:** bounded, and the boot exits on time;
- **a 70 s outage:** exits at 60–65 s, and the successor recovers the
work exactly once;
  - fenced: stops;
  - the settle re-check;
- graceful stop; off on a volume-exclusive Railway deployment, on for a
stateless one, and the on/off override;
  - a running legacy run keeps the lease live.
- Red, with the proof's instance check removed: the replaced-container,
live, stopped and successor cases fail.
- Wider: task-drain, idle-sleep, idle-local-work, owner-proof and
ownership suites, plus the maintenance quiet-window tests. Server
low-memory typecheck is clean.

## Risks

- **Clock skew.** The proof compares a record's `updatedAt` (often the
writer's clock) with the lease (database clock). The 30 s grace absorbs
ordinary skew.
- **A new exit path exists.** A server whose database is unreachable for
more than 60 s while it holds work now exits, after first quiescing.
This is the intended safety backstop.
- **Bounded trailing activity.** A lease that holds work issues one
statement every 30 s, plus at most one check when its work settles. An
idle server issues none.
- **Boots from before this release** have no row, and their records stay
unprovable. The board "mark owner gone" action for those follows in a
separate PR.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 11:06:32 -06:00