## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane recovers in-progress records after a server
restart through a shared ownership path
> - Run-less sandbox leases (login, probe, harness verification) were
released only by a five-minute idle sweep
> - That sweep releases by age: it leaves a dead holder's sandbox for
minutes and takes a live login session's sandbox
> - This pull request settles a run-less lease only when its stamped
holder is proven gone
> - The benefit is no stuck sandbox after a restart, and no live session
loses its sandbox
## Linked Issues or Issue Description
Related: #19965 (named the owner of login/probe leases), #19964 (the
run-lease settle this mirrors). No duplicate PR was found.
**What happened?**
The final Daytona chaos wave `w-20261108T083036-s9401` recorded I2
`environment_login_probe_lease` (`pid_dead`) on 2 records. These were
the codex agent's harness-verification leases. A restart killed their
holder. The leases stayed `active` until the five-minute idle sweep
released them. The same sweep releases any run-less lease after five
idle minutes, also when its holder is alive.
**Expected behavior**
A run-less lease is released as soon as its stamped holder is proven
gone. A lease whose holder is alive is never released, however long it
is idle.
**Steps to reproduce**
1. Start a Daytona wave: `node cli/node_modules/tsx/dist/cli.mjs
tests/restart-chaos/main.ts --env daytona --daytona-image <image>`
(K1/K4 faults).
2. Let a fault kill the server while an agent's harness verification
holds its sandbox lease.
3. See the reconciler report I2 `environment_login_probe_lease` until
the idle sweep releases it.
**Paperclip version or commit**
`master` at `8475826d2d`.
## What Changed
- New `settleEnvironmentLoginProbeLease`, registered on
`environment_login_probe_lease`. It runs only for a stamped holder that
is proven gone, and compares and swaps on that exact stamp.
- It moves the lease to `pending_cleanup` with the recovered-orphan
reason, so the cleanup sweep destroys the sandbox in its next tick.
- It leaves the lease alone when a legacy shutdown fence applies, or
when another live lease holds the same provider resource.
- The idle sweep for run-less leases now covers only legacy rows saved
before holders were stamped (no `leaseController` key). Stamped rows are
never released by age.
- `holderUnchanged` is now exported from the run-lease settle so both
settles share it.
- The controller stamp (`leaseController`, and every other
`ControllerProcess` stamp) now records the platform deployment when the
platform names one (Railway: project, environment, service, deployment,
volume). `controllerStamp` passes it to `proveOwnerGone`. A container
redeploy changes the PID space, and the old stamp would read
`different_process_space` (unprovable) forever. Now the existing
`deployment_replaced` proof shows the old holder gone. This also covers
`environment_run_lease_active` and the other kinds built on
`controllerStamp`.
- `currentDeploymentIdentity` in `owner-proof.ts` now delegates to
`currentControllerDeployment`, so there is one parser.
- Data doc: added the final Daytona wave, the six Daytona-only crash
points (4 pass, 2 not reached, no violations), and the cause of the
roughly 450 s codex TTFT. Runs waited behind the agent's harness
verification, which restarted after each fault; boot itself took about
32 s.
- Restart-chaos README: corrected which Daytona crash points a plain
task reaches.
## Verification
- Red, before the change:
`environment-login-probe-lease-ownership.test.ts` failed 3 of 6. No
settle happened, and the idle sweep released 3 live holders' leases.
- Green: `pnpm exec vitest run` passes for
`environment-login-probe-lease-ownership`,
`environment-run-lease-ownership` and
`heartbeat-orphaned-active-lease-sweep` (45/45), and for
`environment-lease-owner-stamps`,
`heartbeat-observed-workspace-preservation`,
`ownership-reconciliation-report`, `heartbeat/recovery` and
`workspace-recovery-warm-handoff`.
- Redeploy: a stamp from a replaced Railway deployment settles with
`deployment_replaced`. A stamp from the same deployment in another PID
space stays `active`. This was red when `controllerStamp` dropped the
deployment.
- Live-owner counter-cases: the current process, another live PID,
another PID space, and a lease re-stamped after the proof all stay
`active`. A read-only pass never settles.
- Not re-run live: the Daytona wave and crash points ran before this
fix. The next Daytona wave should show no
`environment_login_probe_lease` I2.
## Risks
- A stamped run-less lease whose live holder never releases it, because
of a bug in that process, is no longer released after five idle minutes.
It is released once that process exits. This is the intended trade:
never take a live holder's sandbox.
- On an ephemeral server without a platform deployment identity (not
Railway), a stamp from a replaced container is still unprovable. Such a
run-less lease stays `active` until the provider expires the sandbox,
and the reconciler reports it as unprovable. Before this PR, the
five-minute sweep released it by age. This is the same open gap as
unstamped legacy leases on an ephemeral server: there is no operator
action yet. It is listed as a Phase 2 follow-up.
- Legacy unstamped run-less leases keep the age fallback. These are rows
written before #19965, and no current code writes them.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with
tool use.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Server tests that need a real PostgreSQL process must use
./helpers/embedded-postgres.ts instead of constructing embedded-postgres
directly.
The shared helper creates a throwaway data directory and a reserved-safe
loopback port for each test database. This protects the live Paperclip
control-plane Postgres from server vitest runs; see PAP-2033 for the incident
that introduced this guard.
