Files
PaperClipAI/server
DottaandPaperclip 016e40d532 fix(recovery): preserve shutdown export ownership through finalizer drain (#16843)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Legacy adapters must save sandbox files before Paperclip releases
their source.
> - Graceful shutdown gave each adapter a shared 30-second settlement
wait.
> - When that wait ended, cleanup could stop the sandbox while the
adapter still exported files.
> - That race created a restore failure and blocked the next turn.
> - This pull request keeps export ownership through the finalizer drain
and renews its cleanup fence.
> - A successful late export can release its source and schedule one
bounded continuation.

## Linked Issues or Issue Description

Refs #15857. This corrects the timeout path introduced by the shutdown
settlement fence.
Refs #16478. That repair clears verified historical holds; this change
prevents an active export from losing its source.
Refs #16835. That merged fix handles remote native runner ownership.
This change handles legacy shutdown settlement.

## What Changed

- Keep a late adapter settlement in the shared execution finalizer set.
Wait for it before lease release, recovery classification, and retry
scheduling.
- Renew the exact owned settlement fence every 10 seconds. Use a
30-second lease, one renewal in flight, a one-second database lock
timeout, and a five-second statement timeout. Report bounded failure
categories without raw database errors or credentials.
- Refuse ordinary cleanup while the local adapter still owns settlement.
Preserve the explicit owner stop-and-retain path.
- Give the finalizer drain a further shared 30-second wait after the
initial shutdown drain.
- Document the ownership rule, the acceptance recipe, and the
controller-loss limit.

## Verification

- Red before the repair: the database-backed expired-wait regression
expected cleanup denial but received permission to stop the still-owned
source.
- Green on `615a4382c5`: all 526 tests in `recovery.test.ts`,
`legacy-workspace-restore-recovery.test.ts`, `shutdown.test.ts`, and
`heartbeat-process-recovery.test.ts` passed.
- The assembled heartbeat regression passed. It verifies no early
release or retry, joining the late owner, successful cleanup, one
durable scheduled continuation, and no duplicate after service reload.
- Safety coverage includes other-controller fencing, controller loss,
exact source retention, stop-only cancellation, completion winning the
status race, and real database lock contention during renewal.
- `pnpm -r typecheck`, `pnpm build`, and `node scripts/check-maps.mjs`
passed on the final source revision.
- A broad local `pnpm test:run` crossed the rebase and reported eight
companion supervision failures. It was stopped and its log retained. The
complete companion file then passed all 753 tests on current source.
This is not a passing full local suite; current-head CI covers the
complete repository checks.
- The 34 recovery tests include real lock-timeout classification and
credential-shaped error redaction.
- Current-head CI: 53 checks passed; two optional Storybook checks
skipped. Greptile is 5/5 on `615a4382c5`; no unresolved review threads.
- Live sandbox acceptance remains unverified. Three disposable
local-controller/Daytona attempts used legacy Codex ACP, `gpt-6-astra`,
API-key authentication, and the verified image
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:a2bd08cefb68d9217bdd75cf87a6b51f1705b18a4796eff3f97b2726a548b8fb`.
ACP initialization exited before model execution on macOS Node 26.4.0
and again on Node 24.21.0; one attempt exposed a local process-session
socket `ECONNRESET`. That does not establish the transport failure's
root cause. No attempt reached the delayed export or second turn, so
this PR remains draft pending faithful live acceptance. Probe setup
defects were retained separately from product failures, and all
test-owned sandboxes were cleaned up.

## Risks

- Shutdown can spend up to 30 more seconds joining finalizers. The
shipped Compose stop grace remains 90 seconds.
- Renewal adds at most one database operation per owned run every 10
seconds during shutdown. Lock and statement deadlines bound database
contention. The local adapter still protects its source if renewal
fails.
- Controller death or an export that outlives the whole process still
needs retained-source repair. This change does not reconstruct a legacy
export after process death or clear historical failed-restore holds.
- No schema or public API changes. Native restart ownership, approval
gates, and retry limits retain their existing behavior.
- This PR has not been deployed. It does not recover an existing
retained sandbox by itself.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection, code
execution, and test tools. The exact backend model identifier and
context window were not exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 09:19:09 -05:00
..