Commit Graph
2 Commits
Author SHA1 Message Date
DottaandPaperclip 60a99307b3 fix(server): settle cleanup after reusable sandbox destruction (#18633)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web-server replacement must preserve work and let completed
cleanup become idle.
> - Several task turns can use one remote sandbox through recorded lease
handoffs.
> - The last owner can destroy that sandbox while an earlier owner still
waits for its own termination receipt.
> - A completed agent-file cleanup can also retain a retry deadline from
a competing callback.
> - This pull request resolves confirmed destruction through the
existing handoff contract and settles obsolete cleanup retries.
> - The benefit is correct cleanup recovery without deleting history or
weakening sleep checks.

## Linked Issues or Issue Description

Refs: #18554. Related work: #14695 handles warm agent-file ownership;
#13291 handles interrupted conversation continuation. This PR fixes
cleanup recovery after confirmed final destruction. It does not change
provider process reuse or start a new run.

**What happened?**

After ten qualified staging web-server replacements, a bounded
database-idle check failed. Completed native runs still owned pending
agent-file cleanup. Normal workspace archive confirmed final sandbox
destruction. An earlier lease in the same recorded handoff chain
remained unresolved. Two completed cleanup receipts also kept stale
retry deadlines.

**Expected behavior**

Resolve an earlier owner's cleanup only when the complete recorded
handoff chain leads to an exact provider-confirmed destruction receipt.
Clear a retry when its cleanup is already complete. Keep real or
uncertain work pending.

**Steps to reproduce**

1. Complete a remote run, reuse its sandbox for a second run, then
archive the completed workspace.
2. Recover agent-file collection for the first run after the final lease
confirms destruction.
3. Separately, reconstruct a completed cleanup receipt that still has a
retry deadline.
4. Observe that both records remain unfinished without this fix.

**Paperclip version or commit**

Observed on `1a6017d6460267ad89d7e1784e2b521c28d8e9ac`. The changed
source was unchanged in master
`97c4dc969e13364464765ffbde74955005bf44d3`, which is this branch's base.

**Deployment mode**

A managed ephemeral web server with PostgreSQL, object storage and a
remote sandbox. Regression tests use local PostgreSQL and no live
provider or model calls.

## What Changed

- Resolve remote termination at read time through the existing strict
reusable-lease handoff validator. Require one final owner and a valid
destruction receipt. Preserve historical lease rows.
- Use the same proof in agent-directory cleanup and remote-stop
inspection. Keep stopped, ambiguous, broken and cross-company chains
blocked.
- Reconcile completed cleanup receipts with stale retry deadlines
through the existing deadline worker. Compare the receipt before
clearing a deadline.
- Add recovery, refusal, restart quietness and concurrent-obligation
regressions.
- Document the cleanup contract and preserve deployment evidence, failed
checks and remaining acceptance work in the existing plan.

## Verification

- Both owning service suites passed 169 tests before the final receipt
comparison was added.
- The seven-case selection passed again after the release-race review
fix: `pnpm exec vitest run
server/src/__tests__/agent-directory-working-copies.test.ts -t 'stale
retry|destroyed successor chain|stale-deadline settlement'`.
- Repository-wide `pnpm -r typecheck` and `pnpm build` passed on final
review fix `7935d99bed`. The full `pnpm test:run` is still running.
Latest-head CI has 55 successful checks, three skips, and one neutral
review result. Greptile is 5/5. The Paperclip reviewer’s race finding is
fixed; its optional query-deduplication suggestion is documented and the
fresh lock-boundary checks are retained.
- The reviewer identified an early return after a lost receipt
comparison. The strengthened race regression failed before the fix and
passed afterward: release now acts on the returned authoritative row,
completes the new cleanup obligation, and preserves canonical files.
- The first focused attempt failed the new stale-deadline test because
the public release guard skipped settlement. That guard was fixed. The
original failure is retained. A sandboxed typecheck could not fetch pnpm
signature metadata; the normal network-enabled command passed without
changing verification policy.
- Prior deployed evidence: ten Railway replacements, 12,521 reads, 1,566
verified saves, 34 matching file hashes, no HTTP failures, actual
browser reconnects of 1.16–1.29 seconds, and one remote execution
retained across replacement. Two harness false alarms were separately
adjudicated without redeploying those steps.
- That deployment evidence predates this PR. Current-image natural idle,
automatic sleep/wake browser return and the isolated production canary
remain unverified. Production has not changed.

## Risks

- Cleanup authority must not cross a company or allocation boundary. The
existing validator checks the entire recorded chain; stopped successors
and unexpected owners remain unresolved.
- Confirmed destruction permits no-save cleanup. It does not claim that
uncollected agent files were saved.
- Read-time resolution adds a scoped lease-history query only for
expired reusable leases without direct termination proof. No new
recurring worker or polling cadence is added.
- The deadline update must not erase new work. It requires the same
cleanup receipt and preserves concurrent changes.
- No schema or public API changes are included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution and test tools. The precise backend revision and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 11:53:44 -06:00
DottaandPaperclip 663c44cb2b fix: continue conversations after confirmed remote runner stop (#13254)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Users can stop a run and send another message on the same task.
> - Remote runners need evidence from their sandbox provider that
execution stopped.
> - Local process checks cannot prove that a remote process exited.
> - This pull request records provider stop receipts and uses them for
conversation admission.
> - New user messages can proceed after confirmed cleanup without
repeating interrupted actions.

## Linked Issues or Issue Description

Refs #13237 and #13239. Related: #13163 covers app-restart recovery;
this change covers an explicit stop followed by a new user message.

**What happened?**

A stopped remote Claude ACP task kept its execution hold after Daytona
cleanup succeeded. Native runners also rejected remote process
identities and retained stale session cleanup gates. A message sent
during cleanup could stay deferred after the sandbox stopped.

**Expected behavior**

After the provider confirms that the old execution stopped, a new user
message starts a fresh turn. Pending user messages must not need another
message to trigger admission. Prior action outcomes remain recorded.

**Steps to reproduce**

1. Start a long-running task in Daytona with a legacy Claude ACP or
native ACP runner.
2. Cancel the run while its tool is active.
3. Send a new message immediately, or after cleanup completes.
4. Observe the execution hold despite the old sandbox having stopped.

**Paperclip version or commit**

Reproduced on master at 7b829efdf6. The
branch is rebased on current master.

## What Changed

- Add optional provider stop receipts to sandbox release and destroy
hooks. Old plugins remain compatible.
- Bind receipts to the company, run, lease, and provider resource.
Failed cleanup cannot supply stop authority.
- Acknowledge legacy remote cancellation after confirmed termination.
- Admit native user continuations using remote receipts instead of host
process checks.
- Retire only the settled cleanup owner matching the stopped company,
run, and provider resource. Isolate cleanup gates between remote
sandboxes, including two sandboxes owned by one run.
- Reconsider user messages deferred during remote cleanup through normal
admission, including successful later cleanup retries.
- Preserve receipts through cleanup retries and inline cleanup after
failed startup.
- Permit provider destruction after a terminal remote checkpoint
failure. Busy ownership still blocks destruction.
- Add regression tests and update execution semantics.
- Give the real preview fixture ten seconds for cold startup. Reuse
release-mode Rust artifacts for the filtered parity checks, avoiding a
duplicate debug test build that exhausted CI disk twice; test filters
and assertions are unchanged.

## Verification

- Recursive typecheck and build passed.
- Targeted server, Daytona plugin, and native runtime tests passed. They
cover missing or mismatched receipts, failed cleanup, local process
protection, exact cleanup ownership, and a message sent during cleanup.
- Live Daytona tests passed for legacy Claude ACP, native per-turn,
native warm, and a newly created native runner using disposable
sandboxes. Each original run was cancelled; its explicit follow-up
completed with no execution hold. The disposable case confirmed a new
sandbox after deletion.
- Native tests used the provider's Opus 5 selector, `opus[1m]`, and the
Linux runner bundle from the sandbox image. Existing checkpoint/sync
finalization warnings remained visible before the native follow-ups
reached committed success. This change does not repair those separate
warnings or guarantee recovery of uncheckpointed files.
- A direct live provider test also passed with the final delete-wait
change: the destroy hook returned its receipt only after Daytona
reported the sandbox destroyed. The final Daytona plugin suite passed
all 153 tests; its build passed.
- After rebasing on master, 284 targeted server/plugin tests and 313
native executor tests passed. The native session runtime suite passed
all 128 tests.
- The review regression passed all 145 tests across the continuation,
environment runtime, and pending-cleanup sweep suites. Recursive
typecheck and build passed again after that fix.
- The security review's exact-resource finding is fixed. Cleanup
completion requires the same provider-resource scope used during session
creation. All 128 native runtime tests, 61 server continuation/cleanup
tests, recursive typecheck, and build passed after this fix. The
two-sandboxes-in-one-run regression proves one receipt cannot retire the
other quarantine.
- Greptile reviewed the final commit at 5/5, the security scan passed,
and no review threads remain unresolved. The final native executor suite
also passed all 313 tests.
- The full local suite reached 10,616 passing tests before stopping on
four failures. All four now pass in focused reruns: the final-code
continuation/teardown suites, the attachment suite, and the native
session test after building its required fake-provider binary. The
original full run overlapped source edits and did not reach the
remaining groups; it is not counted as a full-suite pass.
- The preview exposure suite passed all 25 applicable tests (three
Linux-only tests skipped locally) after the startup allowance change.
Both release-mode Rust parity commands passed locally. All final-head PR
checks passed, including full server/workspace/browser test groups, full
native runner verification, build, typecheck, canary dry-run, and the
aggregate gates.

## Risks

- A provider must return a receipt only after confirmed termination.
Incorrect provider claims could permit overlapping execution.
- Older providers without receipts retain the existing hold. Missing
evidence, failed cleanup, active ownership, pauses, approvals, and
budgets still block admission.
- This change preserves unknown action outcomes and old checkpoints. It
does not authorize replay or alter historical runs.
- No database migration or telemetry contract change is required.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, code execution,
and browser tools. The exact backend revision and context-window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub references)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run targeted tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-11 15:18:37 -05:00