Files
PaperClipAI/doc
DottaandPaperclip 1a6017d646 fix(server): resume suspended sessions after controller replacement (#18554)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web server replacement must preserve a completed conversation as
well as active execution.
> - A warm remote Runner can suspend before the replacement admits
another turn.
> - The replacement restores its controller checkpoint from PostgreSQL,
but lacks the Runner suspension receipt.
> - It therefore tries to adopt a live process and rejects the safely
suspended source.
> - This change restores the verified receipt and preserves retained
launch files while the old authority is reconciled.
> - Later turns can keep the same provider conversation after the old
web server filesystem is deleted.

## Linked Issues or Issue Description

Refs #18552, #18543, #18366.

**What happened?**

The real rolling-deployment journey passed both HTTP handoffs, with 765
reads and 78 verified saves. The first three agent turns kept their
native and provider sessions. After the second replacement, the fourth
continuation failed with `remote_warm_session_not_idle`. The remote
Runner was suspended with an empty outbox and no pending terminal. A
focused regression reproduced that exact failure. After restoring its
suspension receipt, the same test exposed a stale launch-file read
during source recovery.

**Expected behavior**

A replacement can resume the same provider conversation from a settled
suspended Runner. It must validate the physical sandbox and all
persisted identity fields. A live source must still pass the existing
adoption checks.

**Steps to reproduce**

Enable the Linux Runner fixture for
`native-runner-restart-recovery.integration.test.ts`. Run the
`diskless-warm-suspended` cases. The fixture completes and gracefully
suspends each source, deletes the controller projections, and admits a
later turn from another process. It does not write a synthetic
suspension receipt.

## What Changed

- Reuse the existing per-turn suspended-receipt reader before warm
admission chooses live adoption or suspended recovery.
- Keep the receipt read retryable when a failed remote read leaves only
the controller projection.
- Leave every non-suspended warm state to the existing live-adoption
checks. A preliminary backpressure observation cannot reject admission
before that owner inspects the current Runner state.
- Preserve current sandbox binding, settled-outbox, exact-identity,
local-evidence and admission checks.
- Pass the structured launch specification to remote asset preparation.
Source reconciliation keeps the old authority's retained launch files; a
current-authority launch still stages its current files.
- Add two Linux Runner/PostgreSQL regressions for two controller-cache
deletions and three turns on one provider thread, including an
interrupted receipt read.
- Update the deployment qualification record, including the failed live
evidence and the successful staging database-idle observation.

## Verification

- Before the fix, the focused regression fails with the exact live
`remote_warm_session_not_idle` error.
- Both new regressions pass with real PostgreSQL and Linux Runner. The
provider is deterministic; this is not a live model qualification.
- The first receipt-only fix failed on the deleted controller skills
cache. The completed fix passes that same scenario without recreating
fixture launch assets.
- All 1,067 surrounding launch supervision, warm recovery, launch-asset
and controller recovery tests pass. The exact `cc23019691` server build
passes. All ten Linux continuity cases pass, including live adoption,
graceful suspension, per-turn resume, changed configuration and
failed-read retry. Each case checks provider-thread identity and exact
turn counts. The live Codex/Daytona journey passes on the same clean
source and exact build: 518 reads, 53 saves and 53 verified saves with
zero probe failures; both predecessor homes deleted; the active
execution completes once; all four turns preserve the provider
conversation and native receipts; saved files and the final command
output are independently checked. No automatic retries. Child credential
and object-storage bucket cleanup pass. Browser reconnection remains a
separate hosted acceptance check.
- The review refinement has a failing-before regression: only the
preliminary read reports transient backpressure; the subsequent adoption
reads the real Runner state. It and both suspension cases pass on the
final source (three cases), with server typecheck. The existing idle
check remains unchanged. An initial retry hit a fixture port left by the
intentional pre-fix failure; a fresh container resolved that
infrastructure error.
- Independent cleanup verification found two archived setup sandboxes
left by the live harness. Both were deleted and confirmed absent. The
failed automatic-cleanup check remains recorded.
- No claim of the ten-deploy Railway campaign or production canary yet.

## Risks

- A cold warm-session admission can now make one bounded remote state
read when local lifecycle evidence is absent. Its failure stops
admission and remains retryable.
- This path cannot replace contradictory local evidence, transfer
authority, or treat a suspended process as a live one.
- Source recovery must keep the previous authority's launch files until
reconciliation completes. The launch identity selects that behavior;
ordinary launches still prepare current assets.
- No schema migration or public API change.

## Model Used

OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

CI qualification note: first final-head run `41322376054` passed the
native/runtime and remaining suites but failed the unrelated
`automatic_message_on_reactivation` browser fixture at its
pre-reactivation deferred-queue count (0 rather than 1). The unchanged
test passed locally against isolated PostgreSQL (1 test, 43.6 s);
inspection found an asynchronous company-pause/fixture-seeding timing
window. Original CI logs and local evidence are retained. The single
failed-job retry passed; all final-head CI gates are green. Both reviews
are 5/5 and no review threads remain unresolved. The first local
embedded-Postgres launch failed before tests during cluster
initialization; the isolated external-Postgres test passed, and its
disposable database was removed after correcting the private launcher
cleanup call.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 10:08:51 -06:00
..
…
…
…