Commit Graph
2 Commits
Author SHA1 Message Date
DottaandPaperclip db11d899cf fix(native): recover remote sessions while preserving warm runs (#16451)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native agents keep remote processes and conversation history across
warm turns.
> - A completed turn can leave handoff authority only in controller
memory.
> - Losing that controller can block a follow-up even while its sandbox
survives.
> - This pull request verifies durable ownership and reconnects the
exact prior process before handing it to the next run.
> - Healthy follow-ups stay warm. Recovery preserves saved work without
replaying a completed turn.

## Linked Issues or Issue Description

**What happened?**

A remote warm turn can finish without suspended backup authority. A
replacement controller rejects that prior owner even when its exact
process and completed checkpoint remain. A single unavailable identity
probe can also stop an otherwise valid reattachment.

**Expected behavior**

Keep healthy follow-ups warm. Recover a completed session after proving
that its controller stopped and that the successor holds current
authority. Reuse compatible processes; otherwise save and resume the
conversation. Retry unavailable identity reads within one shared
deadline.

**Steps to reproduce**

Use `doc/testing-native-remote-continuity.md`. Its Linux cases execute
real turns through PostgreSQL, TLS ingress, and the Rust runner. They
kill a separate controller process, recover old checkpoint metadata, and
change process configuration. The provider is deterministic.

Related: #16432 and #16441 fix live warm-owner handoffs. #16341 adds
recovery diagnostics. #14695 covers provider process retention.

## What Changed

- Preserve warm process reuse between healthy remote turns.
- Verify a committed prior result, stopped controller, matching
checkpoint and sandbox, and current task/coordinator ownership before
recovering an orphaned warm session.
- Reconnect through the prior run route, then use the existing
authenticated run attachment to change authority.
- Persist a process-configuration digest. Close and save recovered
sessions when old or changed configuration cannot support reuse.
- Retry unavailable state, marker, and process reads within one
30-second deadline. Keep signals and definitive identity failures
single-attempt.
- Record closed recovery diagnostics, including unreadable or malformed
saved state.
- Retire the recovery-only ownership check after verified attachment, so
later warm turns use their current authority.
- Add real-process Linux regressions and refusal controls.
- Expose the existing source-only file-recovery form for retained native
holds, with the same board authority, evidence, and exact stop-receipt
requirements as legacy holds. Recording recovered files does not change
task status or replay a run.

## Verification

- `pnpm -r typecheck` passed; the server check was repeated for the
third-turn fix.
- `pnpm build` passed after the final UI change. The UI typecheck, token
gates, and map checks also pass.
- Four Linux continuity cases passed: warm reuse, controller loss,
legacy checkpoint metadata, and changed configuration. Each creates one
conversation and starts exactly three turns, including a healthy
follow-up after recovery. The fixture uses expired prior leases; the
third turn keeps the runner and provider processes.
- Greptile caught the retained recovery gate; the third-turn regression
now covers it. An initial test omitted the heartbeat’s saved input and
left a test process after failure; the fixture was corrected and its
disposable container reset before the passing run.
- The current focused executor/restart group passes all 749 tests,
including the original failure classifications.
- Native file-recovery and UI tests pass (52 tests), including blocked
tasks and both native and legacy holds.
- After merging current master and retaining its equivalent CI
expression check, the CI policy tests pass (15 tests) and all four
three-turn Linux continuity cases pass again on
`dec418ae55ff21b4fb067132613658cce445f181`.
- `pnpm test:run` completed its general server group with 20,572 passes,
103 skips, and three failures: the skill-audit child process and two
HTTP test timeouts. The same three suites then passed in isolation (89
tests) without code changes. That local command stopped before later
groups; the complete final-head CI, including general, serialized,
Runner, build, and browser gates, is green.
- Greptile reviewed the combined head
`dec418ae55ff21b4fb067132613658cce445f181` at 5/5 with no actionable
findings; both earlier findings are resolved. All final-head CI gates
and the security scan passed. Staging deployment and retained-incident
disposition remain deployment acceptance work.

## Risks

- Recovery requires exact durable evidence. Live conflicting
controllers, replacement sandboxes, Stop/pause, and uncertain effects
must remain blocked.
- After controller loss, tool-contract or process-configuration changes
can restart the provider. Normal healthy follow-ups retain their
processes.
- The new idle-recovery path has Linux fake-provider evidence. Live
Codex/OpenCode and retained quarantined-state recovery still need
staging qualification.
- No database schema, telemetry, or runner protocol changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution, and browser tools. The session does not expose a more
specific model version or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks and
isolated reruns; full-run limitation recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on
`dec418ae55ff21b4fb067132613658cce445f181`
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 13:26:09 -05:00
Devin FoleyandPaperclip 5a6d0d08a7 fix: record verified workspace file recovery from tasks (#16104)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Recovery must preserve files and show the next operator action.
> - A stopped run can leave a file repair hold after its recovery action
is resolved.
> - The task notice shows that hold but has no form to record verified
file recovery.
> - A chat connection can also replace the file repair instruction with
a chat retry instruction.
> - This pull request adds a file recovery form and preserves the file
repair instruction.
> - The form records evidence for the original run and keeps the current
task status.

## Linked Issues or Issue Description

**What happened?**

A retained workspace file hold can block a task after its recovery
action is resolved. The task page offers no file recovery form. A chat
binding can hide the required file repair instruction.

**Expected behavior**

An authorized board member can record verified recovery of the original
files. The server checks the original stop proof. The task keeps its
current status and does not start another run. Chat instructions must
not replace file repair instructions.

**Steps to reproduce**

1. Create a stopped legacy run with a retained workspace recovery marker
and a resolved recovery action.
2. Open its task page, including a task with a chat connection.
3. Observe that the file hold has no supported form to record verified
recovery.

Refs: #15857, #15824, #16095. Those PRs address shutdown export, native
finalization and workspace source isolation. This PR uses the existing
file repair API.

## What Changed

- Add a board-only file recovery form with evidence and an explicit file
verification confirmation.
- Preserve the current task status when recording file recovery.
- Project whether the original recovery marker supports file recording.
- Keep file repair instructions when a task has a chat binding.
- Add UI, route and Storybook examples for success, denial and
chat-connected file holds.
- Document the recovery flow and update the affected Storybook catalog
entries.

## Verification

- Merged master `c8efdbfba43` and resolved both rounds of catalog/index
conflicts. Upstream review notes and classifications are preserved,
together with all four retained-file recovery examples and their stable
story IDs. Both fix branches also combine without merge conflicts or
catalog fingerprint drift.
- Passed locally after conflict resolution: 47 tests covering the file
recovery UI, execution blocker notice and retained workspace repair
route. The earlier ownership-route pass also covered 196 tests.
- Passed: UI token gates, the complete Storybook catalog check with no
drift errors, feature and agent map structure checks, and all 17
catalog/index script tests. Verification inventory counts match all 262
authored suites and 2,485 story IDs. The upstream production
verification obligations remain intact.
- Passed: full workspace typecheck, build and Storybook build on latest
head `10ac22b35d0a4fe469cf620914845c509046da48`. The built index retains
all four file-recovery story IDs.
- Latest head `10ac22b35d0a4fe469cf620914845c509046da48` has 54
successful checks and two intentionally skipped Storybook checks. Apex
is 5/5 on that exact commit, with no unresolved findings. The full
general/serialized suites and all eight browser shards passed in CI.
Local tests use the focused suites above. Storybook browser interactions
have not been executed locally.
- Manual check: open a task with a retained file hold. Verify that
evidence and confirmation are required. Record verified file recovery.
Confirm that its status stays unchanged and no run starts. Repeat
without board runtime access and with an unverified stop receipt.

## Risks

- The form records an operator attestation. It does not recover files
itself or prove external action outcomes.
- A stale source or missing stop proof must remain rejected by the
existing server route.
- No database migration or automatic task retry is added.

## Model Used

OpenAI GPT-6 (`gpt-6`), used through Codex with reasoning, repository
tools, code editing and test execution. The exact context window size is
not exposed by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-13 13:47:10 -05:00