Files
PaperClipAI/doc
DottaandPaperclip 61b2455319 fix(runtime): preserve failed sources and durable checkpoint authority (#19038)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free web server must preserve retained native session data.
> - An old failed follow-up can keep its own snapshot while its
controller file still belongs to the preceding successful turn.
> - The Runner correctly refuses to import that file as the failed
turn's execution authority.
> - That refusal also blocks migration when no warm supervisor is
present.
> - This pull request preserves the file after it proves the older run's
lineage, and uses existing typed database authority when present.
> - Failed history and files survive, while normal execution authority
stays unchanged.

## Linked Issues or Issue Description

Refs #19024, which preserves the related older-supervisor case. The
duplicate search found no PR for this older-controller case.

**What happened?**

Offline native-controller migration stops with
`native_runner_checkpoint_binding_mismatch` for a failed retained
follow-up. The local controller still contains the preceding successful
run. A per-turn source can have no supervisor file, so the
older-supervisor archive path does not apply. Read-only checks matched
the database snapshots and controller-file hashes between the original
source and the isolated rehearsal copy.

**Expected behavior**

Preserve the available file as evidence through the existing encrypted
archive. Require a valid older controller, a completed successful prior
run, and the same company, agent, task, native session, Runner and
provider thread. Keep successful sources and active owners outside this
archival fallback. When a valid typed database checkpoint already
exists, derive only its missing recovery metadata and ignore obsolete
local projections.

**Steps to reproduce**

Create a successful native turn and a later failed follow-up in the same
session. Retain the successful controller file. Keep the failed snapshot
and admitted input in the database. Leave the per-turn source without a
supervisor. Run offline migration. The new positive regression fails
with the binding mismatch before this change.

## What Changed

- Share the scoped older-successful-run proof with the existing
supervisor preservation path.
- Validate the older controller through the Runner parser and artifact
checks before archive publication.
- Archive this failed legacy source without changing its snapshot,
status, ownership, queued work or session pointer.
- Preserve the normal refusal for foreign, changed, active, successful,
invalid and already-durable cases.
- Permit evidence archival for a stale observed coordinator only with
exact host-owned stop receipts for every remote lease, no current task
ownership, no result and no retry deadline. Recheck the proof before
publication.
- Use execution start for lifecycle ordering. A queued follow-up can
predate its prior turn completion, and a remote stop can predate a later
recovery timeout.
- Resolve destruction receipts through the existing reusable-lease
handoff verifier. Reject stopped tips, broken chains and competing tips.
- Prefer a valid typed database checkpoint over a stale or absent local
warm supervisor. Add only missing recovery metadata after current
admitted-input and checkpoint binding checks.
- Document the evidence and execution-authority boundary.

## Verification

- Positive regression: red before the owner change and green afterward.
- `pnpm exec vitest run
server/src/services/native-runtime/native-controller-migration.integration.test.ts
server/src/services/native-runtime/native-controller-recovery.integration.test.ts
server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
81 checks passed across migration, recovery and durable storage.
- The checks cover storage failure, repeated migration, exact file
restoration after old-home deletion, unchanged history and ownership,
and refusal to treat the archive as executable recovery authority.
- A second regression reproduced the archive fence refusal for an
observed failed source with a valid stopped-remote receipt. Negative
controls cover missing, foreign, old and revoked receipts, live leases,
pending results, retry deadlines and task ownership. Revoking the
receipt during upload prevents publication.
- Two additional regressions cover a follow-up queued before the prior
turn finished and a remote stop before a later logical failure. Both
fail with the earlier timestamp comparisons and pass with
execution-start ordering. Genuine execution overlap and old receipts
stay blocked.
- Two historical source snapshots and controller-file hashes matched in
a read-only live-versus-copy check. The first candidate reached the
archive fence and exposed the observed-coordinator case. That failure is
retained. A fresh candidate replay is required for the expanded proof.
It does not qualify the full packaged migration or hosted Railway
handoff.
- Existing typed-authority checks cover stale and missing supervisors,
repeat migration, byte-identical checkpoint rows, unchanged
snapshots/owners/history, old-home deletion and ordinary queued-turn
recovery. Foreign Runner, session and snapshot controls remain blocked.
- Handoff checks cover an exact destroyed successor, a stopped
successor, a broken proof and a competing tip.
- Isolated replay of the two retained typed-checkpoint sources imported
only derived recovery metadata. Checks confirmed unchanged checkpoint
rows (12 and 9 parts), history, snapshots, finalization owners and
task-session pointers. The replay used three source-module overlays with
a blocked provider network; it does not qualify the released bundle.
- Final-head CI attempt 1 passed typecheck, build and the migration
suites but failed one unrelated budget-concurrency assertion in
`decision-models.test.ts` (one provider call observed instead of two).
The original log is retained. The isolated local comparison cannot
classify the timing failure because this checkout has stale linked
budget schema exports. The one retry of that failed CI job passed, and
final-head CI attempt 2 is green. The original failure remains retained;
its cause is not established.
- Full local typecheck, repository tests and build are not claimed. The
server typecheck reports pre-existing linked declaration mismatches
outside the changed files; the full clean CI typecheck has passed. This
checkout uses linked generated declarations; clean CI is required before
merge. Prior head d71dc913 had passing full CI and a 5/5 Greptile
review. Head 8ff3a22109 adds the handoff and typed-authority fixes. Its
fresh full CI is green and Greptile is 5/5, with no unresolved review
threads. Earlier head7729 hit unrelated 5-second CLI ZIP and 15-second
setup-token test timeouts; those failures remain retained.

## Risks

This path preserves evidence for a source that has already failed. It
does not make that source resumable. The Runner must accept the older
controller under its own identity and artifact binding. Scoped
successful history must match its session and provider thread. The
archive owner still requires stopped writers, a finished failed run and
no live lease. An observed coordinator also needs exact remote-stop
receipts after execution started, no accepted result or retry deadline,
and no current task ownership. The coordinator stays unchanged. Existing
durable authority stays authoritative.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and
code execution. The host does not expose context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run the relevant tests locally; verification limits are
listed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 15:37:54 -06:00
..
…
…
…
…
…