Commit Graph
5 Commits
Author SHA1 Message Date
Nicky LeachandClaude Opus 5.5 11ac90e31f fix(build): typecheck server/src/__tests__ and fix every type error (#18739)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server has about 1,100 test and helper files in
`server/src/__tests__`
> - `server/tsconfig.json` excludes that directory, Vitest does not
check types, and no script runs the tsconfig in that directory
> - So a test can call a function with the wrong arguments and still
pass, and no tool reports it
> - This gap hid 2,168 real type errors, and some of them hid tests that
did not test what they claim
> - This pull request fixes all of these errors and adds the directory
to the server layers from paperclipai/paperclip#18737
> - The benefit is that a type error in a server test now fails the
typecheck, with no baseline and no backlog

## Linked Issues or Issue Description

Refs paperclipai/paperclip#18737, which is merged and added the server
layers. Refs paperclipai/paperclip#18738, which is merged and checks the
test projects outside `pnpm -r`, so it is not a duplicate. Refs
paperclipai/paperclip#18137, which records that this directory has no
check.

**What happened?**

No tool checked the types of 1,074 of the 1,101 files in
`server/src/__tests__`. I searched `package.json`, the workflows, and
all tsconfig files. 27 files are in the program of another tsconfig, and
these 27 have 0 errors. 11 of the 27 are in
`helpers/maintenance/tsconfig.json`. No script ran that project when I
measured this. paperclipai/paperclip#18738 now runs it.

**Expected behavior**

The server typecheck checks these files, and a type error fails the
check.

**Steps to reproduce**

1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json
--noEmit`. It reports 6,536 errors and needs 12.3 GiB.
2. Add a type error to any file in `server/src/__tests__`.
3. Run `pnpm --filter @paperclipai/server typecheck`. It passes.

**Paperclip version or commit**

I measured the error breakdown below on commit `8dfd632452`, with
TypeScript 7.0.2. The branch now includes master up to
paperclipai/paperclip#18865.

**Deployment mode**

Not applicable. This issue applies to development and CI. No production
source file in the server changes.

### Error breakdown

Each row adds one change to the row above.

| Step | Total errors | Change | Cause |
| --- | --- | --- | --- |
| Config as committed | 6,536 | | |
| Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request
types were missing. |
| Set `rootDir` to the repository root | 2,436 | -2,011 | Config. |
| Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest
accepts `.ts` imports. |
| Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has
`Promise.withResolvers`. |
| Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8
`TS7016` errors and shows new real errors. |
| Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing.
|
| Count only files in `server/src` | 2,058 | -84 | Other packages: 53
Daytona, 29 Runner test, 2 UI. |

So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058
are real type errors in 314 files. The 29 Runner test errors are also
real, which gives 2,087 errors. Tests that master added after this
measurement had 81 more errors, so this pull request fixes 2,168 errors.

This table classifies the first 2,058 errors.

| Kind of real error | Errors |
| --- | --- |
| Wrong argument or assignment type | 1,023 |
| Stale or partial fixture objects | 525 |
| Property that the type does not have | 317 |
| Missing narrowing of null or unknown | 101 |
| Implicit any | 60 |
| Missing name or module | 16 |
| Other | 16 |

### What the type errors hid

Most errors were stale fixtures and loose mocks. These errors hid real
test problems, and this pull request fixes them:

- **A test that never ran.** `instrumentation.test.ts` required
`sdk-trace-base`, which is not installed. So 3 span tests were skipped.
They now run and pass.
- **Assertions that could not fail.**
- `issue-thread-interactions-service` used a company ID as an issue ID.
- `status-card-evaluation-work` compared `[]` with `[]` in a rollback
check.
- `slack-conversation-lifecycle` read `item.issueId`, which does not
exist.
  - `email-channels` used a permission key that does not exist.
- `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is
not exported. The value was undefined.
- **Postgres processes that did not stop.** 11 test files called `stop`
on the test database helper. That method does not exist, and the helper
method is `cleanup`.
- **Arguments that the service ignored.**
- `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft,
1Password, and Railway tests.
- 108 calls passed `"owner-user"` to a parameter that expects an options
object.
- 9 chat wizard fixtures passed `managementContext` as `source`, so they
ran `existing_app` and not `manifest`.
  - 2 tests passed a `{ userId }` actor that the function ignores.
- Inserts into `environments` passed a `companyId` column that the table
does not have.
- **Problems in tests that master added during this work.**
- `sandbox-creation-operations` read `database.connectionString` when an
external test database was set. In that mode `database` is undefined, so
the fixture would stop with a TypeError. It now uses the external URL.
- `idle-completed-history` set a Dot operation to `dispatched`. That is
a Muse status, and the Dot type does not have it. The fixture now uses
`admitted`, which is a real in-flight Dot status.

### Possible product problems, not changed

These may be real problems in production code or types. I did not change
them, because this pull request does not change production behavior:

- `decisions-service` `service.create` does not validate the lenient
`cancel_issue_tree` input.
- `createConnection` ignores `input.connectionPurpose`.
- The `createConnection` route schema default stores `{}` for
`transportConfig`.
- `prepareManagedAiRuntime` accepted the invalid value `mode:
"connection"`.
- `createBinding` accepts invalid `projectionClass` strings outside
class 3.
- `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by
spread.
- `routineVariableSchema.label` is optional, but `RoutineVariable.label`
is required.
- `ToolRunContext.projectId` may be too strict.
- `withdrawInteraction` reads `issue.status`, which its type omits.
- `issueService.list` and `reconcileNativeFinalizations` omit fields
that they return at runtime.
- `createGitHubExternalObjectProvider` handles `tokenProvider: null`,
but its type forbids this value.

## What Changed

- Fixed the 2,168 type errors. The diff changes 322 test and helper
files.
- Corrected `server/src/__tests__/tsconfig.json`. Editors and the
typecheck now read the same options and report the same errors.
- Added three `test-tree` layers to
`server/scripts/typecheck-layers.mjs`. They read the server layers as
declarations, and any error fails them. A hash of the path selects the
shard, so a new file does not move other files.
- Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests
reach Daytona plugin source, and the root install does not have
`@daytonaio/sdk`.
- Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use
`.js` extensions, because server tests import this file.
- Made `declarationsConfig` and `--declarations-only` skip the
`test-tree` layers, because these layers emit nothing. Other projects
read only the source layers. Added a test for this.
- Updated `doc/DEVELOPING.md`.

## Verification

Run these commands:

```sh
pnpm --filter @paperclipai/server typecheck
pnpm exec vitest run server/scripts/typecheck-layers.test.mjs
```

Results on macOS (arm64), Node 24, 16 cores. Each check ran after the
merge of paperclipai/paperclip#18738 unless the row says otherwise:

| Check | Commit | Result |
| --- | --- | --- |
| Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers |
`6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB |
| `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e`
| Exit 0, 0 errors, 42 seconds, peak 3.25 GiB |
| The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0
errors |
| `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` |
`01db5634e4` | 90 passed. These include real compiler runs for a pass, a
type error, and an invalid tsconfig. |
| `tsconfig.declarations.json` after a full server typecheck |
`0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No
generated config names a `tsconfig.layers.json` file. |
| Owned runtime test in `idle-completed-history` | `e8f2f448ac` |
Passed, with the fix that master later replaced |
| `idle-incoming-wake`, `plugin-idle-manager`, and the changed
`chat-channels` test | `6e45a08573` | 30 passed |
| Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed |
| The 19 server test files that changed since `eab4dcc1f4`, and
`typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The
2 failures are in `chat-channels` and are in the list of 7 below. |

The peak is the largest single process, from `/usr/bin/time -l`. Peaks
for the server typecheck were between 3.42 GiB and 3.56 GiB across 8
runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2
runs. I did not measure a cgroup.

**Runtime check of the fixes.** Types do not prove that a test still
works. So I ran all 309 changed server test files, 11,929 tests, before
the rebase onto paperclipai/paperclip#18737:

| Result | Tests | Detail |
| --- | --- | --- |
| Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests
that passed in a later run, as described below. |
| Failed | 7 | The same 7 tests also fail on the commit before the
fixes. |
| Skipped | 9 | These tests have their own skip conditions: `sentry` 5,
and 1 each in `bundled-plugin-registration`, `customer-success-cloud`,
`onepassword-connection`, and `workspace-runtime-https-live-exercise`. |

The 7 tests that fail with and without this change, on my machine:

- 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on
a `SKILL.md` file. The path is in the parent checkout, above the git
worktree that I used, so I expect that the worktree is the cause. I did
not prove this.
- `workspace-instance-cleanup` compares `/private/var/...` with
`/var/...`. This is a macOS symlink difference.
- 2 tests in `git-reproducible-managed-checkouts` get
`unknown_base_git_files` and expect `unpushed_managed_checkout`. I did
not find the cause.
- 1 more test in `chat-channels`, `materializes direct external-chat
finals once`, gets `undefined`. I did not find the cause.

CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`.

Failures during this work:

- My fix to `agent-runtime-connector-access` changed an invalid
permission key in the setup, but not in the assertion. The runtime check
found this. I corrected the assertion, and the file passes.
- `heartbeat-plugin-environment` failed one wait in the full run. It
passed three times alone, so I think that load caused the failure. It
can fail again under load.
- `dot-runner` skipped all 54 tests in the full run because `cargo` was
not on my `PATH`. It then passed 54 of 54 with the toolchain on the
`PATH`.
- My first `lib` value removed the default DOM types, and the error
count went up by 375. I added the DOM libraries.
- My Daytona stub with bare `any` left 48 errors, because generic
helpers then inferred `unknown`. I added shapes that return promises.
- One fix to `agent-secrets-routes` used `class_3_static_lease`, and the
test failed. That class allows only specific paths. I reverted to
`unclassified`.
- `ci / General tests (server (8/12))` failed on `259737b700` because of
the flaky port in `idle-completed-history`. paperclipai/paperclip#18853
fixed this test on master, so this pull request keeps that fix.

**Audit of the diff.** I counted these patterns in added and removed
lines, to find fixes that only hide an error:

| Pattern | Added | Removed |
| --- | --- | --- |
| `@ts-ignore` | 0 | 0 |
| `@ts-expect-error` | 0 | 1 |
| `.skip` | 0 | 0 |
| `as any` | 9 | 9 |
| `: any` | 11 | 2 |
| `as never` | 8 | 2 |
| `as unknown as` | 35 | 4 |
| `Promise<unknown>` | 16 | 5 |
| `expect(` | 128 | 113 |

No hunk removes more `expect(` calls than it adds. Three removals of
test content were intentional, and I checked each one:

- `muse-control` removed `signalToken`. Line 1012 already asserts that
it is absent.
- `merged-pr-confirmation-sweep` removed legacy aliases. Other tests
cover them.
- `heartbeat-workspace-branch-containment` removed a `finalize` branch
that no test row can reach.

Some partial fixtures stay unchecked through `as unknown as` and
`Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes`
and `issue-comment-reopen-routes`.

**Accuracy of the check.** One program over the tree and the three
shards reported the same errors at the same positions. Memory, as the
peak of the largest process on one thread, was 3.93 GiB for one program
and 2.43 GiB for the largest shard. I measured these two values before
the fixes.

Not run or not proven:

- I did not run the full `pnpm test:run`, or a root `pnpm build`. I
expect CI to run the full suites.
- I did not run the 4 GiB container check, or the full
`scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so
they are not a Linux cgroup measurement.
- I did not run the UI typecheck after the merges.
- `workspace-runtime-https-live-exercise` is skipped, so its `cleanup`
fix did not run.

## Risks

- **A large diff in test files.** 328 files change. A pull request that
edits the same tests can get merge conflicts. New tests on master can
also add type errors before this merges. Any such error fails this check
and needs a fix here.
- **A higher cost for changes to source types.** A change to a type can
add errors in test files that the author did not touch. The author must
then fix them. This is the purpose of the check, but it is new work.
- **Changed test behavior.** Some tests now run paths that they did not
run before, such as the 9 chat wizard fixtures and the 3 span tests.
They pass here, but they can be more sensitive to timing or to the
environment.
- **The Daytona stub can go stale.** The stub is written by hand. If the
plugin uses a new SDK member, the typecheck reports an error until
someone updates the stub.
- **The release path.** `release-verify.yml` and `scripts/release.sh`
run `pnpm -r typecheck`, so they now run this check.
- **The tsconfig change.** The test transform reads
`server/src/__tests__/tsconfig.json`. The options that I changed are for
type checks, and the runs above passed.
- **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and
in most sandbox provider plugins exclude `*.test.ts`. This check does
not read most of those test files, so they can still drift. That work is
separate from this pull request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with
reasoning, tool use, code execution, and subagents. The runtime does not
show the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (7 tests fail locally, with
and without this change. See Verification.)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:27:38 -08:00
DottaandPaperclip 0b87308d5d fix(native): defer warm continuation during predecessor finalization (#18811)
## Thinking Path

> - Paperclip manages agent work and preserves its saved results.
> - Native warm sessions can continue on a retained remote Runner.
> - A completed predecessor can still have a live finalization lease.
> - The continuation must wait for that owner before it can start work.
> - Two admission checks treated that temporary lease as ambiguous
state.
> - This pull request uses the existing deadline-based wait for both
checks.
> - The same business run can continue once, with its saved context and
gates.

## Linked Issues or Issue Description

**What happened?**

A compatible remote warm continuation can fail before provider work
starts. Its predecessor has completed, but a finalizer still holds a
live lease. The warm-session check reports
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`. The earlier retained-sandbox
check has the same problem on the command transport.

**Expected behavior**

Keep the unstarted continuation on its existing run. Wait until the
recorded owner deadline. Recheck ordinary ownership, Stop, credentials
and checkpoint evidence before execution.

**Steps to reproduce**

1. Complete a remote warm turn and retain its controller records and
provider session.
2. Give that completed predecessor a live finalization lease.
3. Admit a compatible follow-up with full local controller projections.
4. Before this change, admission fails permanently. It must instead
persist a bounded pre-provider wait.

**Paperclip version or commit**

The PostgreSQL regression failed before production edits on
`adb49a4434`. The current head is `5762dfbe4e`, based on master
`e491b23eb5`.

**Deployment mode**

Native remote sandbox execution. No database migration or provider
artifact change is required.

Related: #18718 refreshes stale warm-controller projections. #18343 adds
controller replacement recovery. #18554 restores suspended remote
sessions. This change handles a live predecessor finalizer after those
source checks.

The shared historical error does not identify its failed predicate. This
regression proves the finalizer race; it does not prove that every
historical instance of that error had this cause.

## What Changed

- Defer a compatible warm continuation while its completed predecessor
has a live finalization lease.
- Use the same wait in the preceding retained-sandbox read check.
- Preserve the current run, provider attempt count and checkpoint. Reuse
the existing resumption scheduler.
- Add PostgreSQL cases for both finalization phases, lease expiry,
company scope, incompatible configuration, unfinished commands, Stop and
duplicate resumption claims.
- Add an opt-in three-turn Linux Runner case. Update the
child-controller fixture to own and register a local API listener.
- Document the waiting behavior and acceptance case.
- Repair two readiness barriers in the refresh-worker integration
fixture after a required CI shard failed twice: wait for paginated
startup and for active worker promises to settle before checking idle
SQL.

## Verification

- **Native regression RED:** before production edits on `adb49a4434`,
the PostgreSQL boundary case received the permanent identity error
instead of `NativeControllerRecoveryWaitingError`.
- **Regression GREEN on the current head:** all **71** focused recovery,
settlement, heartbeat safety and refresh-worker cases pass on
`5762dfbe4e` with Node 24.21.0 and PostgreSQL 18. They cover
incompatible identities, company/configuration mismatches, expired or
missing deadlines, unfinished commands, Stop, retained files and
duplicate resumption claims.
- **Real process acceptance on the current head:** the three-turn
`finalizer-wait` case passes in 58.01 seconds. PostgreSQL, TLS, a
rebuilt Linux Runner and the deterministic Codex app-server fixture
confirm zero attempts and no provider turn during the wait, one claim
resuming the same run, retained history, a healthy third warm turn and
final suspension. The sibling `warm-suspended` case from master also
passes, in a fresh container, in 46.02 seconds. Both runs use
`5762dfbe4e`; the artifacts were rebuilt from that checkout with the
repository's pinned Rust 1.97.1 toolchain.
- Acceptance used Node 24.21.0 and glibc 2.41. Earlier probes exposed
the missing fixture API listener and the preceding retained-sandbox
guard; both are corrected. A stale disposable listener and an
incompatible libc image failed setup before the successful
fresh-container run. These setup failures are not the original
regression's RED.
- Repository typecheck/build passed locally before the rebase. Server
typecheck passes after the rebase. **Full CI passes on the current
head:** 48 successful jobs and one expected scope skip, including
typecheck, build, Runner checks, all serialized-server shards, general
tests, E2E and the final aggregate. Module-boundary and whitespace
checks also pass.
- Broad local inventories are incomplete: six chat shards passed (1,834
tests), but untouched server/UI/adapter cases hit timeouts. The exact
shared-base expiration case passes alone; a full-file rerun passed all
89 assertions but timed out in teardown. A Node 24 rerun timed out in
another lifecycle case and teardown. A later PostgreSQL recheck could
not allocate shared memory; this task's redundant broad controllers were
stopped and their logs retained.
- CI attempt 1 failed a Cursor Cloud cancellation deadline, an E2E
cleanup cancellation with a PostgreSQL lock timeout, and the
refresh-worker startup settling limit. One failed-only retry passed
Runner and E2E; the same refresh startup case failed again. Two
aggregate jobs had previously stalled in the hosted-runner queue and
blocked the retry; that obsolete attempt was cancelled. No repeated
blind rerun is being used.
- **Readiness repair RED/GREEN:** real PostgreSQL 18 behind a loopback
proxy with 20 ms delay per direction reproduces the exact startup error
before the test change; the same probe passes afterward. An initial
controlled probe hit the dedicated-database naming guard before tests
ran and is excluded from RED evidence. The full-file check then exposed
a separate false-idle barrier: a worker could be running while its
scheduler had no timer. The fixture now also checks the existing
active-work tracker. All **24 cases pass** with both repairs. The
at-most-six scoped SQL reads and zero recurring idle-SQL assertions are
unchanged; startup alone has an explicit bounded condition wait. No
production behavior or timeout changes.
- Greptile and Paperclip Review are both **5/5 on the current head**,
with no open findings. Both reviews used source inspection; the runtime
evidence above was obtained separately. The previous review workspace
was recovered on the first production attempt after repairing its
partial-clone export. Its preserved result and real follow-up both
finalized successfully, saved snapshots and released their writers and
finalization leases. Three current-head CI shards received hosted-runner
shutdown signals and were cancelled. All three passed their one
justified retry. The queued aggregate had to be cancelled to schedule
that retry; the final aggregate also passed. [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41706617909).
- This acceptance does not qualify a live Codex model or Daytona API run
of the native repair. No deployment or recovery of the original task is
claimed.

## Risks

- A wait must not grant a second executor. The current attempt stays at
zero until the existing scheduler claims it. Expired or missing
deadlines retain their existing hold.
- Saved state must survive the wait. Admission still requires the same
company, task, session, configuration and physical sandbox evidence.
- Current Stop, budget, credential and permission gates remain with
ordinary resumption. No polling loop is added.

## Model Used

OpenAI Codex, GPT-6 family. The runtime does not expose the exact
serving model ID or context-window size. Capabilities: reasoning,
repository inspection, code editing, shell tools and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 14:18:18 -06:00
DottaandPaperclip 1a6017d646 fix(server): resume suspended sessions after controller replacement (#18554)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web server replacement must preserve a completed conversation as
well as active execution.
> - A warm remote Runner can suspend before the replacement admits
another turn.
> - The replacement restores its controller checkpoint from PostgreSQL,
but lacks the Runner suspension receipt.
> - It therefore tries to adopt a live process and rejects the safely
suspended source.
> - This change restores the verified receipt and preserves retained
launch files while the old authority is reconciled.
> - Later turns can keep the same provider conversation after the old
web server filesystem is deleted.

## Linked Issues or Issue Description

Refs #18552, #18543, #18366.

**What happened?**

The real rolling-deployment journey passed both HTTP handoffs, with 765
reads and 78 verified saves. The first three agent turns kept their
native and provider sessions. After the second replacement, the fourth
continuation failed with `remote_warm_session_not_idle`. The remote
Runner was suspended with an empty outbox and no pending terminal. A
focused regression reproduced that exact failure. After restoring its
suspension receipt, the same test exposed a stale launch-file read
during source recovery.

**Expected behavior**

A replacement can resume the same provider conversation from a settled
suspended Runner. It must validate the physical sandbox and all
persisted identity fields. A live source must still pass the existing
adoption checks.

**Steps to reproduce**

Enable the Linux Runner fixture for
`native-runner-restart-recovery.integration.test.ts`. Run the
`diskless-warm-suspended` cases. The fixture completes and gracefully
suspends each source, deletes the controller projections, and admits a
later turn from another process. It does not write a synthetic
suspension receipt.

## What Changed

- Reuse the existing per-turn suspended-receipt reader before warm
admission chooses live adoption or suspended recovery.
- Keep the receipt read retryable when a failed remote read leaves only
the controller projection.
- Leave every non-suspended warm state to the existing live-adoption
checks. A preliminary backpressure observation cannot reject admission
before that owner inspects the current Runner state.
- Preserve current sandbox binding, settled-outbox, exact-identity,
local-evidence and admission checks.
- Pass the structured launch specification to remote asset preparation.
Source reconciliation keeps the old authority's retained launch files; a
current-authority launch still stages its current files.
- Add two Linux Runner/PostgreSQL regressions for two controller-cache
deletions and three turns on one provider thread, including an
interrupted receipt read.
- Update the deployment qualification record, including the failed live
evidence and the successful staging database-idle observation.

## Verification

- Before the fix, the focused regression fails with the exact live
`remote_warm_session_not_idle` error.
- Both new regressions pass with real PostgreSQL and Linux Runner. The
provider is deterministic; this is not a live model qualification.
- The first receipt-only fix failed on the deleted controller skills
cache. The completed fix passes that same scenario without recreating
fixture launch assets.
- All 1,067 surrounding launch supervision, warm recovery, launch-asset
and controller recovery tests pass. The exact `cc23019691` server build
passes. All ten Linux continuity cases pass, including live adoption,
graceful suspension, per-turn resume, changed configuration and
failed-read retry. Each case checks provider-thread identity and exact
turn counts. The live Codex/Daytona journey passes on the same clean
source and exact build: 518 reads, 53 saves and 53 verified saves with
zero probe failures; both predecessor homes deleted; the active
execution completes once; all four turns preserve the provider
conversation and native receipts; saved files and the final command
output are independently checked. No automatic retries. Child credential
and object-storage bucket cleanup pass. Browser reconnection remains a
separate hosted acceptance check.
- The review refinement has a failing-before regression: only the
preliminary read reports transient backpressure; the subsequent adoption
reads the real Runner state. It and both suspension cases pass on the
final source (three cases), with server typecheck. The existing idle
check remains unchanged. An initial retry hit a fixture port left by the
intentional pre-fix failure; a fresh container resolved that
infrastructure error.
- Independent cleanup verification found two archived setup sandboxes
left by the live harness. Both were deleted and confirmed absent. The
failed automatic-cleanup check remains recorded.
- No claim of the ten-deploy Railway campaign or production canary yet.

## Risks

- A cold warm-session admission can now make one bounded remote state
read when local lifecycle evidence is absent. Its failure stops
admission and remains retryable.
- This path cannot replace contradictory local evidence, transfer
authority, or treat a suspended process as a live one.
- Source recovery must keep the previous authority's launch files until
reconciliation completes. The launch identity selects that behavior;
ordinary launches still prepare current assets.
- No schema migration or public API change.

## Model Used

OpenAI GPT-6 Codex, with reasoning, repository editing, shell execution
and integration-test tools. The exact runtime model variant and
context-window size are not exposed in this session.

## Checklist


- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

CI qualification note: first final-head run `41322376054` passed the
native/runtime and remaining suites but failed the unrelated
`automatic_message_on_reactivation` browser fixture at its
pre-reactivation deferred-queue count (0 rather than 1). The unchanged
test passed locally against isolated PostgreSQL (1 test, 43.6 s);
inspection found an asynchronous company-pause/fixture-seeding timing
window. Original CI logs and local evidence are retained. The single
failed-job retry passed; all final-head CI gates are green. Both reviews
are 5/5 and no review threads remain unresolved. The first local
embedded-Postgres launch failed before tests during cluster
initialization; the isolated external-Postgres test passed, and its
disposable database was removed after correcting the private launcher
cleanup call.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 10:08:51 -06:00
DottaandPaperclip 492d537340 fix(native): recover controllers across webserver replacement (#18343)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native execution runs in a remote Runner while the webserver owns
its controller.
> - Encrypted PRP checkpoints and session snapshots already survive a
webserver restart in Postgres.
> - Ordinary warm and per-turn admission still needs local markers and
supervisor metadata to find that state.
> - Replacing the webserver with an empty native controller directory
can therefore lose a usable retained session.
> - This change adds bounded recovery metadata to the existing snapshot
transaction and reconstructs local projections from the selected durable
source.
> - The benefit is native execution continuity across webserver
replacement, with the existing ownership and agent lifecycle policies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Native remote controller recovery and retained-session continuation
during a webserver upgrade.

**Subsystem affected**

server/ — native runtime, checkpoint persistence, recovery scheduling
and internal upgrade reporting. Product E2E qualification also changes.

**Current behavior**

Without both the local PRP marker and supervisor file, normal
continuation can return before discovering the saved Postgres
checkpoint. The snapshot already exists in Postgres, but its local reuse
compatibility envelope cannot be reconstructed.

**Proposed behavior**

Select the existing durable source, validate its identity and ownership,
restore its PRP state and supervisor projection, then use normal
authenticated admission. Missing or inconsistent evidence holds
recovery. It does not authorize fresh provider work.

**Reason and benefit**

A replacement webserver must recover active execution and later continue
its retained session without the old server's native files. This is one
component of deployment continuity. HTTP traffic switching, workspace
storage migration and Railway rollout remain separate.

**Breaking changes**

No new public endpoint, schema migration or database pool. The private
run-profile envelope is additive. Disk-independent rollback also
requires a reader for the staged controller format and compact
checkpoint part; the first production release containing these readers
is the floor. Older file-backed sessions need an owner checkpoint or the
bounded internal importer.

**Additional context**

Related foundations: #17401, #17916, #17254, #14621, #14622. The
implementation contract and open acceptance gates are in [the dated
plan](doc/plans/2026-10-30-native-controller-checkpoints.md).

## What Changed

- Persist a versioned recovery envelope, at most 4 KiB, beside the
redacted session snapshot in its existing transaction. Keep it out of
public run projections.
- Compare frozen admission separately from prepared attachment
instructions during staged recovery, preserving exact
template/identity/authority checks.
- Bound the existing lazy checkpoint pool at three connections after
measured two-slot burst backlog; keep request/renewal capacities and SQL
operation counts unchanged. The tuning is a separate commit.
- Discover and restore the exact selected retained source without local
native markers. Preserve the 32-candidate window, progress barriers and
current authority checks.
- Fence snapshot and result writes. Acquire coordinator, issue and run
locks in order and check expiry after lock waits. Combine snapshot
ownership/read work into one statement.
- Reuse existing recovery deadlines and drain/resume scheduling.
Retained-owner waits preserve the original unstarted run and actual
lease deadline. Stop, ownership and provider-progress fences precede
deferral. Add bounded native evidence reporting, retained-source import
and retention protection.
- Admit a settled warm source with an authenticated idle goal-read
suffix while retaining active-work and mixed-authority holds.
- Permit lifecycle `turn.stop` checkpoints after accepted results under
existing ownership fences. Claim the existing successor coordinator
before retiring the old warm session, and stage shared attachments only
after verified retirement. Preserve omitted isolated-home defaults and
existing fresh skill-selection order during recovery without hiding real
configuration changes.
- Carry the exact durable six-field PRP binding during active recovery
after a healthy warm handoff. Keep the Runner mismatch guard and
preserve opaque attachment IDs.
- Preserve an already verified active or idle warm process during remote
preparation; revalidate its proof/liveness and retain later
authenticated PRP attachment. Ordinary resumed leases retain suspension
checks.
- Drain retained source events under their original authority, reconcile
immutable source accounting and block unknown source tool dispatch
before target work.
- Persist compact staged handoff intent through authenticated adoption
and recover direct/listener command-loss windows without another Runner
or duplicate attachment.
- Rebuild suspended lifecycle receipts for resumed prepared tasks, and
retry missing lifecycle reads after partial PRP materialization without
overwriting contradictory evidence.
- Add normal warm/per-turn diskless continuation coverage, real Postgres
races and SQL-cost checks. Add a reproducible paired load workload.
- Give only the explicit outage fixture a 15-minute provider idle
window, with a red/green catalog regression preserving all production
timing and lifecycle policies.
- Publish the goal-less task-session pointer in the existing
post-acceptance terminal checkpoint transaction, with current ownership,
accepted-result, reset and newer-progress fences. Keep a private bound
receipt so late completion cannot duplicate or resurrect publication.
- Use a retained warm sandbox in the dedicated continuity fixture, and
accept an exact output token followed by an explanation. Keep the strict
identity oracle.
- Extend the explicit Daytona cell to a five-minute active outage and
another replacement after completion. Require retained provider
identity, a new tool receipt and a useful follow-up result. Add an owned
loopback Postgres E2E fixture for hosts that cannot start embedded
Postgres.

## Verification

**Merge recommendation — `fc68584ce6231a02e47ecd743f55b036ccea034f`:
ready for the qualified native-controller component.** Full CI, fresh
review, local application paths, controlled performance qualification
and the strict live Linux/Daytona case pass. Independent resource
cleanup is complete. Broader rollout, browser reconnection and the
unexecuted transport/platform combinations remain integration gates.
Integrated 484 passed full local recursive typecheck/build and ordinary
active→completion→cache-loss→C component qualification. Fresh review
then found a valid attached-wake failure: preparation appends attachment
instructions before the verifier compares execution to the frozen
admission. 9f4 fixes that by separately checking the pre-preparation
admission and exact prepared execution. Both hashes are captured
privately per admission; the full template, identity, configuration and
authority checks remain. Its regression fails before the fix; 36 focused
cases, 42 staged source/target cases and server typecheck pass
afterward. The review thread is resolved. Exact-head full CI passes
([run
41168228822](https://github.com/paperclipai/paperclip/actions/runs/41168228822):
48 successful jobs, one intended skip); the complete `pnpm test:run`
partitions are forced, with the five native suites in the Runner lane.
This is not a claim that an unsharded local command passed. The fresh
exact-head Greptile check succeeds with 5/5 and all six threads
resolved.

The original two-connection database comparison closes **steady PASS and
burst FAIL**. Every relative CPU/WAL/request-p99 gate passes and all
operations/ownership remain correct, but candidate arm 3 drains in
1,015.99 ms and baseline arm 4 in 1,059.67 ms against the unchanged
1,000 ms absolute gate. Those results remain failed. A separate 2→3→4→2
diagnostic reproduces drain times 1,051.57 / 956.79 / 882.02 / 1,065.72
ms: three is the smallest tested passing cap, and the repeated two-cap
failure survives run-order/cache effects.

Separately reviewable `fc68584ce6` changes the existing lazy checkpoint
pool default to three connections. Request capacity stays ten and
liveness two; idle release, shutdown fencing, transaction counts and
pool count are unchanged. Thirteen DB lifecycle/cap tests and DB
typecheck pass. Full recursive typecheck/build pass at exact fc68584,
with the matching server build stamp. The final paired native
feature-cost comparison explicitly uses the **same cap-three policy and
common current DB factory in both variants**; its baseline is
controlled, not the unmodified historical cap-two configuration.
Original gates and workload counts remain unchanged. The one strict
Linux/Daytona case started at 04:40:56 UTC only after both new profiles
passed; it passed on the first attempt with all 49 workflow checks.
Independent cleanup confirms all three owned sandboxes absent, owned
Postgres/controllers/observer gone, VM terminated, root volume absent
and ephemeral SSH key/agent removed.

**Final controlled cap-three qualification: PASS.** Baseline native
implementation `59da15066a` and candidate `fc68584ce6` use the same
current DB factory and checkpoint policy (request/liveness/checkpoint =
10/2/3). Eight 65-second ABBA arms cover 300 controllers and 10,000
historical tasks. All correctness, absolute and 10% relative gates pass;
original cap-two failures remain unchanged.

| Mode | DB CPU µs/op, baseline → candidate | WAL B/op, baseline →
candidate | DB-request p99 ms, baseline → candidate | Maximum drain |
| --- | --- | --- | --- | --- |
| Steady | 542.40 → 554.83 (+2.29%) | 1509.01 → 1558.27 (+3.26%) |
107.11 → 111.90 (+4.48%) | 26.35 ms |
| Burst | 1036.36 → 1063.11 (+2.58%) | 4153.12 → 4317.64 (+3.96%) |
103.736 → 103.841 (+0.10%) | 930.14 ms |

Every matched arm preserves operation/statement/transaction counts,
including 3,900 PRP and 3,900 snapshot transactions. All three actual
pools are included; there are zero lost owners, missed lease deadlines,
dropped operations or errors. Peak Postgres CPU is 18.31%/9.47% of two
cores and memory 614.29/684.42 MiB (steady/burst). The benchmark uses
one Node process simulating 300 controllers; it does not qualify
two-webserver overlap, HTTP latency or hosted capacity. Its observer SQL
contributes to measured database CPU. The original unmodified cap-two
baseline is separately retained as failed; this cap-three comparison is
explicitly controlled.

The exact final-source 8/512/8,192-chunk test also passes: zero recovery
SQL, transactions or changed rows in each text-only interval, zero idle
SQL, seven lifecycle transactions and twelve changed rows per sample.
Existing lease traffic remains separate. The final ordinary managed
application replay passes active recovery, completion, native-cache
removal and C continuation; its UNKNOWN local debug-daemon provenance
and partial instruction-observer limits remain recorded. The live case
uses the attested Linux producer instead.


**Live qualification: PASS, one attempt, no retries.** Campaign
`native-controller-retained-fix-fc68584ce6-ec2-20261031T044055Z` started
04:40:56 UTC on the exact final Linux webserver source; suite definition
hash `12f9bb97823c1537e43f0398e7cddf39890f77bce7b413ca4dfd944f4a30e5d5`.
The existing immutable image and attested Bookworm Runner artifact are
unchanged.

- A was SIGKILLed and its native cache discarded. B started after
300,149 ms, naturally acquired generation 2, authenticated at 316,624 ms
and resumed at 317,682 ms within the original 420,000 ms grace. The same
active run, Runner, provider PID/start fingerprint and one
submitted/started/accepted turn were preserved; one final
result/terminal and workspace finalization completed, with no
retry/recovery action.
- After active and queued completion, B was SIGKILLed and its native
cache discarded again. C received exactly one ordinary task-comment API
follow-up, completed a new tool and produced a useful reply while
preserving native/provider/Runner identities. This closes the earlier
failed post-C gate. A separate C OS-process PID field was not exported;
the physical-PID proof is specifically A→B.
- Four distinct results were accepted and published; all four
finalizations committed at attempt 1, all four accounting-v2 projections
settled, and the task-session pointer advanced to C. The coordinator
independently checked these facts and all 49 canonical checks against
the sanitized receipts.
- Canonical cleanup passed and independent checks confirmed three exact
sandbox 404s plus complete inventory/zero owned remnants, no owned PG or
controller/observer processes, EC2 terminated, root volume absent, and
the ephemeral SSH key/agent removed by 04:54:25 UTC.

Live scope is classic remote Codex warm continuity with non-native
storage dependencies available. Prepared-task completed B→C, full
per-turn application disk loss, browser reconnection, two-server load
and overall Railway/HTTP continuity were not live-qualified here.
Earlier failed campaigns and the original cap-two burst failure remain
failed historical evidence. A report-renderer cache setup failure was
preserved; the normal sanitizer subsequently succeeded with an explicit
cache path and a new output directory, without changing validators.


The repair keeps retained events under their original six-field
authority until persistence and accounting accept them. Unknown semantic
effects stay unresolved. A compact staged target survives source
bootstrap, cached-command delivery and authenticated handoff, including
an uncertain commit response. Direct and same-guest-listener recovery
require the verified original Runner and the exact endpoint. Target
snapshot publication marks readiness in memory, with retirement
coalesced into an existing command write. Headers contain only the
staged descriptor/digest; a prequeue template is written once as a
changed checkpoint part and then recovered from the existing
command/receipt. No second session snapshot transaction, streaming SQL
path, new pool or recurring idle scan is added.

| Current repair gate | Evidence and scope |
| --- | --- |
| Source-tail regression | Real Runner protocol RED reproduced
attachment rejection. Frozen core/transport pass five source-tail cases:
source usage attribution, held persistence/no premature ACK, lost commit
response plus cache loss, unknown semantic hold and accounting rejection
hold. |
| Handoff loss windows | Direct and listener modes pass
queued/prepared/activated/completed boundaries (eight cells), preserving
the actual Runner/provider process. Missing adoption proof and changed
endpoints reject recovery. |
| Compact storage | A 31 KiB attachment grant writes its separate part
once, leaves it unchanged across source snapshots and removes it at
queueing. Headers do not duplicate the grant. |
| Accounting | 21 real-Postgres usage tests and 19 source-binding tests
pass. Recovery hydrates immutable source receipts without resetting
their producer or billing classification; conflicting/incomplete usage
holds. |
| Admission and malformed evidence | Driver 16, staged evidence 42,
restart classifier 27 and source usage 19 focused cases pass. Four
malformed-file regressions pass at d6a7, whose executor is
byte-identical at 484, and server typecheck passes. Full executor file:
990 pass; 16 embedded-Postgres `initdb` setup failures before
assertions, retained separately. |
| Required checks | Exact fc68584 recursive typecheck and build pass
locally, with matching server stamp. Forced full CI succeeds (48 jobs
pass, one intended skip); all PR checks are clean. Maps/token gates pass
for their unchanged source surfaces. Fresh exact-head Greptile is 5/5,
all six threads resolved. |
| Ordinary application path | Exact fc68584 managed component passes
active recovery after natural expiry, completion, native-cache loss and
ordinary C continuation. Production bytes match before/after; temporary
fixtures restored and owned DB/services cleaned. UNKNOWN local
debug-daemon/partial instruction-observer limits remain; the separate
live case uses the attested Linux producer. |
| Producer | Locked Bookworm build from exact 5b7 Rust tree
d03e494217, equal to final fc68584 Rust
source. SHA-256
65b8f8cfdd6099e9ad24f1ca154e8a260777269cc82f6e23c6e6a180c531eaaf;
38,515,328 bytes. Actual Debian 12 execution and live admission pass.
Controller/admission/DB changes do not relabel the producer source. |
| Historical cap-two comparison | Exact 59da baseline vs 5b7 candidate,
300 controllers/10,000 history, ABBA 4×65 s per mode, all three actual
pools and real snapshots. Steady CPU/op 541.36→554.64 µs (+2.45%);
WAL/op 1512.13→1560.99 B (+3.23%); DB-request p99 102.78→105.99 ms
(+3.12%): PASS. Burst CPU/op +4.71%, WAL/op +5.29%, p99 −0.10% pass
relative gates, but absolute drains above fail. Identical operation/SQL
counts, zero lost leases/drops/errors. One Node process simulates 300
controllers; this does not qualify two-webserver overlap or HTTP
capacity. |
| Live acceptance | Strict final-source Linux classic Codex case passes
1/1 on attempt 1, all 49 checks. Active recovery within grace and useful
retained C continuation succeed; four
accepted/published/committed/settled outcomes. Independent provider,
PG/process, VM/root and SSH cleanup complete. Prepared/task and broader
rollout limits are explicit above. |

**Historical qualification snapshot through `9eb9ac6`.** The head at
this snapshot was `9eb9ac6f9e312f3ebe8040f5a7565e88ae8824f3`. Server and
Runner package sources are byte-identical to `283e05ffaa`; the two later
commits change the explicitly scoped qualification and its documentation
only. It includes master through the storage-owner fix
`45a3a92e6dc52d20923b9aa34a22fee2ec5443f9` (#18357). At that snapshot,
CI/review passed but the strict post-C native recovery gate failed; the
PR stayed draft and required another native repair. Prior `39a4343` CI
and review passed, but its full live workflow failed; that grade remains
unchanged. The first repaired `435ccd2` live attempt failed before any
injected outage. It exposed a legitimate terminal snapshot before result
acceptance; no accepted result ID or publication receipt was observed.
Its canonical report remains 0/1 `candidate_failure`, with cleanup
passed and independent provider absence confirmed. `0862062` defers
publication until result acceptance and fixes the fresh review's
coordinator/issue lock-order finding. Both failures have deterministic
real-Postgres regressions. The `0862062` live attempt then proved
accepted-result publication but failed active recovery: a resumed-lease
preparation guard demanded suspension after the exact live Runner was
already verified for adoption. `64c0b746` preserves that verified
active/idle-warm process, revalidates its proof/liveness, and keeps
ordinary suspension checks and later PRP authentication. Its live run
passed preparation but exposed a second active-warm boundary: all
durable binding fields matched except opaque turn/item attachment IDs,
which the server had regenerated. `7a203274` carries the complete
verified six-field binding and keeps the strict Runner mismatch guard.
Its cloud case then passed active adoption, completion, publication and
accounting. The queued continuation exposed accepted-result retirement
writes and an implicit-home fingerprint drift; both are fixed with
deterministic regressions in `39a4343` before this fresh full
qualification.

The prior live case exposed a real publication gap. A goal-less native
turn could persist its accepted result and terminal snapshot before
provider cleanup, while the task-session pointer was still empty. Normal
later admission then selected a fresh session. The fix publishes that
pointer and a private bound receipt in the **existing post-acceptance
terminal checkpoint transaction**, before cleanup. Earlier terminal
observations persist without a handled receipt or continuation pointer.
No result-acceptance or snapshot transaction was added. The late
completion path recognizes the receipt. This also prevents a late
callback from restoring a pointer after a user reset. Goal/control
publication keeps its existing owner.

The dedicated Daytona fixture now uses the existing warm/reusable
environment. Its sandbox, Runner and provider storage must survive. The
former generic per-turn fixture disposed the sandbox after completion
and kept provider-history backup bytes under the discarded directory.
Whole per-turn disk-loss acceptance therefore has a separate
provider-backup storage dependency. It remains distinct from the passing
native per-turn component checks with backup storage available. The
60-second lease, 300-second outage and 420-second grace are unchanged.

| Gate | Current repair evidence |
| --- | --- |
| Assembled regression | Baseline fails with an accepted result and
terminal snapshot but a NULL task pointer while real provider cleanup is
held. The fix passes normal continuation and reset cases, with no
pointer seeding. Exact merged `92b7447` passed 2/2 twice, including
after normal package TypeScript output refresh. Production files are
byte-identical at frozen `435ccd2`. |
| Persistence and concurrency | 19 helper/transaction cases and all 59
existing control-port cases pass on `0862062`. They cover bound result
IDs, reset/newer epochs, Stop, lock waits and expiry, receipt replay,
missing stubs, foreign bindings and atomic 4 KiB rollback. Existing
snapshot transaction count stays one. The new tests reproduce the exact
live missing-result-ID error and the inverse replay/status lock order
against `435ccd2`, then pass after the correction. Replayed results
still reject stale generations. |
| Completion | All 32 completion-boundary tests pass on merged
`92b7447`; the source is unchanged at `435ccd2`. Native result-shaped
session metadata is preserved. |
| Qualification fixture | A valid review report reproduced three
rejected correct replies. Exact token-boundary matching fixes them. 140
focused E2E/catalog cases and Product E2E typecheck pass. Wrong tokens,
marker prefixes/suffixes and wrong authors remain rejected;
retained-identity checks are unchanged. |
| Required checks | Full recursive typecheck and standard build passed
at `435ccd2`. Both exact commands also passed on `0862062`, with its
exact server build stamp; both also pass on `7a203274`, with its exact
server build stamp. Fresh `0862062` Greptile passed 5/5, with zero
unresolved threads. The `64c0b746` normal CI then passed all 49 jobs (48
success, one expected skip) and Greptile passed 5/5 with zero unresolved
findings. The `7a203274` normal CI also passed 48 jobs with one expected
skip, with fresh Greptile 5/5 and zero unresolved findings. Current
`39a4343` recursive typecheck and standard build both pass with
persisted exit codes and its exact build stamp; fresh Greptile is 5/5
with zero unresolved findings. Its normal CI completed successfully: 48
jobs passed and one expected job skipped. The fresh live result remains
separately failed. The `0862062` normal CI had 46 passing jobs, one
expected skip, one worker communication failure and its aggregate
failure; GitHub states the self-hosted runner lost communication. No
failed assertion or specific OOM/spot cause was established, and no
blind retry was used. |
| PostgreSQL supplement | One corrected all-pool pair at the preceding
repair passed 300-controller/10,000-history correctness and absolute
gates. SQL 76,590→75,990; transactions 9,013→8,713; UPDATE rows 11,613
each; 300 correct pointer writes each. PG CPU 31.99→30.46 s, WAL
70.36→67.87 MB, request p99 100.98→103.12 ms. This is one pair, not a
repeated improvement claim or an exact `0862062` CPU measurement. Its
two later changes defer premature publication and reorder existing
locks; they add no transaction, stream/idle writer, pool or timer. The
first collapsed-pool attempt remains invalid and retained. This is not
an exact final-head CPU/WAL measurement; later fixes change ordering,
identity selection and in-memory skill replacement without adding
snapshot transactions, pools, stream writes or idle SQL. |
| Live publication proof (`0862062`) | Baseline run
`06fcc806-f186-4190-9b57-087b61d75533` accepted result
`58f912dc-94d9-4b4d-bd3d-d5f689c83fd3`; the publication receipt names
the same result, the task pointer has parameters and `lastRunId`
matches, and the normalized native identity matches the following active
run. Accounting v2 settled. |
| Active recovery (`0862062`) | **FAIL.** SIGKILL/cache discard at
23:07:28.666 UTC; hold 300,000.859 ms; B ready at 308,896.486 ms;
natural generation-2 ownership at 312,906 ms. At about 319 s,
preparation failed `runner_harness_state_mismatch:
resumed_not_suspended`. No authenticated after-receipt passed within 420
s. Exact original Runner PID 668/start/fingerprint was independently
still alive; provider auto-stop was 15 minutes. No retained C follow-up
qualified. Canonical 0/1 `cleanup_failure`/cleanup-failed report remains
failed; subsequent independent cleanup deleted all three owned resources
with exact 404s and confirmed full intended-account inventory with zero
owned resources. |
| Verified adoption preparation (`64c0b746`) | 91 focused cases pass,
including actual preparation for four active providers and a distinct
prior/current warm proof. Unproved resumed state, dead process and stale
warm owner remain rejected. The earlier options-only fixture missed this
gate. No spawn, backup restore or PRP-authentication bypass was added.
Server typecheck passes. |
| Active warm identity (`64c0b746`) | **FAIL.** B claimed generation 2
at 312,995 ms. Preparation passed, then strict PRP recovery rejected the
generated attachment IDs at about 319 s. The saved and desired Runner,
lease, normalized session and run IDs matched; only
`turn_attachment_…`/`item_attachment_…` differed from generated
`turn-<run>`/`item-<run>`. Canonical 0/1 cleanup-failed report remains
failed; independent cleanup confirmed deletion/404s and zero owned
provider resources. |
| Six-field identity repair (`7a203274`) | Actual server-factory
regression RED4→GREEN38. A real Rust Runner test passes healthy first
run → random-ID warm handoff → held active second run → restored
controller. It verifies a new live authenticated connection, exact
identity, same Runner/provider OS identities, no duplicate launch/turn,
and no redispatch or invented receipt for an uncertain tool. The
synthetic wrong pair still fails the unchanged guard. This protocol case
does not claim business completion. Server and Runner typechecks pass. |
| Active recovery/completion (`7a203274`) | **PASS for the active
component.** Full five-minute SIGKILL/cache-loss hold. B ready at
308,936.651 ms; natural generation 2 observed 312,979 ms; authenticated
at 317,720 ms and resumed processing at 318,527 ms. Same run
`1e1ccb00-0728-4935-97da-57562d81cb42`, Runner
`246f73f7-a689-4177-a9c2-7ce711d63aed`, remote PID 669/start
23:48:27.470, provider lease/session and one submitted turn. Accepted
result `49097529-5310-4ac3-af95-5bf9b91d19f4`, publication, finalization
and accounting v2 all settled. |
| Full workflow (`7a203274`) | **FAIL, preserved.** Queued
late-requirement run `5e61f38d-c7d1-4fc1-bcbf-bce9d46ce4eb` failed
`runner did not durably suspend before checkpoint` before outage 2.
Canonical 0/1 candidate failure, cleanup passed, evidence valid. C
retention was not exercised. Independent provider absence,
controllers/PG cleanup, EC2 termination and root-volume absence all
confirmed; closed receipt SHA-256
`676a3b15eb9937b6bf995c36fb8666c88964d509e5ceb9109fb02da0aa2bc909`. |
| Retirement and stable configuration (`39a4343`) | Real Runner +
Postgres RED reproduced the exact suspension failure and nine owner
rejections. `turn.stop` is now lifecycle checkpoint maintenance under
unchanged owner/result/generation fences. GREEN25/25 proves actual
persisted suspension, exact identity and completed stop/drain/suspend;
unaccepted, foreign, stale and new-work negatives remain rejected. Two
default-home fingerprint REDs become GREEN9/9; explicit mode changes
still invalidate. No global matching exemption or historical fingerprint
rewrite. |
| Fresh full Daytona warm cell (`39a4343`) | **FAIL, preserved.**
Launched 00:15:29 UTC with automatic retries disabled. Active adoption
passed: actual offline hold 300,000.848 ms; B ready 309,922.155 ms;
natural generation 2 at 311,821 ms; authenticated 317,989 ms; resumed
330,440 ms. Same run `3e126819-e786-4b3b-b923-c975cf4f5b79`, Runner
`4079138a-8d7c-44a4-8558-07a5411d1f55`, PID 668/start and
provider/session identities. Accepted result
`08e504af-4dec-48a8-bf05-bf0a32280559` and publication passed. Queued
run `31d168c9-7ab9-4700-abb6-fafb90912237` failed the same strict
suspension error before outage 2. Its row selected a new native
identity, but no new third provider process or invocation was observed.
Recovery-time configuration still reset only `adapterConfig`; native
provider, context and workspace subtrees were unchanged. Canonical 0/1
candidate failure, evidence valid, normal cleanup passed; all campaign
provider/PG/process resources independently absent. Closed campaign
receipt SHA-256
`5e7d6281c13c812f3a946667f97f3c7d339b5ad86a3a819bd185b2fa73bc4a19`;
EC2/SSH remain temporarily held for the concrete follow-up repair.
Original 60/300/420-second limits and strict queued/second-replacement
checks remain unchanged. Direct-process path; no new live prepared-task
claim. |
| Native retirement authority (`283e05ffaa`) | Exact `39a4343` facade
regression RED3→GREEN9. The successor now uses its existing production
coordinator claim and prompt renewal before strict retirement. A real
Runner/Postgres case starts in the actual native-observed/unclaimed
phase, claims through production code, then closes and durably suspends
the predecessor in 1.657 s. All 25 real-Postgres cases pass;
unclaimed/foreign writers remain rejected. Shared attachment staging
stays after retirement; Stop and denied claims neither close nor stage.
No new claim or SQL path was added. |
| Configuration continuity (`283e05ffaa`) | Real shared/project-primary
environment selection → configuration with four skill versions
reproduces `reset: true`, changed category exactly `adapterConfig`,
after a Postgres JSONB read. Pinned restoration had reordered the
generated preference while actual runtime skills stayed identical.
Replacing existing pinned rows in place preserves the original fresh
name/key order and fingerprint. The assembled regression passes, as do
three existing pinned-version/rename/alias cases and server typecheck.
Genuine model/skill-selection changes still invalidate. No
fingerprint-format change or broad compatibility exemption. |
| Final production repair (`283e05ffaa`) | Full recursive
typecheck/build and [normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41151145106)
PASS: 48 jobs plus one expected skip. Fresh review 5/5, zero unresolved
threads. Live baseline, recovered run and queued continuation all
succeeded, each accepted/published/committed once with accounting v2
settled. Their native/provider identities and configuration fingerprint
match; both recovery and continuation report no reset, no changed
categories and task-session reuse. Active outage/authentication remains
within original timing. |
| Browser-dependent continuation (`283e05ffaa`, definition 6) | **Whole
cell FAIL, preserved.** After B was killed at 00:46:38.928 and its cache
discarded, C became ready in 9,151.410 ms. `openParent()` then hit a
30-second `page.goto`/DOMContentLoaded timeout before posting the final
request. No fourth native run or C continuation was exercised. Cause of
navigation timeout is unknown. C task had no execution lock, queued
comments, handoff or recovery blocker; all three prior runs were
committed/accounted. Normal cleanup passed; independent provider
absence, controller/PG cleanup, EC2 termination and exact root-volume
absence confirmed. Sanitized report bundle SHA-256
`2a3b841d62efb2ae30ef14f96a431ac4fa7fe3a782f1144831512f78edc65022`. |
| Native API continuation qualification (`9eb9ac6f9e`, definition 7) |
**FAIL, preserved.** The real post-C API request dispatched run
`c71fbb18-f979-4fe4-93ce-9c4be46dbf21`, which failed
`native_session_interrupted` / `native_session_recovery_failed` at
01:10:00.640 UTC. It selected the same normalized native identity,
Runner and checkpoint-provider identity as the three succeeded turns. No
C result was accepted; pending accounting remains explicit. Active
recovery passed: actual offline hold 300,001.098 ms, B ready 308,335.019
ms, natural generation 2 observed 311,818 ms, authenticated 315,540 ms,
resumed 316,612 ms. Queued continuation succeeded/accepted/published. B
was then killed/cache discarded at 01:09:34.500 and C ready in 8,783.803
ms. Original classifier `transient_infrastructure`, whole-case failure
and cleanup passed are unchanged; that classifier does not establish the
root cause. All owned provider/PG/process resources, EC2, root volume
and SSH agent independently absent. Closed receipt SHA-256
`7f21ac4a22efc07b849fa7e17866083b6127b3321af12671b4fc054ce57039a9`;
sanitized bundle
`71ffef14a30a553e008320a31840a6d6c0de85c1bfdb60f783fdb53a6aae812a`.
Exact backend detail is unavailable in approved structured evidence. |
| Current required checks (`9eb9ac6f9e`) | Recursive typecheck, standard
build and exact build stamp PASS. [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41152908417)
PASS: 48 jobs plus one expected skip. Fresh Greptile 5/5, latest edited
summary checked, zero unresolved threads. Exact8/512/8,192 stream probes
again report zero recovery SQL/transactions/rows and settled-idle SQL;
seven control transactions/twelve changed rows each, with 36 separate
legitimate renewals in the longest case. CI success does not supersede
the failed live C gate. |

The completion supplement observed secondary request/liveness/checkpoint
transaction p99 increases of 23.0%/45.7%/15.6% in its single pair.
Renewal p99 stayed below 0.865 s, with 900 successful renewals per arm
and no lost leases. The named pool caps stayed 10+2+2. An aggregate
activity sample peaked at 15 non-observer entries without backend-type
classification, so no claim is made that all were client connections. No
sampled lock episodes does not establish zero lock waits.

Twelve pure reader/selector cases also pass with the new adjacent
receipt. This is not another executed process rollback. Development
reader `5cf7aead` can parse the recovery state, but its old run-detail
redactor can expose receipt IDs/outcome to an authorized run reader. The
first production release containing recovery readers, complete
attachment-identity handling and both private-field redactions is the
rollback floor. The implicit-home fix prevents new recovery-time
fingerprint drift; it does not rewrite previously stored fingerprints
from failed candidates. The development probe is not a published
supported rollback target.

Latest actual `0862062` Runner tests independently passed 8/512/8,192
text chunks with zero recovery SQL, transactions and changed rows, plus
zero settled-idle SQL. All three observed seven setup/control
transactions and twelve changed rows. The 8,192-chunk case separately
recorded 31 legitimate lease-renewal statements, with no lost lease.
These are component SQL observations, not a production CPU/WAL claim.
The new adoption and identity branches affect recovery preparation only.
The `64c0b746`, `7a203274` and exact `39a4343` CI independently passed
all three stream sizes with the same zero-SQL/idle result; each latest
8,192-chunk case recorded 36 separate renewal statements. Exact
`39a4343` Runner 2/2 passed 133 files / 2,317 tests plus two skips; its
separate native group passed 150 tests plus eight skips.

The first API fixture revision omitted the mandatory final screenshot;
this was caught before paid launch and allocated no provider/model work.
The correction captures the actual retained Chromium page with an
explicit unqualified-browser label, without changing the standard report
validator.

Automatic approval review rejected exporting raw private API/workflow
snapshots because of sensitive payloads and insufficient
destination-specific authorization. Sanitized reports and
bound/hash-only receipts are retained; raw snapshots were not copied
through another channel. This does not change the failed grade.

Repair characterization also preserves its failed attempts. An
unconditional staging relocation broke three existing same-scope
scripted cases; the final implementation defers staging only for an
actual superseded warm scope, and all nine order/race cases pass. One
baseline transform overlapped an edit and is explicitly invalid; the
actual immutable `39a4343` loader then produced the three genuine REDs.
The independent task-directory hypothesis produced a RED but was
excluded by the live shared-workspace evidence; no directory policy
change was landed. A shared-workspace fixture authorization omission was
corrected separately from the four-skill semantic RED.

Setup failures are retained separately from semantic outcomes. These
include an old host port after the owned Postgres fixture restarted and
a sandbox-denied TSX IPC pipe. The supported external-Postgres run
passed, and the final exact recursive typecheck passes with normal
process permissions. Seven stale declaration errors after the master
merge cleared after compiling the actual merged SDK/Runner TypeScript
sources. No declarations were edited by hand.

<details>
<summary>Earlier qualification measurements and failed attempts
(historical, before this repair)</summary>

**Historical blocked gate (before the repair below).** Final native
implementation is `abed921e12d543b62f57a56435419c0dafb0bc33`; current
branch head is `af9e2d157dcc349e55374ad186c9be2d6cb96bf9`. The two later
commits change only qualification fixtures, their matcher and
documentation; `server/` and `packages/` are unchanged. It includes
master through `fd008979f5794fda3d5f54151d36acd31d0de140`.

The live Daytona measurement proves five-minute active recovery,
once-only completion and accounting, but **fails retained-session
identity after the second replacement**. The saved checkpoint and
recovery metadata exist. The task-session pointer remains empty, and
ordinary admission creates new native/provider sessions. The exact
skipped publication boundary remains unproven. A useful saved-file
result alone does not satisfy retained-session continuation. This
application-path gap blocks merge independently of CI.

A separate new-master CI failure is owned by the storage workstream (fix
in #18357). No storage guard, session-identity oracle, lease/grace limit
or provider result was relaxed.

| Gate | Observed result and exact scope |
| --- | --- |
| Distinct source recovery and compatible rollback | PASS at A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → B
`5ea45f8d17e33c9a62bd79982dcf0375b07733d0` → fresh B → A. Four
controller processes, natural 60-second lease expiry, absent native
projections before replacements, three succeeded/committed unique
results, one provider thread, unchanged Runner identity, and strict idle
cleanup. 177.782 seconds. |
| Immutable installed native component | PASS at installed A
`5cf7aead09a2d9c7e2f2929d761f535ef3c1d798` → installed B
`eaef0d56552c087aaa4fa68a8655282ffbd12667` → fresh B → A. Each
controller imports its contained compiled server and DB. Three unique
committed results, stable Runner/thread, no extra provider restart on
same-contract rollback, strict cleanup. 106.672 seconds. This is
compiled component proof, separate from full CLI-native application
acceptance. |
| Normal public CLI startup | PASS for B
`eaef0d56552c087aaa4fa68a8655282ffbd12667`, Node 24.22.1, darwin-arm64:
fresh external DB, 415 migrations, health ready, root HTML 200, company
POST 201 plus GET readback. The installed tree remained unchanged after
execution. |
| Five-minute component outage | Actual controller absence 300,168 ms.
Natural generation-2 claim at 407.196 seconds after crash, same provider
attempt and Runner, original run succeeded/committed. The larger
540-second case timed out before its retained follow-ups, so it is an
active-adoption pass, not a complete four-controller pass. The complete
short source/installed cases above independently cover follow-up and
rollback. |
| Drain/resume review regression | Fixed in
`226f03e736d1ee99fea1762ab32ea3a8a9241c20`. Five facade-level cases
pass: overdue normal/restart waits, future timer cancellation, another
facade resuming the original owner, and transient retry. Against exact
prior scheduler `5ea45f8d17e33c9a62bd79982dcf0375b07733d0`, four fail
and one passes. The later head changes documentation only. |
| Final corrected-fixture Daytona | **PARTIAL: active recovery PASS;
retained identity FAIL**, exact live source `0b6afc499f`. A was
SIGKILLed and native cache discarded; actual offline hold 300,000.375
ms. B was ready at 308,326.961 ms, naturally claimed generation 2 at
312,059 ms, authenticated at 315,996 ms and resumed processing at
316,519 ms, within the original 420-second boundary. The same run,
Runner PID/start, session, lease and event prefix survived; one turn
submitted. Original execution succeeded and settled accounting. After
completion, B was killed and its native cache discarded; C was ready in
9,077.779 ms. C executed the saved project and returned correct
`queue_r`, with all four runs succeeded/accounted. However, native and
true provider-session identities changed: the unchanged retained oracle
fails. No whole-workflow pass is claimed. |
| Prior matched-producer Daytona | FAILED at exact `abed921e12`. The
matching current-source Runner passed normal admission, completed the
first task and settled its accounting. During the active revision, A was
SIGKILLed and its native cache discarded. Actual offline hold was
300,000.727 ms; B was ready at 308,304 ms and naturally claimed
generation 2 at 309,651 ms. At about 318 seconds, the remote process
check reported `process_not_running` for the original admitted Runner.
No authenticated after-receipt or retained follow-up passed. The current
failure is preserved; no automatic retry, lease/grace relaxation, debt
edit or provider replay is used. Bounded forensic read found preserved
exact-bound state with no recoverable failure, a transport-reset
diagnostic and no terminal delivery; this supports external process loss
without proving the exact cause. The sandbox had a five-minute auto-stop
policy, conflicting with the required surviving-sandbox assumption.
Owned cleanup precedes the separate corrected-fixture case. |

**Live retained-session boundary.** Prior completed run `3806…` and C
follow-up `8adf1df3-6744-40bf-b35d-100451f29d9f` both have database
session snapshots and `nativeControllerRecovery`. Their tool
fingerprint, provider configuration, workspace descriptor,
runtime-context digest, driver and work mode match. There is no recorded
force-fresh/workspace-selection change, and this is an ordinary task
rather than a conversation-generation reset. The same issue/agent
task-session row has `session_params_json: null` and `last_run_id: null`
from baseline time. Both native normalized UUID and actual provider
session/thread hashes differ after C. Full application admission
therefore did not select the retained source. No fallback to file
execution or relaxed identity comparison is accepted as proof of
continuity.

**Independent cleanup completed.** Full current-account provider
inventory confirms no resources owned by either final fixture, with
exact-ID absence checks. Owned controllers and campaign Postgres are
stopped/removed. EC2 `i-01551c09af098dde5` is terminated, its
delete-on-termination root volume returned `InvalidVolume.NotFound` at
22:14:08 UTC, and the temporary SSH key/agent were removed. The
canonical failed report and immutable oracle input are retained. Closed
qualification receipt SHA-256:
`b4b5fefce287ebfdbf2cc2238efaad62d8ad485aed0e9c0628ae95dbc18a8575`.

**Original measurement and grader repair remain separate.** The C reply
contains the requested marker followed by a newline and the correct
output. The old workflow waited for one literal space, so its normal
receipt collection stalled; that original attempt was closed and its
canonical `cleanup_failure`/interrupted result is preserved. Commit
`af9e2d157d` repairs only this reply predicate, with 144 focused cases,
E2E typecheck, discovery and diff checks passing. Ordinary string
matching stays literal. The corrected predicate accepts the saved reply,
but the unchanged retained-identity oracle still fails. Pure
active-recovery grading of the captured measurement passes 7/7; this is
not a rewritten passing campaign.

The sandbox idle-window conflict was corrected only for this explicit
fixture (15 minutes). Actual baseline and later sandbox metadata confirm
that policy; the original active box was disposed before a post-gate
policy read, so its 15-minute setting is established by the normal
create/environment configuration rather than an after-read of that box.
Production native lifecycle and the 60/300/420-second constants remain
unchanged.

The development compatibility reader A is a genuine backport commit
based on `59ff89a18fe1312b5f0bb908439bbdb1237b0252`, with the required
idle-goal reader fixes. It is not a previously published release. Its
older executor differs from B. The rollback floor must include this
reader capability. The source component cases admitted the Linux ARM64
Runner packaged at `59ff89a18fe1312b5f0bb908439bbdb1237b0252`; the
installed component case admitted the `5cf7aead` package. Their Runner
bytes are identical: SHA-256
`b1f04caa1ce04e720dec7eb62ede170eb3621948c0849b8f305ea5c5116bced8`. Each
admitted artifact remained unchanged throughout its case.

[Final public package
build](https://github.com/paperclipai/paperclip/actions/runs/41121460635)
and [reader-floor
build](https://github.com/paperclipai/paperclip/actions/runs/41122170506)
both passed all four daemon builds, package assembly and both Darwin
clean consumers. Both Linux ARM consumers failed before provider work
with the same existing local Codex permission-profile error: bubblewrap
attempts `activity.fifo/.git` and reports “Not a directory.” This
matched baseline failure remains open; it is not a native remote
continuity pass. Full post-run tree and build-stamp revalidation also
passed after the compiled native case. The installed component trees are
A `c29114c1bf71c5af00f761aeb9e3d84ed3d3075b9f5600dee3ef2543eb2b6ec5` and
B `eda575505373a3bdc630fa4919862b359791fe3a7ef637961a25b73b38efe4b0`.

**Controlled PostgreSQL qualification passed.**
Baseline/candidate/candidate/baseline workloads used 300 controllers,
10,000 historical tasks, real snapshot/encrypted PRP writes and
unchanged pools. PostgreSQL and client were each limited to two CPUs and
1536 MiB on an isolated eight-vCPU Linux host.

| Workload | PG CPU per operation | WAL per operation | Database request
p99 |
| --- | ---: | ---: | ---: |
| Steady | +1.18% | +1.81% | +1.91% |
| Aligned burst | +3.40% | +2.56% | +3.81% |
| Two logical servers | +3.11% | +2.56% | +9.68% |

All arms passed absolute/correctness gates with unchanged control
transaction counts, no dropped operations and no owner loss. Overlap p99
has little margin below the relative threshold. It uses two clients in
one Node process, not two HTTP servers. [Closed measurements and
limitations](doc/plans/2026-10-30-native-controller-qualification-evidence.json)
and exact-source reproduction patches are retained. [Earlier
contended-host
failures](doc/plans/2026-10-30-native-controller-load-evidence.json)
remain unchanged. The production DB-claim probe claimed 300 owners in
2.84 seconds and restored/committed controls and snapshots in 8.74
seconds, with zero owner loss or settled-idle SQL; its provider
references were synthetic. The measured checkpoint/snapshot/renewal
paths retain the same persistence behavior after integration; later
accounting and business writes have separate owners.

Actual Runner stream tests at `da8c7f6e1837ddd896dfe83a7524812fee813487`
passed 8/512/8,192 chunks: zero recovery SQL, transactions and changed
rows during text intervals; zero settled-idle SQL. Each retained seven
control transactions and twelve changed records. Existing renewal SQL
was measured separately. The guard/scheduler fixes add no stream path,
pool or recurring idle scan.

The live fixture used current-source producer SHA-256
`f0000c90cb83880191479da639201e659aa6ce0fc7d67c859faa172ec977849e`, a
normal locked exact-head build with pinned Rust 1.97.1 in Bookworm.
Actual network-isolated, read-only ABI execution passed (GLIBC
requirement 2.34 against runtime 2.36). The immutable base image
remained
`ghcr.io/paperclipai/paperclip-daytona-runner@sha256:36069788f53c51f228764fa36180575443dfb5d1f84a342d7c0daceb28dc61cc`,
source `a0ef3ce7`. This was an unpublished development producer,
explicitly distinct from the base image and admitted before the fresh
fixture began. The earlier host build failed ABI preflight before
admission. The preceding old-producer fixture completed its initial task
but lacked the usage-basis marker required by current master; it
remained pending/unpriced and correctly paused. It was independently
cleaned without debt or budget changes. Two preceding local Linux setup
attempts OOM-killed before provider allocation, and their cleanup
passed. These attempts remain separate from the final recovery failure.

**Failures retained and resolved separately.** The earlier `98f05fc`
probe rejected a valid settled PRP v2 goal suffix; `eaef0d5` fixed that
reader and the fresh follow-up passed. Later component cleanup failures
were traced by a scoped exception observer to the fixture adapter
rejecting directory downloads. A bounded owned-directory transfer with
exact-byte/mode preflight corrected the fixture; source and installed
strict-cleanup cases then passed. No production guard was relaxed. The
initial local integrated control-port run hit its 30-second embedded
Postgres startup hook before executing 59 cases; 18 other focused cases
passed. A bounded rerun used a fresh owned external Postgres fixture,
applied normal migrations before Vitest, and passed all 59 cases with
the original timeouts unchanged. Its container and data were removed;
the failed attempt remains retained.

The additional opt-in full Product E2E Vitest run is not green: 5,313
passed, seven failed and three skipped. An actual `abed921e12` archive
with the same dependencies, Node and IPC permissions reproduces all
seven failures with the same reasons. Five are unchanged
workflow-policy/profile-pin assertions, including Pi profile version 37
versus the test pin 35. Two database suites failed during embedded
Postgres startup before their assertions; semaphore exhaustion is
confirmed in the observer log. The observer passes its supported
external-Postgres rerun with original timeouts and source unchanged. The
controller-database client-kill/postmaster assertions remain unexecuted
after `initdb` bootstrap failure; no external override is supported. The
unit launcher stopped before its Node-test phase, which was then run
independently and passed 181/181. The original failed full run is
preserved. These results remain distinct from the passing focused checks
and normal repository CI. The first local Product E2E typecheck also
encountered stale compiled Runner declarations; a genuine package-local
TypeScript compile refreshed them and the unchanged check passed,
without a dependency install or producer rebuild.

Earlier [Daytona campaign
41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900)
failed at sandbox creation before any agent, with the underlying
provider error unavailable. Its [sanitized
report](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-41108162900-2/index.html)
remains a `cleanup_failure`. Both old creation names are absent in the
current authenticated account, but the old CI account identity was not
retained, so account equivalence is unproven. The subsequent exact
`eaef0d5` source Product E2E attempt passed image/agent setup and the
initial project, then held the controller offline for 300,003.7 ms. Its
Mac supervisor failed owned-process inspection before B readiness; no
adoption passed. Full current-company lease/label checks, provider
deletion, exact-ID 404s and complete inventory confirmed cleanup of that
attempt. Its local server and owned Postgres stopped. The prior AWS
campaigns and database qualification VM were terminated with root
volumes removed.

The old admitted test Runner lacks the newer usage-basis marker
introduced on master. The integrated component B turns retained
unpriced, incomplete accounting obligations with `costAccountingPending:
true` and no settled charge. The compatibility-reader A turn used its
legacy receipt interpretation but likewise retained pending accounting
and no settlement timestamp. This proves obligation preservation, not a
whole-run accounting migration. Runner/receiver release ordering remains
governed by the usage owner's contract.

At native implementation `abed921e12d543b62f57a56435419c0dafb0bc33`,
Greptile was 5/5 with zero unresolved threads and CI completed with 54
successful checks and five expected skips. At `0b6afc499f`, Greptile
again passed 5/5 with zero unresolved threads, but CI ended with 51
successful checks, five skips and three failures. Server 9/12 fails a
deterministic managed-skills maintenance assertion newly inherited from
master `198218bf` in merge `c80e00ad`; that test is absent from the PR
head, and the storage owner has the explicit-StorageService fix in
#18357. Runner 2/2 passed 2,283 unit cases but its runner received a
shutdown signal during the long stream probe; its cancelled step and
aggregate failure are preserved. No blind retry was used. Current
`af9e2d157d` checks are pending; the matcher-only change passes 144
focused cases and Product E2E typecheck. Earlier scheduling-sensitive CI
failures and the interrupted unsharded local test run remain recorded in
the dated plan. CI runs the repository's normal test partitions. Final
verification does not claim Railway rollout, uninterrupted HTTP service
or whole-webserver filesystem loss.

</details>

## Risks

- The task-pointer repair passes its assembled regression. The first
live repair exposed pre-acceptance ordering and the fresh review exposed
a replay lock-order issue; both are fixed and pass real Postgres tests
at `0862062`. The following live attempt exposed the
verified-running-versus-suspended preparation gate. Its narrow fix
passes 91 focused cases at `64c0b746`. The next live case exposed
synthetic attachment IDs; `7a203274` fixes that field selection and
passes real Runner authentication. The subsequent queued continuation
exposed strict retirement maintenance and a false default-home change;
`39a4343` fixes both with real Runner/Postgres and fingerprint
regressions. The `39a4343` live continuation failed again despite green
CI. `283e05ffaa` repairs the actual native-observed admission ordering
and reproduces/fixes the JSONB skill-order fingerprint drift. Both pass
targeted real-Postgres regressions and the live active plus queued
paths. The definition 6 post-C browser navigation timed out before
admission. Explicit definition 7 now verifies native continuation
through the ordinary comment API; its strict post-C native gate failed
after a real dispatch and remains historical failed evidence. The final
fc68584 repair now passes its fresh strict 49-check Linux/Daytona
campaign; the prior failed grades remain unchanged. Browser reconnection
is handed to its own workstream with the original failed evidence. No
prior failed campaign is reclassified as passing.
- Incorrect checkpoint/source binding or stale ownership could duplicate
external work. Missing evidence remains a hold; existing receipt and
finalization owners keep uncertain effects unresolved.
- Recovery must retain maintenance writes after accepted results without
allowing a stale business turn to write. Real lock-wait and generation
tests cover that boundary.
- The current two-connection paired burst profile fails its absolute
drain gate in both baseline and candidate, despite passing relative load
gates and all correctness checks. A causal capacity diagnostic selected
three slots; the final controlled cap-three profiles pass unchanged
thresholds. Full application replacement and two-server overlap capacity
remain separate. There is no new streaming writer, pool or recurring
idle scan.
- The remote sandbox and Runner must survive the recovery window.
Provider auto-stop is a separate lifetime bound from reconnect grace;
the explicit qualification fixture uses a longer idle window.
- Workspace and provider backup availability remain external
dependencies. This PR does not certify whole-webserver filesystem loss,
uninterrupted HTTP service or a Railway rollout.

## Model Used

Implementation and initial verification used OpenAI GPT-6 through Codex;
the initial hosted snapshot was not exposed. The follow-up qualification
used three OpenAI `gpt-6.1-sol` workers coordinated through Codex, with
repository tools, code execution, CI and scoped test infrastructure.
Context-window size is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused and exact-source
application/cost checks pass; full forced CI passes; earlier local
embedded-PG setup failures remain recorded)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (exact fc68584 full CI and
auxiliary checks pass; forced complete test partitions)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(exact fc68584 check and edited summary pass; all six threads resolved)
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 07:33:52 -05:00
DottaandPaperclip db11d899cf fix(native): recover remote sessions while preserving warm runs (#16451)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native agents keep remote processes and conversation history across
warm turns.
> - A completed turn can leave handoff authority only in controller
memory.
> - Losing that controller can block a follow-up even while its sandbox
survives.
> - This pull request verifies durable ownership and reconnects the
exact prior process before handing it to the next run.
> - Healthy follow-ups stay warm. Recovery preserves saved work without
replaying a completed turn.

## Linked Issues or Issue Description

**What happened?**

A remote warm turn can finish without suspended backup authority. A
replacement controller rejects that prior owner even when its exact
process and completed checkpoint remain. A single unavailable identity
probe can also stop an otherwise valid reattachment.

**Expected behavior**

Keep healthy follow-ups warm. Recover a completed session after proving
that its controller stopped and that the successor holds current
authority. Reuse compatible processes; otherwise save and resume the
conversation. Retry unavailable identity reads within one shared
deadline.

**Steps to reproduce**

Use `doc/testing-native-remote-continuity.md`. Its Linux cases execute
real turns through PostgreSQL, TLS ingress, and the Rust runner. They
kill a separate controller process, recover old checkpoint metadata, and
change process configuration. The provider is deterministic.

Related: #16432 and #16441 fix live warm-owner handoffs. #16341 adds
recovery diagnostics. #14695 covers provider process retention.

## What Changed

- Preserve warm process reuse between healthy remote turns.
- Verify a committed prior result, stopped controller, matching
checkpoint and sandbox, and current task/coordinator ownership before
recovering an orphaned warm session.
- Reconnect through the prior run route, then use the existing
authenticated run attachment to change authority.
- Persist a process-configuration digest. Close and save recovered
sessions when old or changed configuration cannot support reuse.
- Retry unavailable state, marker, and process reads within one
30-second deadline. Keep signals and definitive identity failures
single-attempt.
- Record closed recovery diagnostics, including unreadable or malformed
saved state.
- Retire the recovery-only ownership check after verified attachment, so
later warm turns use their current authority.
- Add real-process Linux regressions and refusal controls.
- Expose the existing source-only file-recovery form for retained native
holds, with the same board authority, evidence, and exact stop-receipt
requirements as legacy holds. Recording recovered files does not change
task status or replay a run.

## Verification

- `pnpm -r typecheck` passed; the server check was repeated for the
third-turn fix.
- `pnpm build` passed after the final UI change. The UI typecheck, token
gates, and map checks also pass.
- Four Linux continuity cases passed: warm reuse, controller loss,
legacy checkpoint metadata, and changed configuration. Each creates one
conversation and starts exactly three turns, including a healthy
follow-up after recovery. The fixture uses expired prior leases; the
third turn keeps the runner and provider processes.
- Greptile caught the retained recovery gate; the third-turn regression
now covers it. An initial test omitted the heartbeat’s saved input and
left a test process after failure; the fixture was corrected and its
disposable container reset before the passing run.
- The current focused executor/restart group passes all 749 tests,
including the original failure classifications.
- Native file-recovery and UI tests pass (52 tests), including blocked
tasks and both native and legacy holds.
- After merging current master and retaining its equivalent CI
expression check, the CI policy tests pass (15 tests) and all four
three-turn Linux continuity cases pass again on
`dec418ae55ff21b4fb067132613658cce445f181`.
- `pnpm test:run` completed its general server group with 20,572 passes,
103 skips, and three failures: the skill-audit child process and two
HTTP test timeouts. The same three suites then passed in isolation (89
tests) without code changes. That local command stopped before later
groups; the complete final-head CI, including general, serialized,
Runner, build, and browser gates, is green.
- Greptile reviewed the combined head
`dec418ae55ff21b4fb067132613658cce445f181` at 5/5 with no actionable
findings; both earlier findings are resolved. All final-head CI gates
and the security scan passed. Staging deployment and retained-incident
disposition remain deployment acceptance work.

## Risks

- Recovery requires exact durable evidence. Live conflicting
controllers, replacement sandboxes, Stop/pause, and uncertain effects
must remain blocked.
- After controller loss, tool-contract or process-configuration changes
can restart the provider. Normal healthy follow-ups retain their
processes.
- The new idle-recovery path has Linux fake-provider evidence. Live
Codex/OpenCode and retained quarantined-state recovery still need
staging qualification.
- No database schema, telemetry, or runner protocol changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution, and browser tools. The session does not expose a more
specific model version or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted checks and
isolated reruns; full-run limitation recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green on
`dec418ae55ff21b4fb067132613658cce445f181`
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 13:26:09 -05:00