From 6987c2f20bf2978c84f9c69b9840586407dfad29 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 21:58:09 -0500 Subject: [PATCH] test(upgrades): qualify controller replacement with immutable artifacts and phase evidence (#17455) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/105/changes/paperclip/105 ## Thinking Path > - Paperclip needs evidence that an admitted execution survives a main-server replacement. > - Runner mechanics and server adoption belong to #17097 and #17439. > - A healthy HTTP endpoint does not prove recovery of the original execution. > - Qualification must identify the serving artifact, natural ownership, authenticated attachment and resumed processing separately. > - It must also retain failed runs and measure normal-operation costs. > - This PR extends the existing qualification machinery and acceptance record. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operators can assess upgrade continuity from phase evidence tied to exact artifacts and profiles. **Current behavior** A same-artifact restart or synthetic executor pass cannot establish different-artifact upgrade and rollback support. Idle SQL gates also do not measure active-run capacity. **Proposed behavior** Use immutable A/B artifacts and strict phase receipts through the existing supervisor. Preserve failed evidence and add measured before/after workload costs. Builds on merged #17097 and #17439. The base is now `master`. #17456 follows for review convenience. This PR lands qualification machinery and the evidence record. The remaining hosted gates stay open for deployment qualification; merging this machinery does not claim those experiments passed. ## What Changed - Extend the main-server supervisor and installed-artifact verification for actual serving revisions, forward migrations and compatible rollback. - Distinguish HTTP readiness, natural ownership, authenticated attachment, resumed processing and backlog settlement. - Retain the original reconnect grace and execution identity checks. Never substitute a new task or edited lease for continuity. - Reuse real-harness and Daytona qualification paths, with exact profile and artifact provenance. - Bind current Copilot attestation inputs in the Daytona image fingerprint. - Keep one acceptance record, including both failed file-only stress phases and the final passing bounded-pool comparison. ## Verification After retargeting to master, all 158 focused qualification cases across 13 files pass, with E2E typecheck. The first 12-file batch has four report-lock test failures and 133 passes. Master #17811 had committed the exact already-approved output lock, while that fixture assumed HEAD must be the uncorrected input. The fixture now checks both pinned forms, executes the committed-lock path, and reconstructs the pinned input to retain all rejection tests. The production helper, policy hashes, patch and dependency resolution are unchanged. The corrected batch explicitly includes both process-preservation suites; the prior 146-case receipt has its own source and selection. Current candidate `8f1d6ec02f` integrates master `09ad9adce2`, including the merged server slice. Its diff contains only qualification code and documentation. The [qualification record](https://github.com/paperclipai/paperclip/blob/8f1d6ec02f5bf6c90e7b27da8156953cabd9a180/doc/plans/2026-10-22-hot-upgrade-qualification.md) is the acceptance ledger and retains source-attributed failures and hosted limits. On the eighth integration, all 146 cases across 12 focused qualification files pass, including report-lock preparation, workflow security and workflow targeting. E2E typecheck passes. The earlier 118-case result remains attributed to its prior source. The first sandbox invocation has 117 passes and one failure because process inspection was denied during inert fixture cleanup. The permitted repeat passes all 118 and a targeted process check finds no remaining fixture daemon. The earlier 96-case receipt remains source-attributed in the ledger. The latest two review regressions first fail: an intermediate replacement can cite another runner/session despite a valid final step, and malformed receipt JSON aborts evidence packaging. Each replacement now binds run/company/issue/runner/session, and corrupt receipts retain explicit invalid metadata plus sanitized diagnostics while other logs and receipts still package. These checks prove qualification machinery with injected supervisor operations, configuration subprocesses and real filesystem packaging. They do not qualify a hosted harness. Three request-reader cases, 14 artifact/configuration/cache/history cases, and three lost-partial-receipt cases fail before their respective corrections. The final collector preserves sanitized startup, readiness and recovery phase receipts through packaging and temporary-root deletion. Earlier embedded-Postgres initialization, report-catalog drift, CI scheduling/cleanup failures and fixture collisions remain recorded with their corrections or limits. The previous Greptile service error is retained as historical review evidence, not a current review pass. The record includes prior source `9bb9d3417e`'s actual Linux contention/fairness successes and the ordinary-warm regression proof from the server slice. The historical 300-session component campaign passes four after phases and retains earlier burst failures. Those component costs do not establish hosted capacity. The hosted API-only pair has zero agents/viewers, fails both absolute latency gates and the candidate memory gate, and does not complete the active-agent/viewer matrix. The strict ten-agent same-artifact restart remains attributed to its original artifact. It does not qualify this updated stack. Actual A→B/B→A, the normal drain-free Cloud rollout, and other hosted harnesses remain open. No new provider turn, deployment or hosted qualification was run during merge preparation. Prior source `57d3767eac` jobs 122590643944, 122590643983 and 122590644161 ended after runner shutdown/cancellation without final Vitest results. The shutdown cause is unproven. No retry was issued before this source was superseded by the required server recovery fix. These attempts remain failures/incomplete evidence. Fresh exact-head CI, legacy status, completed Greptile 5/5, resolved feedback and normal merge eligibility remain required. ## Risks Different-artifact A→B/B→A, the normal drain-free Cloud rollout path, other hosted harnesses and the complete concurrency/viewer matrix remain open. Frozen branch builds are development-build evidence. This PR owns qualification and evidence; production defects must be fixed in their earlier owning slice. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-22-hot-upgrade-qualification.md | 1617 +++++++++++++++++ tests/runner-e2e/EVERYDAY-WORKFLOWS.md | 65 + tests/runner-e2e/OPERATIONS.md | 2 +- tests/runner-e2e/README.md | 2 +- tests/runner-e2e/catalog.ts | 8 +- tests/runner-e2e/controller-database.test.ts | 79 + tests/runner-e2e/controller-database.ts | 43 + tests/runner-e2e/controller-outage.test.ts | 141 ++ tests/runner-e2e/controller-outage.ts | 159 ++ ...ller-recovery-observer.integration.test.ts | 84 + .../controller-recovery-observer.test.ts | 46 + .../controller-recovery-observer.ts | 169 ++ .../controller-replacement-smoke.test.ts | 51 + .../controller-replacement-smoke.ts | 171 ++ .../controller-smoke-cleanup.test.ts | 34 + tests/runner-e2e/controller-smoke-cleanup.ts | 25 + tests/runner-e2e/daytona-image-content.ts | 4 + tests/runner-e2e/everyday-flow.ts | 70 +- .../everyday-remote-restart.test.ts | 73 +- tests/runner-e2e/everyday-remote-restart.ts | 78 +- tests/runner-e2e/harness-env.ts | 6 +- tests/runner-e2e/installed-cli.test.ts | 2 +- tests/runner-e2e/installed-cli.ts | 3 +- tests/runner-e2e/instance-isolation.test.ts | 40 + tests/runner-e2e/instance-isolation.ts | 50 +- tests/runner-e2e/launch.ts | 31 +- tests/runner-e2e/prepare-report-lock.test.ts | 16 +- tests/runner-e2e/runner.spec.ts | 34 +- .../runner-e2e/server-restart-request.test.ts | 60 + tests/runner-e2e/server-restart-request.ts | 68 + tests/runner-e2e/server.ts | 148 +- tests/runner-e2e/shared-evidence.test.ts | 96 + tests/runner-e2e/shared-evidence.ts | 47 + tests/runner-e2e/upgrade-artifacts.test.ts | 61 + tests/runner-e2e/upgrade-artifacts.ts | 99 + 35 files changed, 3541 insertions(+), 141 deletions(-) create mode 100644 doc/plans/2026-10-22-hot-upgrade-qualification.md create mode 100644 tests/runner-e2e/controller-database.test.ts create mode 100644 tests/runner-e2e/controller-database.ts create mode 100644 tests/runner-e2e/controller-outage.test.ts create mode 100644 tests/runner-e2e/controller-outage.ts create mode 100644 tests/runner-e2e/controller-recovery-observer.integration.test.ts create mode 100644 tests/runner-e2e/controller-recovery-observer.test.ts create mode 100644 tests/runner-e2e/controller-recovery-observer.ts create mode 100644 tests/runner-e2e/controller-replacement-smoke.test.ts create mode 100644 tests/runner-e2e/controller-replacement-smoke.ts create mode 100644 tests/runner-e2e/controller-smoke-cleanup.test.ts create mode 100644 tests/runner-e2e/controller-smoke-cleanup.ts create mode 100644 tests/runner-e2e/instance-isolation.test.ts create mode 100644 tests/runner-e2e/server-restart-request.test.ts create mode 100644 tests/runner-e2e/server-restart-request.ts create mode 100644 tests/runner-e2e/shared-evidence.test.ts create mode 100644 tests/runner-e2e/shared-evidence.ts create mode 100644 tests/runner-e2e/upgrade-artifacts.test.ts create mode 100644 tests/runner-e2e/upgrade-artifacts.ts diff --git a/doc/plans/2026-10-22-hot-upgrade-qualification.md b/doc/plans/2026-10-22-hot-upgrade-qualification.md new file mode 100644 index 00000000000..78a1f20ea09 --- /dev/null +++ b/doc/plans/2026-10-22-hot-upgrade-qualification.md @@ -0,0 +1,1617 @@ +# Hot-upgrade qualification + +Date: 2026-10-22. Owner: the existing Runner protocol, native runtime integration, +and Product E2E harnesses. Related implementation: [hot upgrades](2026-10-21-daytona-hot-upgrades.md), +[continuity runbook](../testing-native-remote-continuity.md), PR #17097. + +## Acceptance contract + +An admitted remote agent keeps its sandbox, runner process, provider session and +active turn while the control plane is unavailable for up to 300 seconds. +Server-dependent calls wait and reconnect. After HTTP returns, admitted work and +queued launches recover automatically and new work can launch without an operator +drain. The runner's existing 420-second reconnect grace includes server startup +and ownership acquisition; tests must not reset that deadline on each retry. + +Postgres, the instance encryption key and the sandbox survive. This does not +promise HTTP availability while the only web server is offline, survival of a +destroyed sandbox, retroactive support in older running binaries, or reconstruction +of an acknowledged display-only log tail. Never repeat a provider turn or external +effect to fill a display gap. + +## Review stack and ownership (2026-10-24) + +Four PRs, reviewed in this order. PR #17097 is reused for the runner slice. +The full pre-split branch is retained as `codex/hot-upgrades-reference-2026-10-24` +at `858cfd1fdf621c56a8ce7615bdf5aecc9cf1bb61`. + +| Slice | Owns | Acceptance evidence | +| --- | --- | --- | +| Runner contract | Optional opaque checkpoints, authenticated cursors, durability barriers, retention and deadlines; one-process scripted executor fixture | Outage progress, waiting, replay, persistence failures and backpressure; consumer compatibility and protocol/dependency checks | +| Server adoption | Postgres migration/encryption, transactional fencing, startup/expired-owner recovery, tool authority, detach/Stop/finalization and bounded retention | Real Postgres races, empty disk, uncertain mutations with independent effect ledger, admission and SQL gates | +| Upgrade qualification | Existing supervisor and installed-artifact verification; immutable A/B upgrade and compatible rollback, phase receipts and hosted paths | Natural lease acquisition, original execution identity, exact artifacts, short/full outages, cleanup and admission within 420 seconds | +| Transcript delivery | Additive WebSocket negotiation and run interests; shared visible browser subscriptions, catch-up and bounded delivery | Native/legacy journeys, subscription races, reconnect, congestion, company/access changes and measured viewer costs | + +Tests and documentation ship with their owning behavior. Qualification defects +are fixed in the owning earlier slice. Transcript delivery is functionally +independent and has no migration. Browser interest never changes harness execution, +PRP ACKs, checkpoints, accounting or canonical logs. + +The deterministic fixture links the production runner core and a scripted +`CommandExecutor` in one process. Its counter records completed workload steps; +no auxiliary daemon, HTTP service, production test endpoint or fake-provider +admission option is introduced. It distinguishes useful progress from waiting +on control-plane results and from bounded output backpressure. Synthetic execution +never qualifies an underlying harness. + +Compatibility starts at the first hot-upgrade-capable protocol release. Test the +oldest supported artifact and previous release, as well as compatible rollback. +A frozen branch artifact is explicitly development-build evidence until releases +exist; HTTP readiness or a replacement task is insufficient evidence of continuity. + +## Current execution evidence (2026-10-24) + +The review stack is [Runner #17097](https://github.com/paperclipai/paperclip/pull/17097) +→ [Server #17439](https://github.com/paperclipai/paperclip/pull/17439) +→ [Qualification #17455](https://github.com/paperclipai/paperclip/pull/17455) +→ [Transcript delivery #17456](https://github.com/paperclipai/paperclip/pull/17456). +None is deployed. The remaining acceptance gates below are not waived. + +The final security review found that cache-independent adoption skipped a +preparation callback which also checked managed-project grants. The correction +keeps policy in the plugin adapter: Runner retains at most two opaque configuration +IDs bound to exact execution identities and calls an optional authorization hook +before registration. The adapter calls existing `check_project_auth` with current +project IDs. Existing consumers without the hook keep the preparation callback; +missing cache or bindings fail closed. No extra process, periodic check, or text +checkpoint is introduced. Four transport regressions fail on the previous code; +28 focused runner tests, the complete 429-test controller/transport suites, all +34 plugin-owner tests and three real-Postgres SDK integration cases pass with the +correction. The SDK regression exercises current-grant revocation through the +actual runtime and plugin serialization, with synthetic provider resources. +One warm-handoff case timed out during concurrent server compilation; the isolated +rerun passed. The first Postgres attempt failed before tests because Docker +returned an engine startup error and no database was listening; after engine and +database readiness were confirmed, the integration rerun passed. Runner/server typechecks and protocol/dependency checks pass. Current +CI and review status remain on the linked PRs rather than treating older heads as +proof of this correction. + +The next review found a stale Copilot source identity and a checkpoint fence that +sampled database time before waiting for a row lock. The generated active profile +and existing controller mocks are updated; all 90 affected runner compatibility +and permission tests pass. The writer now locks in a CTE and samples wall time in +the outer query, retaining one SQL statement and round trip. Its real-Postgres +regression fails on the prior writer and passes with the fix, without changing +stored checkpoints after expiry. The full recovery run passed 38 cases and skipped +five optional profiles; two unchanged renewal cases failed while Docker became +unavailable. After restoring the daemon and a confirmed-ready database, both +renewal cases and the new lock-expiry test pass together with natural lease timing. +The repeated 8/512-chunk SQL gate still records eight checkpoint writes, seven +transactions, fourteen changed rows, zero streaming SQL and zero idle SQL. WAL +was 37,424/37,720 bytes (diagnostic only). Runner identity checks and server +typecheck pass. The first CTE implementation failed typecheck on a nested subquery +projection; selecting its three needed fields explicitly fixed it before testing. + +The next server review found that instruction update/restore receipts use call +IDs rather than generic idempotency keys. Recovery now reuses their existing +locked receipt and current target-authorization checks in a mode that cannot +reserve or repeat a write. Both lost-commit-acknowledgement regressions fail on +the prior code. All 13 real-Postgres/filesystem instruction tests and 29 existing +tool-authority tests pass, including unchanged bytes and one audit effect, +conflicting arguments, revoked permissions, Stop, unknown outcomes and a receipt +removed before locked replay. An initial companion-suite attempt exhausted macOS +Postgres semaphores before tests; both suites then used the same disposable-host +helper with independent databases. Server typecheck passes. + +Transcript review also found legacy catch-up restarting sibling reads and an +interest acknowledgement clearing a stalled delivery deadline. Two legacy +regressions and the blocked-delivery deadline regression fail on the prior code. +The corrected hooks pass 31 focused tests; all 12 delivery tests, UI typecheck +and token gates pass. Terminal refreshes coalesce per run, and resubscription +cannot extend the original five-second backlog deadline. Existing protocol, +checkpoint and canonical-log ownership is unchanged. Updated CI, review and +browser evidence remain attributed to their candidate revisions on the PRs. + +The separate Paperclip review found premature local expiry after a delayed +successful renewal and unlimited retries in opportunistic startup retention. +The renewal owner now samples a monotonic deadline immediately before UPDATE, +after acquiring the connection and row lock, while still deducting UPDATE/commit +response time. Its public void-returning API and SQL work are unchanged. Startup +retention stops after three failed passes, including query failures, and defers +remaining rows until the next startup. The delayed-success and both retention +regressions fail on the prior implementation. All 44 renewal/retention tests pass, +including quiet time after the retry cap and successful cleanup at the next +startup. Two real-Postgres renewal cases also pass in 107.45 seconds: one saturates +the liveness pool through two delayed successful renewals with natural 60-second +lease timing; the other retains application-pool isolation, SQL timeout and +expiry-during-lock checks. Server typecheck passes. Final review and CI status +remain attached to the exact candidate revisions on the PRs. + +The final runner review found that a rejected persistence callback permanently +poisoned later flushes, and an adoption authorization hook could be silently +ignored without launch preparation. The runner now rejects that invalid option +combination before registration. Persistence keeps the last successful revision, +resets the next write after an uncertain outcome, and calculates queued deltas +against successful state. Three persistence regressions cover failure before +commit, a lost commit response, reconnection with the same lease and command, +and a queued successor. Those tests and the adoption-option regression fail on +the preceding implementation. The corrected controller/transport run passed +432 of 433 tests; the unchanged child-diagnostics test exceeded its five-second +timeout and passed a bounded isolated retry without changes. The failed attempt +is retained. All 39 focused profile tests, both Runner TypeScript checks, +protocol manifest/schema and module-boundary checks pass. The refreshed source +profile is `sha256:fac29e13622eadf9ab6f6dd1c47b2384ffa6307a2edb942dd78d3aae8e8d501f`. +This identity refresh is not new live-provider qualification. + +CI 40141476713 found a race in the new persistent-retention regression: an empty +next-deadline value could mean a retry was still running. The test now waits for +the owner's final deferral warning and exactly three failed attempts before +checking three hours of simulated quiet time. All 39 retention tests pass with +disposable Postgres. The first local run timed out in database setup; a warm retry +passed with the same setup timeout. Product retry limits and quiet-time assertions +are unchanged. The runner fixes belong to PR 1 and this test fix belongs to PR 2. +Final CI/review results remain attached to each exact PR candidate. + +CI 40144348549 then caught two test issues. The regenerated Copilot identity +still had an old pinned digest in the installation test; the corrected pin +passes all 95 installation/profile/permission tests. The in-process recovery +fixture observed all 64 events at the controller before the last eight +acknowledgements reached the runner. It now waits for the independent runner +receipt to report 64 acknowledged events and zero retained events. All three +fixture cases pass in 16.18 seconds, including filesystem-only compatibility +and deadline expiry. This strengthens the settlement witness without changing +production behavior, the workload count or the reconnect deadline. Both failed +CI shards remain retained; fresh CI is required for the corrected candidate. + +| Layer | Latest evidence | Limit / next gate | +| --- | --- | --- | +| Runner | Initial split: 191 controller/profile, 96 Rust, 11 parity tests and three one-process fixture cases passed. After refresh onto master, protocol manifest/schema/replay and dependency-boundary checks pass. `2d72e827c991feb0f88d178aaf1ad1e0628f3e83` has Greptile 5/5 and passing unit, Rust, typecheck and build lanes. | The scoped rerun passed after an earlier browser fixture cancellation hit a lock timeout; CI is green for this revision. Synthetic execution does not qualify Codex/Claude. | +| Server | Refreshed recovery, renewal and plugin-adoption suites: 43 passed, five optional profiles skipped. The subsequent liveness-pool fix passed all 822 native executor/renewal tests with real Postgres. Saturated application-pool, cancelled row-lock wait, expiry-during-lock and natural 60-second claim-lifetime regressions passed. | `57278aa4aa11e3106d645615632571a33764eff5` has passing CI and Greptile 5/5 with no unresolved findings. An earlier unchanged cleanup-page failure is retained. Hosted admission/continuity remains outstanding. | +| Qualification | 80 focused tests, the separately executed real-Postgres observer and both embedded-database ownership tests in Linux pass. `29606367add9d6bc0fbc5ae86f8b41ac9392cb61` has passing CI and Greptile 5/5. Short and full 300-second immutable A→B→A startup smokes passed, with zero agent turns. | Active A/B/A, real migration change, natural stale-owner partition, representative concurrent admission and hosted proof remain outstanding. Local Linux ownership proof does not qualify a hosted agent. | +| Transcript delivery | Native notification and per-run catch-up isolation corrections pass 75 focused tests, UI typecheck, token gates and the native two-view/reconnect/reload journey. Earlier native and legacy browser journeys and 37 delivery regressions pass. | An earlier CI attempt failed the unchanged cleanup-pagination timing test; the unchanged suite passes all 14 tests in local Linux and passes the next CI run. Final revision checks and review remain the PR gate. Deterministic browser dependencies are not live-harness evidence. | + +The supervisor observes HTTP readiness, natural ownership acquisition, +authenticated attachment, resumed canonical ingestion, and backlog settlement +separately. The read-only observer uses one repeatable-read database snapshot and +the Runner-owned checkpoint evidence projection. A new durable authentication +count and new canonical source sequence are both required; replaying saved events +cannot pass. The observer never renews or edits a lease or invokes recovery. +Its SQL is qualification overhead, excluded from product SQL cost measurements. +Postgres belongs to the supervisor rather than the killed serving process; +external Postgres is left alone, while an owned embedded database starts empty. +Each serving artifact applies its own migrations. + +The credential-free startup smoke used these **development builds**, both 0.3.1: + +| Artifact | Serving revision | Complete installation SHA-256 | +| --- | --- | --- | +| A | `f695ca7faac27d055441bb60bf377831bda85622` | `af894bdc71e390f11cfdd9128c828007e94095c0ba7bdab34c0ba1a7ecd01284` | +| B | `5ff132042f1071fdbc45624f6ab1e4d1bfac6598` | `bc3dd69e2c704a6c7c98804a1653d3b8a4e8fd8b88676c043dad891c0e506d15` | + +The short smoke held the server down for 1,000 ms per replacement. Readiness +arrived after 25,305.985 and 33,658.933 ms. The full smoke held it down for 300,000 ms +per replacement; readiness arrived after 327,622.436 and 327,668.142 ms, within +420 seconds. Both preserved the database, read existing data, accepted a fresh +company mutation after each replacement, and cleaned up successfully. Receipts +explicitly record `providerTurns: 0` and `activeAgentContinuityQualified: false`. +A/B have identical migration journals: this establishes compatible startup and +rollback, not a nontrivial forward migration. These builds predate the final +server claim-renewal fix, so they cannot qualify that candidate's active recovery. + +Private receipt locations: `paperclip-runner-e2e-k6aRaI/replacement-evidence.json` +(short) and `paperclip-runner-e2e-EXG9cW/replacement-evidence.json` (300 seconds) +under the host temporary directory. Harness revisions were respectively +`faaf31890e5ae96227bcfc0270bd5d94a92a2641` and +`446d89245bad472abe6b5a793365004858df1e63`. + +Real Postgres cost comparison: 8 and 512 text chunks each produced 8 checkpoint +writes, 7 transactions and 14 changed rows for the same control operations. +Both measured zero streaming checkpoint SQL/transactions/rows and zero idle SQL. +WAL was 37,408 and 37,528 bytes; this diagnostic is not a production capacity +estimate. Receipt: `/tmp/paperclip-hot-upgrade-pr2-sql-cost-20261024-2.json`. +Independent mutation tests covered a lost HTTP response after the business row +committed and a lost runner response after its API receipt was saved. Both +checked the business table independently and counted one dispatch. + +Failed attempts remain distinct from evidence: plain `pnpm deploy` omitted +public manifests, bundled dependencies and runtime skills; the existing release +preparers fixed the artifact before it was frozen. An initial 300-second smoke +could not connect to stopped Docker and launched no agents. Native macOS Postgres +setup repeatedly exhausted host semaphores; integration checks moved to a +throwaway Docker database without changing product recovery. The earlier hosted +attempt exhausted BuildKit disk during image export, before launching a sandbox. +No expensive retry should proceed without resolving its infrastructure cause. +The current trusted hosted workflow has no per-run disk-capacity override or +upgrade-manifest input. No local Daytona/model credentials are configured. +Workflow changes on this branch cannot replace its required default-branch +workflow authorization. Hosted continuity remains unqualified until an authorized +path can build the image and consume the immutable A/B artifacts. + +After refreshing the stack onto master, workspace typecheck and all 43 targeted +recovery, renewal and plugin-adoption tests passed (five optional profiles skipped). +Master already fixed the stale stylesheet-preload browser fixture. Runner, server +and qualification CI pass at the revisions above. The older broad Linux source +verification of `c7bdc72416c98b6781b11452eba878036a9f862a`, +[run 40127795703](https://github.com/paperclipai/paperclip/actions/runs/40127795703), +was cancelled after the missing embedded-database tests passed directly in Linux. +It is not a full-source pass. Current revision CI remains required. + +### Transcript delivery measurements + +The fixed workload emits 1,024 events of 2,048 characters in bursts of 16. These +are measurements of the in-process delivery boundary with simulated sockets. +They exclude canonical persistence, real network behavior, browser rendering, +and production capacity. Heap deltas depend on garbage collection. + +| Views | Full transcript deliveries | Encoded bytes | Authorization calls | Elapsed ms | CPU ms | Heap delta (bytes) | p95 delivery ms | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| Zero interested views | 0 | 230 | 1 | 2.310 | 3.014 | +1,364,856 | — | +| One view | 1,024 | 2,259,853 | 1,024 | 8.205 | 18.437 | +2,254,512 | 0.082334 | +| Three browser connections | 3,072 | 6,779,559 | 3,072 | 12.543 | 12.780 | −2,520,744 | 0.124167 | +| Congested connection | 464 | 1,023,959 | 465 | 2.954 | 2.903 | −2,499,248 | 0.038083 | + +The zero-interest case coalesced one lightweight activity notification. The +normal one-view peak retained queue was 16 items / 35,344 bytes. The congested +case requested resynchronization while staying inside the delivery bound. +Private receipt: `/tmp/paperclip-live-delivery-cost-20261024.json`. + +The native Playwright journey passed in 39.5 seconds (50.8 including setup) on +`267d221f4659ff17d95c99ad5fac5e77f37c8538` plus its test-only transport fault +injection and document-scoped connection accounting. It used a real runner, +disposable Docker Postgres, and the deterministic Codex app-server. The run +`efd7ca4d-7b63-4b02-8312-44bbcad32f91` completed once; both views retained the +final output through reload and the business comment table contained one final +comment. There were zero transcript history requests during a settled 2.5-second +streaming interval, exceeding the fallback's 2-second poll interval. Earlier +attempts exposed obsolete test selectors, provider-discovery precedence, and +Chromium's failure to close existing sockets during offline emulation; none was +recorded as a passing reconnect check. + +## Evidence layers + +| Layer | Execution and owner | What it proves | Cadence | +| --- | --- | --- | --- | +| Protocol | Runner tests; controlled ACK/persistence/effect faults | Replay, authenticated sparse cursors, durable barriers, bounded queues | Every relevant PR | +| Recovery integration | Real Postgres and Rust runner; deterministic provider | Ownership races, encryption, empty cache, side-effect settlement, same turn | Every relevant PR | +| Server replacement | Product E2E wrapper kills/replaces the actual main server; production HTTP workflow | Startup adoption, natural lease timing, same sandbox/run, finalization and admission | Relevant PR; exact hosted cell before release | +| Version compatibility | Immutable server A then B, admitted runner digest, real migrations; compatible rollback | Upgrade contract across different artifacts | Before release | +| Efficiency | Identical scripted workload with and without checkpoints, plus restart burst | Incremental DB work, latency, throughput, backlog and resource bounds | Relevant PR; capacity campaign before release | + +Keep these claims separate. A scripted provider is a reproducible correctness +instrument, not a qualification of the real provider or Daytona. A same-version +restart is not an A-to-B upgrade. A wrapper test is not an agent continuity test. + +## Fault and workload matrix + +Start with one server replacement and one cost comparison; extend the same harness +rather than create parallel recovery implementations or production test switches. + +| Axis | Required cases | +| --- | --- | +| Fault | Graceful stop; SIGKILL; transport partition with old owner alive | +| Unavailability | Short; around the natural lease expiry; full 300 seconds | +| Work at boundary | Streaming; server read waiting; mutation committed with response lost; completion during outage; queued launch | +| Competition | Two successors; stale owner resumes; Stop/authority change; second restart during recovery | +| Versions | A to B plus migrations; B to compatible A; unsupported runner fails explicitly | +| Load | Expected peak; twice peak; synchronized reconnects; idle with retained history | + +The harness records that the original run was active at injection. An unexercised +boundary fails, even if a later task succeeds. Use deterministic gates for the +scripted workload; wall-clock sleeps alone cannot establish where an effect was. +Never edit lease timestamps or recovery state in a server-replacement journey. + +## Hard correctness gates + +- Preserve sandbox lease, runner PID **and process start identity**, provider + session and active turn. The provider call ledger must show one turn start. +- Check an external fixture's effect ledger independently: a write occurs once, + including when its response is lost. Tool-ID deduplication alone is insufficient. +- An uncertain write is visibly held; an ordinary read can retry under current + authority. A stale owner cannot write a command or acknowledge an event. +- The original run commits once, including accounting and workspace finalization; + no stranded operation or recovery action remains. +- Queued/admitted work resumes and a newly submitted task launches after readiness. +- Record fault receipt, actual elapsed outage, server readiness, natural ownership + eligibility, reattachment and backlog-drained times separately. Recovery must + fit the original 420-second grace. A healthy HTTP endpoint alone cannot pass. +- Missing provenance, missing boundary evidence, unexpected skips, cleanup failure + and automatic retry do not qualify an attempt. + +## Efficiency gates and measurements + +The lean design keeps control state, not a second transcript. The existing run +log, business writes and 20-second lease renewals remain baseline work. + +1. **Zero checkpoint writes for ordinary text chunks.** Compare a small stream + and a much larger stream with the same control operations. Assert callback, + SQL and changed-row counts, not just final table size. Updates to one row are + still load. Calibration must make a deliberate extra write fail the gate. +2. **Zero added checkpoint SQL while idle after startup.** Exercise both empty + and retained history. Retention may do its bounded startup pass; the existing + filesystem cleanup timer must not query checkpoint rows. +3. **Measure each control boundary.** Capture checkpoint transactions, inserted/ + updated/deleted rows, payload bytes, WAL bytes and lock waits in an isolated + Postgres database. Report measurement-query overhead separately. Never count + a mocked persistence callback as a WAL measurement. +4. **Compare normal-operation cost first**, then restart cost: same fixed seed, + concurrent agents, output rate, tool mix and duration; warmup plus repeated + paired samples. Report throughput, CPU, peak RSS and p50/p95/p99 control latency. + Provisional acceptance is at most 5% throughput regression at expected peak; + establish the baseline and expected peak before treating that as a release gate. +5. **Bound backlog and recovery pressure.** A five-minute outage must fit the + configured runner outbox budget and drain without starving new commands or + fresh admissions. At twice peak, pressure must remain bounded and explicit. + +Do not infer production dollars or capacity from a laptop timing result. Persist +raw sample counts and distributions; noisy timing is diagnostic, not a flaky +single-sample CI assertion. Structural zero-write checks are deterministic gates. + +## Attempt record and implementation order + +Every attempt retains the two server revisions/artifact digests, runner digest, +workload configuration, fault receipt, monotonic elapsed times, independent effect +ledger, recovery identity, database measurements and cleanup outcome. Store private +logs under the existing harness private-artifact policy; publish only sanitized +receipts. Keep failed attempts alongside subsequent successes; no automatic retries. + +1. Add a bounded, cancellable actual-server outage to the existing restart wrapper, + with a receipt the remote continuity grader requires. Test missing/incorrect + receipts and real process replacement. Leave the existing finalization fault + and local process ownership contracts intact. +2. Add repeatable Postgres checkpoint-write measurements to the existing native + integration suite, including the zero-stream-write gate and a positive control. +3. Run the exact hosted remote continuity cell at short and full-contract outages. + Resolve the image-builder disk failure before another paid attempt. Do not + promote a suite that failed before sandbox launch. +4. Extend to pinned A/B artifacts, deterministic mixed workload/effect faults, + two-owner contention, outbox pressure and the paired peak-load benchmark. + +## Evidence status at the start of this slice + +- Existing tests cover authenticated sparse replay, transactional checkpoint + fencing, uncertain tool recovery, delta updates and idle/retention behavior. +- The 300-second integration proof uses real Postgres/Rust and a scripted provider, + deletes controller files, and retains the active turn. It **manually expires a + lease and does not replace the main server**; it is not the server-upgrade gate. +- The hosted `everyday-workflows.runner-codex.daytona.recover-controller` cell + already replaces the server and grades original-run identity and finalization, + but needs an explicit outage receipt and configurable crash/outage duration. +- Hosted attempts 39613279281 and 39616972414 failed in image preparation with + ENOSPC before sandbox launch. No Daytona continuity pass is claimed. +- Different server versions, independent external-effect counting, mixed-load + capacity/WAL measurements and production cost estimates are not yet qualified. + +Implementation results and exact commands belong below, with their limitations. + +## Implemented qualification slice + +- `tests/runner-e2e/controller-outage.ts` owns bounded fault injection and + monotonic phase receipts. The actual server wrapper uses it only for the + explicit remote cache-replacement cell. Signal, minimum hold (0..300000 ms), + old/new process IDs and incomplete attempts are inspectable. Cancellation + prevents a successor launch. Missing/short/foreign fault receipts fail grading. +- The existing hosted cell defaults to SIGKILL and a 1000 ms hold, requires this + receipt, and forbids automatic retries. Its definition version and harness + fingerprint changed. The full hold is selectable using + `PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS=300000`; SIGTERM is also selectable. +- `controller-replacement-smoke.ts` is a credential-free actual-main-server + probe with Postgres owned by the test driver. It checks unchanged Postgres + start time, existing HTTP data, new HTTP mutations and complete process cleanup. + It deliberately reports `activeAgentContinuityQualified: false`. +- The native restart integration suite now measures actual checkpoint SQL through + postgres.js, checkpoint transactions/changed rows/bytes, diagnostic cluster WAL, + and elapsed time for identical control operations with 8 versus 512 text chunks. + The fixture holds text until control setup settles, then each released stream + must add zero SQL statements, transactions or changed rows. Total control setup + counts remain diagnostic because batching can vary. A subsequent real control + write calibrates the observer. A settled-stream interval must issue zero SQL. Broader + idle cleanup remains covered by the existing retention tests. These timings and + WAL values are not a paired capacity benchmark or production cost estimate. + +Commands (run from the repository root): + +```sh +pnpm test:e2e:runner:typecheck +pnpm test:e2e:runner:unit -- --maxWorkers=2 +node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/controller-replacement-smoke.ts +PAPERCLIP_TEST_CHECKPOINT_COST_REPORT=/tmp/checkpoint-cost-attempt-1.json pnpm exec vitest run server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'keeps checkpoint SQL independent' --maxWorkers=1 +``` + +Choose a new cost-report filename for each attempt; existing evidence is never +overwritten. Build the existing Rust fake provider after changing its fixture +flags (`cargo build --manifest-path packages/paperclip-runner/runner/Cargo.toml +--locked -p paperclip-runner-core --bin fake-codex-app-server`). No LLM credentials +are needed for these local probes. + +Initial evidence on 2026-10-22: + +- 50 focused fault/oracle tests passed. Server typecheck passed. +- Actual-main-server SIGKILL smoke passed: a 1000 ms confirmed-down hold, readiness + 62.82 seconds after injection, unchanged Postgres start time, data continuity, + new mutation and confirmed cleanup. This used source revision 436b2ce12 plus + the working changes, not two released artifacts. +- The real Postgres/Rust cost gate passed with 64 times as many text chunks. + Earlier 4096-chunk attempts exceeded the original ten-second observation bound; + they did not establish a DB regression or pass. The fast gate now uses 8/512 + and a bounded 60-second observation window. Large backlog capacity remains a + separate gate, not silently qualified by this smaller sample. +- The first smoke launcher attempt failed on workspace package imports before + server launch. Direct source imports fixed the standalone entrypoint. +- The first complete support-suite run had 3571 passing tests, two explicit skips, + and five failures in existing catalog version/count and Daytona image-input + assertions. These must be reconciled before calling this slice PR-ready. + +Follow-up validation on the merged branch (`d0a0b830e7`, plus the request-receipt +correlation check): + +- The full **300000 ms** actual-main-server SIGKILL smoke passed with Postgres + continuously alive, old controller cache unavailable, existing data readable, + a new HTTP mutation accepted and cleanup confirmed. This remains a zero-provider + startup/DB proof; it does not upgrade the hosted agent result to a pass. +- The complete native restart integration file passed **34 tests**, with five + Docker-dependent cases skipped. The focused protocol and Copilot identity + suites passed **191 tests**. Fault/oracle tests passed **50 tests**. +- Workspace `pnpm -r typecheck` and `pnpm build` passed. Earlier typecheck attempts + found a sandbox IPC restriction and Rust fixture formatting; the permitted, + formatted rerun passed. The merged source preserves master's Copilot v58 and + declares the combined recovery transport as v59, including Docker image inputs. +- The reconciled Product E2E support run passed 3573 tests and skipped two, but one + file's worker failed to start. Its separate rerun passed all three tests, and + the follow-on Node suite passed 128 tests. The full invocation remains recorded + as incomplete; it is not relabeled a clean full-suite run. + +Measured control-checkpoint cost for one scripted turn: + +| Text chunks | Transactions | SQL writes | Changed rows | Plaintext checkpoint bytes | Cluster WAL bytes | Settled-stream idle SQL | +| ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| 8 | 7 | 8 | 14 | 13618 | 37224 | 0 | +| 512 | 7 | 8 | 14 | 13636 | 37632 | 0 | + +Control operations are included; the output stream adds no checkpoint writes. +WAL is a diagnostic cluster-level observation, including possible background +work, not an exact attribution or threshold. This is not the promised paired +peak-capacity campaign. A new control command after each sample produced a real +SQL write, establishing that the observer was active. + +Still outstanding: hosted active-agent proof (including the full 300-second +outage), version A/B artifacts and migration/rollback, deterministic mixed +work/effect ledger, natural-lease race matrix, and paired peak/overload measurements. +The hosted blocker was confirmed in the retained build log: BuildKit exhausted +disk while exporting the image, before the workflow could launch any sandbox. + +Review found a read-recovery gap: the copied retry list omitted identity, people, +skill and workspace reads. Their saved inputs could remain pending after a lost +response. Recovery now consumes the Runner's existing canonical side-effect +classification, plus read/mutation metadata on server bridge definitions. The +server shim exposes that existing Runner metadata; it adds no alternate authority. +Read replay calls the normal executor and therefore rechecks advertisement, +company/run scope, current grants and file confinement. External tool annotations +never authorize uncertain-effect replay. Four focused Postgres regressions passed, +including actual file reads, company isolation, missing-file settlement, revoked +workspace grants, stale leases and cancellation. Initial attempts used the wrong +catalog projection and lacked the shim export; existing read tests and typecheck +caught both before the corrected run passed. + +Master advanced during final verification. The merge preserves its released +0372 workspace-restore migration and regenerates the combined checkpoint table +and header index as 0373_powerful_mentor from the complete master snapshot. The +checkpoint SQL retains its idempotent guards; the latest five snapshots remain. +This supersedes the earlier 0372/0373 checkpoint numbering in the historical plan. + +The merged CI run caught a timing assumption in the initial cost comparison: +control messages coalesced into seven SQL writes in one sample and eight in the +other. That is a failed attempt, not a zero-write proof. The fixture now holds +text behind an explicit release file until turn-start events and command results +are durably settled. Each text-only interval must independently cause zero SQL, +transactions and changed rows; total control counts are measurements, not equality +assertions. The existing positive control still verifies the SQL observer. +The first gated local attempt passed: both 8 and 512 chunks added zero SQL, +transactions and changed rows, with zero idle SQL and a successful positive +control. The full merged restart suite passed 36 tests (five Docker-only skips), +and all 29 tool-authority tests passed on a serial rerun after an initial database +setup timeout. Workspace typecheck also passed after the migration merge. + +### Final review corrections (2026-10-24) + +Completed plugin runs may clear their dispatch flag during normal cleanup. The +qualification boundary accepts that state only for the original succeeded run +with its saved task identity and persisted provider-stop confirmation. Active +runs still require dispatch. Missing stop proof and replacement identities fail. + +The server's native renewals now share the existing liveness pool with legacy +renewals. Each active remote lease attempts renewal every 20 seconds, with one +row lock and fenced update inside a transaction and a local 10-second statement +timeout. Expiry is checked after acquiring the row lock. A cancelled attempt +can retry within the original deadline, but a lost or late response cannot +restore expired authority. This adds active-run SQL around renewal; it does not +change the zero-per-text-checkpoint-write or idle-SQL gates. The text-chunk cost +measurement above excludes periodic renewal overhead. + +### Restacked candidate and hands-on browser evidence (2026-10-24) + +Candidate `54fc7c6494470a9d553ef7bce7d57b3f5a57d7ba` passed full workspace +typecheck and build, 80 focused qualification tests, and the separately executed +real-Postgres recovery observer. This includes server renewal fix +`57278aa4aa11e3106d645615632571a33764eff5` and qualification fixes through +`29606367add9d6bc0fbc5ae86f8b41ac9392cb61`. Final-head CI remains the PR gate; +these local results do not replace it. + +A hands-on in-app-browser journey used that candidate, the real runner, the +deterministic Codex app-server and a disposable Docker database. Starting at the +dashboard, the operator created TRA-1 through New Task, opened it from Recent +Tasks, observed live commentary, navigated to Tasks while it was running, returned +to its completed answer and reloaded it. The result and source link remained +visible. API inspection confirmed one successful run +`09cff1ec-5340-48e4-97f0-12334052e4aa` and one final business comment. + +Functional outcome: passed. The observed desktop experience was usable, with no +error banner, duplicate result or lost output. A normal loading skeleton appeared +on navigation; an unrelated announcement overlay was dismissed. The screenshots +do not establish smoothness between every sampled frame. No mobile or live-model +claim is made. Final screenshot: `/tmp/paperclip-transcript-manual-completed-20261024.jpg`. + +### Native notification catch-up correction (2026-10-24) + +Review found that a consecutive native commit notification could be mistaken for +a complete transcript row. The notification contains no PRP body. Native views +now request canonical history through its sequence and never merge the empty +notification over text. The 73 focused hook, projection and sequence tests pass, +including consecutive notifications, delayed duplicates, a notification racing +an older read, failed-read recovery and zero recurring reads for complete live +PRP rows. These tests use the real native transcript projection. Token gates pass. +Payload-free notifications cause event-driven history reads; this is distinct +from the zero recurring reads during settled negotiated streaming. +UI typecheck and the native two-view browser journey also pass after this fix +(39.1-second journey, 53.4 seconds total). That journey rebuilds the shipped UI, +uses the real runner with the deterministic Codex app-server, and verifies socket +reconnect, company changes, terminal catch-up, reload and one final business +comment. It does not qualify a live provider or hosted upgrade. + +### Linux ownership proof and retained CI failure (2026-10-24) + +The previously blocked embedded-Postgres ownership test passes in a disposable +Linux ARM64 container: both tests in `controller-database.test.ts` pass. Killing +the serving client preserves the same database start time and subsequent writes; +the second test rejects paths outside the fixture and leaves external database +configuration untouched. The source checkout is read-only. A temporary dependency +overlay supplies the matching Linux native packages and an unchanged copy of the +repository's patched embedded-Postgres package. No test or production behavior +was changed for this proof. Initial container attempts failed on missing platform +bindings, Vite cache writes, resolver recursion and misplaced native libraries; +their logs are retained with the passing attempt. This is local Linux evidence, +not hosted Daytona evidence. + +CI run 40130375278 on code revision `09d1590432` passed every check except the +unchanged historical-cleanup pagination test: it observed 32 inspections where +33 were expected. Its fixed clock advances can precede completion of asynchronous +database work and registration of the following deadline. The same unchanged +suite passes all 14 tests in the isolated Linux container. The failed CI attempt +is retained and a bounded shard retry was requested; neither test assertions nor +production scheduling were weakened. Final CI status remains visible on the PR. + +### Independent native catch-up (2026-10-24) + +Review subsequently found that a run's live notifications restarted the shared +history effect, aborting slow historical reads for sibling runs. Live catch-up +now addresses only the affected run. In-flight reads and retry timers are bounded +per run; terminal notifications coalesce into one follow-up read without aborting +the current request. Both regression tests fail against the preceding code and +pass after the correction. One holds two historical pages while a sibling receives +repeated commit and terminal notifications; the other verifies terminal catch-up +does not abort or overlap a request. All 75 focused tests, UI typecheck and token +gates pass. The native browser journey passes again (39.6 seconds, 51.9 seconds +including setup), including two views, company changes, socket recovery, terminal +catch-up, reload and exactly one business comment. + +### Agent Detail event preservation (2026-10-24) + +A delayed native commit signal could overwrite a full saved event in Agent +Detail. Both page variants now keep signals out of the merge and require canonical +history only when their persisted cursor does not cover the signal. Both new +regressions fail on the prior merge; all eight page regressions, UI typecheck and +token gates pass. Full live rows still merge without recurring history reads. +The extended browser journey passes (42.3 seconds; 56.7 seconds including setup), +covering the existing two-view/reconnect/completion flow plus the agent's run-event +view and reload. It uses the real runner, deterministic Codex app-server and owned +Postgres. Run `7fcdcc41-e035-4bca-88ea-0bc079083b03` completed once. This remains +browser evidence, not live-harness or hosted-upgrade qualification. + +### Agent Detail reader isolation (2026-10-24) + +The next transcript review found that one native commit signal restarted both +Agent Detail history readers. Event history and shell logs now refresh separately +in both page variants. Each reader finishes its current page and coalesces +terminal notifications into one follow-up. Four regressions hold either reader +while the other receives repeated updates; all fail on the preceding code and +pass with the correction. The first fixture omitted the required agent ID; the +corrected fixture was replayed against both implementations before acceptance. +All 43 focused page/transcript tests, UI typecheck and token gates pass. The actual +native two-view/reconnect/reload and Agent Detail journey passes in 46.1 seconds, +with real Runner, deterministic Codex app-server and disposable Postgres. Run +`b7259b33-bdae-402c-aa25-cc2d827ff454` completed once with one final business comment. +This result remains distinct from hosted upgrade qualification. + + +### Fresh admission renewal and fixture-key isolation (2026-10-24) + +Review found that fresh remote admission acquired its lease before tracing, +accounting-history reads and retained-session preparation, but started renewal +only afterward. The startup owner now starts the existing renewal immediately +after the claim transaction commits and drains it on preparation failure. No new +renewal cadence or idle work is introduced. The controlled preparation-failure +regression fails on the previous implementation. A real-Postgres experiment holds +the claim trace sink until two seconds beyond the original 60-second lease and +confirms the same owner, same attempt and a naturally extended deadline; it +passes in 74.17 seconds. All 824 executor/renewal tests and server typecheck pass. + +The first Postgres fixture used a non-UUID lease identifier; the lease assertions +passed but fixture cleanup failed. The corrected fixture passes. An earlier full +suite could not start embedded Postgres because of host semaphore exhaustion. +The external disposable-Postgres run exposed a cleanup-order regression: awaiting +renewal shutdown before removing the settled startup let detachment reacquire the +old session. Removing the startup synchronously before yielding fixes it; the +preceding 823/824 result is retained alongside the final 824/824 result. The final +run took 625.56 seconds while performing fresh migrated databases for the warm +ownership cases; database activity confirmed progress during that run. + +The qualification supervisor now removes ambient inline/file master-key overrides +at its existing process-launch boundary, matching the serving child's fixture +configuration. A separate observer child decrypts the actual stored checkpoint +under each hostile ambient override using the same fixture key, without changing +any observed rows. This regression fails with the previous launcher and passes +with the fix (6.39 seconds); 134 observer/support/installed-artifact tests pass. +No recovery or cryptography implementation was added to the observer. + +CI attempts 40146472407 and 40146502737 are retained: the former hit the unchanged +agent-device-login route timeout and a following session conflict; the latter +hit EADDRINUSE on port 46274 in the unchanged workspace-runtime HTTPS test. These +are distinct from the fixed lease/observer findings; final candidate CI must run +again rather than deleting the failed attempts or weakening those assertions. + +Hosted acceptance remains open. The trusted workflow was rechecked against master: +it has no per-run image-build capacity override or immutable A/B artifact-manifest +input and still selects the same public-pr fleet used by the earlier BuildKit +ENOSPC failure. No repeat paid image build was launched against unchanged capacity. +The outstanding hosted upgrade/rollback evidence still requires that infrastructure +and artifact-input path; local/CI success does not close this acceptance gap. + + +### Bounded event browsing and stable shared interests (2026-10-24) + +The subsequent transcript review found three presentation defects: Agent Detail +automatically drained every saved event, live retention dropped opening details, +and unrelated interest replacements aborted active history readers. Agent Detail +now automatically reads one 1,000-row page, then offers explicit pagination while +full live delivery continues. The live window retains the first 200 events plus +the latest 2,000, preserving opening details through completion. Explicit page +failures keep existing rows and expose an error with retry. Task transcript +recovery still drains its canonical history normally. + +The shared interest owner now tracks acknowledged run IDs independently of the +connection's recovery generation. Existing interests remain healthy through +unrelated replacements; new interests require their own ACK. Connection changes +and explicit overflow resynchronization still trigger catch-up. An additional +regression covers overflow from the previous revision while a new interest ACK +is pending, so that optimization cannot hide a real gap. + +All seven review regressions fail on the preceding source for their expected +assertions: automatic page count, retained opening details, interrupted native +and legacy reads, and acknowledgement scope. The additional overflow-race test +fails before its correction. All 48 focused transcript tests, final UI typecheck +and token gates pass. Initial large-DOM runs exceeded the five-second test budget; +the two expanded rendering cases now have explicit 30-second budgets, with their +original behavioral assertions unchanged. The final suite took 16.64 seconds. +The earlier partial runs and the corrected before/after replay are retained. + +The final native browser journey passes on source commit `a1f75df8a5` in 59.2 +seconds (1.2 minutes including setup), rebuilding the shipped UI and using the +real runner, deterministic Codex app-server and owned disposable Postgres. Run +`766f50cd-e0a1-48ff-afed-b779c6abd29a` completes once with one business comment; +two views, company switching, socket recovery, terminal catch-up, reload and Agent +Detail events pass. A preceding browser run also passed in 58.5 seconds before +the extra overflow-race correction. These are browser/presentation results, not +live-provider or hosted-upgrade evidence. + + +### Supervisor isolation and initial history errors (2026-10-25) + +The final launcher check still required embedded database mode, although upgrade +qualification correctly moves its fixture Postgres into the supervisor's process +tree and connects the serving child through normal Postgres mode. This could +reclassify a successful recovery as a permanent infrastructure failure. The +supervisor now records its fixture ownership only after starting the database; +the existing isolation check accepts that exact local connection after verifying +the fixture/config/physical data-directory identity, configured port and database +marker. Missing receipts, foreign connections/owners, mismatched ports and a +symlink escaping the fixture remain rejected. The valid owned-connection regression +fails before the fix; the five rejection cases already pass. + +All 117 local isolation/support tests and runner-E2E typecheck pass. A disposable +Linux container runs the real Postgres ownership experiment plus isolation cases: +all eight pass in 2.24 seconds. The actual database remains unchanged when its +serving client is killed, and the launcher check also passes after the supervisor +stops the database. The first Linux attempt lacked a writable Vite cache at the +actual test-local path; mounting only that temporary cache fixes the environment +while keeping repository source read-only. This proves fixture ownership, not +hosted agent continuity. + +Review also found that Agent Detail's empty-state return hid a first history-read +error before the error panel could render. Both variants now display that failure +and an explicit Retry events action even with no existing rows. Both new page +regressions fail before the correction. All 50 focused transcript tests, UI +typecheck and token gates pass. The extended native browser journey passes in +43.9 seconds (58.3 seconds including setup): it forces a 503 for initial event +history while presenting an empty shell log, verifies the error rather than an +empty-state claim, then restores the real route and clicks Retry events to recover +canonical output. Run `1c345f26-0acb-47af-aa76-8f6e2e05ca4d` still completes once +with one business comment. The initial database container launch found its chosen +port occupied; that listener was left alone and a dynamically assigned port was +used for the owned fixture. The prior healthy-streaming and reconnect assertions +remain in this same journey. No hosted qualification claim is added. + +### First hosted stress campaign and preserved identity failure (2026-10-26) + +A new, synthetic test tenant ran the exact published core-only candidate +`00ebac324cc9016a832e7a51e01652af0cbcc758` on Linux, real Postgres and Daytona. +The harness was real native Codex with OpenAI `gpt-5.4-mini`. The authenticated +qualification gateway allowed local tools and namespaces and removed hosted web +search. This is evidence for that profile, not Claude or a full Cloud composition. + +All 13 admitted tasks succeeded: an initial baseline, a short restart, ten +concurrent long-outage tasks and fresh post-restart work. For the ten-way case, +the main Node process was frozen for 300 seconds and then killed. Postgres and +Daytona stayed running. No lease timestamp was edited. HTTP readiness was first +observed at 322.354 seconds after the fault began. Natural ownership claims +occurred at about 315.5–315.9 seconds. All original runners authenticated again +and new canonical ingestion was confirmed by 399.505 seconds, inside the +original 420-second grace. + +Each original workload completed 480 steps, including 298–300 during the +outage. Its PID, boot/start identity and nonce were unchanged. The largest step +gap was 1.135 seconds. Independent recomputation confirmed all ten final hash +chains and exclusive effect files. Each original turn produced one completion, +result acceptance and terminal receipt. All ten workspace exports committed, +and no scheduled or active recovery remained. Fresh native work started in +1.299 seconds while seven original runs were still active, then completed. + +**Strict qualification failed.** The run, runner, native session, PID and sandbox +lease stayed unchanged, but the recorded `processStartedAt` changed to attachment +time in all ten runs. The identity assertion remains in place; the campaign is +not green. The Runner transport reported controller-construction time, and the +remote process handle did not expose the birth time already verified in its +marker. Fixes belong to the runner and server PRs respectively. The two focused +regressions fail before their fixes and pass after. All 293 transport tests, +14 remote supervision tests, repository typecheck and module/package boundary +checks pass on the corrected source. Hosted evidence must be repeated on its +published artifact before this failure can be closed. + +The first short fault attempt checked SIGSTOP synchronously and resumed before +killing anything; it is unexercised. A bounded state-confirmation check repaired +the injector before the actual short and long faults. Setup also preserved +hosted-tool rejection, a credential-directory lock timeout and intermittent +Railway SSH timeouts. Normal finalization released ephemeral sandboxes; final +proof files were read from their successfully committed workspace exports with +exact company/run/workspace binding and path containment. + +Three samples during the freeze show unchanged write counters in +`heartbeat_run_events`, `heartbeat_runs`, `native_run_finalizations` and +`native_runner_checkpoints` while useful workload steps continued. This is not a +healthy-streaming SQL benchmark or evidence of zero database reads. CPU/memory +deltas, throughput savings and production capacity were not measured. + +Final provider receipts recorded 210 requests and **$0.735245850 estimated model +cost**, with no unknown, conflicting or inflight charges. The original $25 cap, +request ceiling and deadline stayed unchanged. Pricing used OpenAI's documented +`gpt-5.4-mini-2026-03-17` snapshot rates; it is not an invoice. The product's rate +card lacks this test model, so campaign policies explicitly allowed unpriced +usage while the independent gateway enforced the cap. Unknown product costs +were not replaced with invented prices. All test work settled before the gateway +closed and test agents were paused. + +Different-artifact A/B and rollback, the 25-way profile and the complete browser +viewer matrix remain open. The current Cloud campaign path still requests task +drain before replacement. These app-only fault results do not qualify a normal +Cloud rollout without drain. Keep deployment-policy work with its Cloud owner; +do not weaken Core drain semantics or report active work as empty. + +### Reviewed full-stack campaign and asynchronous identity failure (2026-10-26) + +The maintained Cloud CLI deployed exact Core-only artifact +`f2945ebeafdffb26fd6234092aeec0c6ff6208ea` to the same dedicated synthetic +Linux/Postgres/Daytona tenant. Publication, rollout and serving health verified +the revision. The approved profile remained production Runner/native Codex with +OpenAI `gpt-5.4-mini`, local tools through the authenticated capped gateway, and +hosted web search removed. + +Ten ordinary assigned tasks each completed 600 deterministic steps and streamed +4 KiB per step. The main Node process was frozen for 300 seconds and killed. +Postgres and Daytona stayed running; no lease deadline was edited. HTTP readiness +was observed by 311.735 seconds. Natural ownership was acquired at +315.999–316.434 seconds. All ten original runners authenticated again and new +canonical ingestion was observed by 378.466 seconds, inside the unchanged +420-second grace. A fresh ordinary task started in 0.964 seconds while all ten +originals were still active, and succeeded. + +Each original workload made 295–300 useful steps during the outage. Its PID, +boot/start-tick identity and nonce stayed unchanged; the largest step gap across +all workloads was 1.792 seconds. Independent recomputation verified all ten hash +chains and exclusive effect files. Every original run retained its runner, +session, PID and sandbox lease, preserved its canonical event prefix, finalized +once and committed its workspace export. No queued recovery remained. + +**Strict qualification remains failed: 0/10.** Every `processStartedAt` moved +forward by 3.040–7.354 seconds to an initial-launch timestamp before the outage. +The prior fix removed replacement-construction time but did not preserve the +asynchronous process owner's getter through `spawnRunner`'s restart wrapper. +Object spread captured undefined before the remote marker arrived. A regression +through this public wrapper fails before the getter-forwarding correction and +passes after; all 143 control-plane tests pass. The correction belongs to PR 1. +This campaign does not qualify that later source. Both failed hosted grades +remain unchanged and inspectable. + +Two successful independent frozen-server observations, 69.332 seconds apart, +show identical insert/update/delete counters for `heartbeat_run_events`, +`heartbeat_runs`, `native_run_finalizations` and `native_runner_checkpoints`, +while each workload advanced by about 68 steps. Checkpoint material stayed at +9,688,877 bytes over 1,607 retained rows, including earlier campaign history. +This is neither a healthy-streaming SQL benchmark nor a zero-read claim. + +The rerun settled 165 model requests at **$0.595740300 estimated** with no held, +unknown, conflicting or inflight charges. Combined campaign cost is +**$1.330986150 estimated**, below the approved $25 maximum. All eleven synthetic +agents are paused, the model gateway is closed, and the requested CLI access is +retained. Estimates use reported provider usage and documented model rates; +they are not invoices. + +Preserved setup failures include an unselected local AWS profile (resolved with +the existing development profile), one normal AI connection probe rejection +followed by a passing diagnostic and scoped binding retry, and one Railway SSH +connection timeout. Later observer connections use a short connection timeout +without changing authentication or host verification. An early incomplete +pre-fault observation remains separate; the actual fault required all ten +complete active boundaries and fresh workload readiness. None of these retries +replaced a workload or edited lease timing. + +Different-artifact active upgrade/rollback, normal Cloud drain-free rollout, +Claude, 25-way load and the full viewer/CPU/memory cost matrix remain open. The +current Cloud campaign still requests task drain. PR 3 remains draft. + +### Qualification receipt review follow-up (2026-10-26) + +The everyday harness digest now includes `controller-recovery-observer.ts` and +`upgrade-artifacts.ts`, whose checks decide upgrade acceptance. The credential-free +smoke captures its owned embedded Postgres PID from the cluster before replacement. +After the bounded database helper returns, it separately requires that process to +have exited before reporting `cleanupConfirmed`. Missing identity, permission +errors and an exit timeout fail confirmation. This probe sends no terminating +signal and does not stop an externally supplied shared cluster; external cleanup +still awaits deletion of only its own disposable database. + +A real child-process regression reproduces the old false success when cleanup +returns with its owner still alive. That regression and the missing-identity case +fail before correction; all 69 affected qualification tests pass after it. The +qualification typecheck initially found stale runner declarations in the separate +worktree. Rebuilding the existing dependency fixed those declaration errors; the +retry passed without a source workaround. One credential-free real embedded- +Postgres/main-server replacement then passed, including continued database +identity, ordinary mutation admission and confirmed final process exit. It reports +zero provider turns and does not qualify a hosted agent or a different artifact. + +This qualification-only follow-up is separate from the immutable production +candidate `73b65d8718fe763a23c53584ac6a82863965b786` being prepared for hosted tests. + +### Strict hosted Codex restart qualification passed (2026-10-26) + +Exact Core artifact `73b65d8718fe763a23c53584ac6a82863965b786` passed the +unchanged strict identity/effect grader on the dedicated synthetic test tenant. +The maintained Cloud deployment verified that revision before model use. This +was Linux, real Postgres, Daytona, production Runner and native Codex with +OpenAI `gpt-5.4-mini` through the approved capped gateway, without customer data. + +A 15-second freeze/kill control passed first. Ten ordinary tasks then each ran +600 deterministic hash-chain steps and emitted 4 KiB per step. The exact main +Node process was frozen at 18:36:05.374Z and killed at 18:41:06.139Z. Postgres and +Daytona stayed up; no lease timestamp was changed. HTTP readiness was observed +by 316.474 seconds, natural ownership at 319.682–320.245 seconds, and new +authenticated attachment plus canonical ingestion for all ten by 389.839 seconds, +inside the original 420-second grace. These are observation upper bounds. + +All ten retained run, runner, native session, PID, process birth time and sandbox +lease while still active after recovery. Each advanced 296–300 useful steps +during the outage and completed all 600 with the independently recomputed chain +and a single exclusive effect. The largest step gap was 1.947 seconds. Each +finalized once, committed its workspace export and left no queued recovery. +Fresh work was admitted while originals were active and completed separately. +The strict grade is **10/10 passed**. The prior two **0/10 failed** grades remain +preserved; the successful getter fix did not remove the process-birth assertion. + +This campaign settled 190 provider requests at $0.642828150 estimated model cost. +All three campaigns total $1.973814300 estimated, below the approved $25 cap, +with no held, unknown or inflight requests. Owned agents were paused and the +temporary gateway closed. Test-instance CLI access was retained as requested. +The complete report and evidence work product are attached to the existing +private test task; private URLs are omitted from this public repository. + +Candidate CI first lost workers in Vitest/browser shards; logs and annotations +remain recorded. A bounded retry passed on the same revision. Full repository +typecheck/build passed. A frozen SQL observer produced no usable receipt in this +campaign, so it adds no new SQL, traffic, CPU, memory or capacity measurement. +The earlier frozen-write observations remain attributed to their older artifact. + +This qualifies a **same-artifact main-process crash/restart for this profile**. +The Cloud rollout path still requests task drain. Active different-artifact +A→B/B→A, a normal drain-free Cloud rollout, other hosted harnesses, 25-way load +and the complete viewer matrix remain open; PR 3 stays draft. Later launch-asset +adoption fixes are not qualified by this artifact's hosted result. + +### Post-campaign review corrections (2026-10-26) + +Runner correction `1fb535d12b` makes live dispatch and semantic recovery share +per-call ownership. Two authenticated-transport regressions first duplicated a +callback, then pass with one authoritative result and no redispatch. Already +queued results bypass repeated resolution. All 145 control-plane and 296 +transport tests pass. The wider 621-test run passed 620 and exposed an old pinned +Copilot digest after source-attestation regeneration; its corrected 56-test +profile retry passes. The active attestation changed; historical released +fixtures did not. TypeScript, protocol generation, source identity and module +boundary checks pass. The earlier failed run remains recorded. + +Server correction `e61605e197` recovers completed case/workflow mutations through +their existing adapters and API receipt keys, without HTTP dispatch or another +receipt reservation. Three real-Postgres/authenticated-route regressions fail +before the correction and pass afterward, including missing, changed, pending, +disabled-feature and Stop outcomes with unchanged domain effects. Server +TypeScript passes. The broader run passed 97 tests and failed three held-output +cases because a debug fixture binary remained from the runner-only PR. Its +source and binary capability differed. Rebuilding the server-owned fixture and +rerunning those three cases passes, including zero streaming checkpoint SQL, +zero idle SQL, positive control calibration and empty-controller-file adoption. +Both runs remain recorded. Build the fixture with the owning PR's source before +running targeted native recovery tests after switching branches. + +These corrections follow the exact 73b65d87 hosted pass. That campaign does not +qualify later artifacts or other harnesses. The superseded 54f3c879 preview build +was cancelled after the fresh race and attestation findings, before deployment. +The next candidate requires fresh checks and review. Different-artifact upgrade, +compatible rollback and the ordinary Cloud drain policy remain open gates. + +### Fresh-call ownership and initial-view recovery review (2026-10-26) + +A subsequent review exposed a distinct ownership case: reconciliation could +claim a fresh input during its persistence barrier, then return no saved outcome +and suppress its original callback. Runner correction `e042344de2` reserves live +ownership before that wait. Fresh missing/stale-outcome overlap tests fail before +it and pass afterward, while saved-input concurrent reconciliation still resolves +once. Failed persistence still cannot dispatch. All 146 control-plane and 296 +transport tests pass, with a final 146-test control-plane retry and TypeScript +check after the callback-binding refinement. + +Agent Detail also needs negotiated tail recovery when acknowledgement aborts its +first history read before a cursor exists. Both page variants now retain one +manual history page, then drain their independent recovery cursor without a new +source event. Two 5,000-row regressions fail before the fix, then show the latest +saved tail, retain manual pagination, and add no recurring reads. All 361 affected +UI tests, UI typecheck and token gates pass. + +The 051bc00a candidate build was cancelled before deployment after those findings. +A new candidate must first receive fresh CI and review. Its earlier local browser +journey pass remains attributed to 051bc00a, rather than the subsequent source. +The 051bc00a chat shard also failed an unchanged assertion: a provider-effect +receipt was still processing instead of processed. Its isolated same-job retry +passed, but the original job remains failed. This is recorded as an observed +assertion-timing failure, not a worker-loss incident or a proven production fix. + +### Normal-operation cost correction (2026-10-27) + +Component comparison uses frozen full-stack source `24e4ea15fc9cb52ea3dec8d51ad551596de2c698` +as before, and `51decdb9fb` as after. This compares two versions of the hot-upgrade +stack, not the pre-feature server. The isolated Linux Node and PostgreSQL 17.11 +containers each have two CPUs and 1,536 MiB. Each phase has 300 controller sessions, +10,000 historical tasks and a 65-second fixed offered workload. Checkpoints use +the production encrypted writer, 4 KiB control results and natural 20-second lease +renewals. The machine and Docker storage remain shared with other local work; +this is not dedicated-host or production-capacity evidence. + +PR 1 makes external persistence the sole durable controller store. A local marker +requires external restoration on restart. Standalone consumers retain the current +staged file journal; the sandbox runner journal is unchanged. The initial +file-only correction passed the steady profile but failed both burst latency gates +at 602.5 and 687.8 ms. Those failed phases remain in the evidence. + +PR 2 gives checkpoint transactions a separate pool capped at two connections per +database owner. Requests and lease renewal retain their own pools. No control +barrier is released early. The real-Postgres backlog regression fails with the +shared request pool, then passes with the bounded pool. It also restores the queued +commands and rejects late writes after shutdown. Cost collection observes both +application and checkpoint clients; a positive control still trips its write oracle. + +The final campaign reverses order for each repeat (steady ABBA, burst BAAB): + +| Profile | Offered requests | Before read p99 | After read p99 | Before/after checkpoint transactions per phase | +| --- | --- | --- | --- | --- | +| Steady | 500 reads/s, 100 writes/s | 107.6 / 106.1 ms | 107.8 / 107.2 ms | 3,899 / 3,899 | +| Aligned burst | 100 reads/s, 20 writes/s | 689.4 / 584.1 ms | 103.3 / 105.4 ms | 3,600 / 3,600 | + +All four after phases pass the unchanged absolute and relative read-latency gates, +99% offered-throughput gate, sub-second drain gate, and database CPU/memory gates. +There are no SQL errors, rejected operations, ownership losses or missed database +admission probes. Each steady phase changes 7,798 checkpoint records; each burst +phase changes 7,200. After phases make zero hot-path local journal fsync calls; +before phases make 7,200–7,798. This does not remove PostgreSQL WAL or fsync. + +Client-issued SQL is effectively unchanged: about 909.5 commands/s in steady +phases and 411.9 commands/s in burst phases. Matched steady phases issue +59,119 before and 59,120 after commands. The extra statement initializes the new +client's PostgreSQL type cache. The final burst phase offers one extra normal +request tick, so its raw total is reported separately rather than treated as +checkpoint overhead. Existing hot-upgrade control and renewal writes remain. + +The measured tradeoff is up to two additional active connections per instance, +not per agent. Steady PostgreSQL CPU rises from 10.15–10.31% to 11.85–12.05% of +the two-CPU allocation; burst CPU stays near 4.4–4.8%. PostgreSQL cgroup peak memory +is 526–532 MiB, including cache. Node RSS overlaps across variants and does not +establish a memory saving. Controller delivery can wait behind its bounded pool; +after burst control p99 is 379–475 ms. No production spend saving is inferred. + +Complete phase JSON, SQL shapes, source/image hashes, sampler output and both +failed and passing attempts are retained under +`/tmp/paperclip-controller-persistence-20261027/`. The real-runner 8/512-chunk +checks still observe zero streaming and idle checkpoint SQL. These component +measurements do not qualify a new hosted artifact, a different harness, actual +A→B/B→A replacement, or the ordinary Cloud rollout policy. Those gates remain +open and PR 3 remains draft. Integrated-stack checks and review status belong to +the exact current PR heads, not to the measured source by inference. + +### Integrated persistence review (2026-10-27) + +The stack incorporates master `663fe667d6`. Its runner controller suite passes +164 tests and the rebuilt transport/one-process fixture lane passes 303. The +transcript window merge regression fails before correction `bccff8f35d` and +passes afterward with all 327 affected UI tests. The production-runner browser +journey passes at `ca600c76e3`, including two views, company switching, reconnect, +terminal catch-up, reload and zero recurring healthy transcript reads. It uses a +deterministic harness fixture, not a model service or hosted harness. + +Fresh review adds three server receipt regressions: title/feedback receipts have +their own stores; committed creation/reassignment still need their idempotent +wake suffix; monitor input and retry keys use owner normalization. All seven +cases fail before `0edb3a736e`. Recovery now reuses those owners without entering +an absent receipt's mutation or cancellation path. The new cases, existing +semantic authority cases and real HTTP/API recovery lane pass all 92 tests on +owned PostgreSQL databases after `d0ce42378c` fixes fixture teardown. That fixture +bug had closed a sibling database's clients on the shared host; its independent +wire-protocol regression fails first, then all four teardown tests pass. + +The first HTTP attempt fails embedded-Postgres initialization on this Mac. The +first external-host attempt passes 90 cases and fails its last two after sibling +teardown, plus teardown itself. Both failures remain recorded. The earlier full +Mac suite was interrupted after repeated cluster initialization failures and is +not a green whole-suite result. Qualification units pass 46 cases but fail their +embedded-database setup case locally. These results are separate from fresh CI. + +CI's server shard initially passes 2,046 cases and fails three provider adoption +fixtures that omitted required Daytona ingress authorization/capability. Test +correction `47a159e3ba` represents the authorized target without weakening the +production transport check; all four affected adoption cases pass locally. +Workspace typecheck passes after the receipt and teardown corrections, including +protocol checks and Rust. The sandbox-denied tsx IPC attempt is retained beside +the permitted retry. Current-head CI and review are still required. + +PR 4 correction `9219fc6bc6` preserves a gap notice that could not be sent before +an unrelated view changed its interests. The reproducer fails first; all 19 +server delivery/browser-interest tests then pass. The original congestion +deadline remains active until the browser is told to catch up. This changes +presentation recovery only; it has no checkpoint, execution or accounting effect. + +### Completion-only recovery review (2026-10-27) + +A later Paperclip review identifies completion inputs that would remain pending +when a standalone caller omits its ordinary-tool outcome lookup. Runner +correction `bcc0785946` wires completion recovery independently in both the +Codex transport and shared external driver. The three restart regressions fail +before their respective corrections and pass afterward, covering both handler +startup orders and an external runner with no ordinary-tool callback. Unknown +ordinary tools remain held. Candidate v66 attestation bytes remain frozen; v67 +binds the changed bridge and compatibility declaration without redefining v66. + +The controller/attestation lane passes 230 tests. The rebuilt full +Codex/Dot/Muse run passes 326 cases and times out one existing mid-goal adoption +case at 30 seconds. That case passes its isolated retry, then the complete Codex +transport file passes all 302 cases on unchanged source and binaries. The +one-process fixture separately passes all three cases. Conformance and replay +parity pass. The initial timeout remains evidence; these retries do not establish +its root cause or qualify a hosted harness. + +CI of the preceding successor heads also reports a login-cleanup idle-work +assertion and an unrecoverable runner close in the timeout-attachment fixture. +Both lie outside this completion correction. The unchanged login-cleanup file +passes all 16 cases against a fresh owned database. The close case passes in the +full 302-case local repeat. Those CI failures remain recorded and fresh +current-head CI is required; an isolated pass does not turn an earlier job green. +The runner-only local API-authority check passes 2,027 cases but its 47 real HTTP +cases cannot initialize embedded Postgres on this Mac. The integrated server's +owned-database HTTP path is verified separately. + +A subsequent qualification review catches drift between the Claude continuity +check and its supervisor request: the former requires outage evidence for ACPX, +while the latter creates it only for Codex. The existing Claude Daytona roster +cell fails a selected-cell fault-receipt regression before the correction. +Both consumers now use the same outage policy. The existing Codex and Claude +cells produce valid replacement fault receipts through the supervisor contract; +local, legacy and Stop cells retain their prior behavior. All 108 affected +qualification tests and the runner E2E typecheck pass. This is selection and +receipt-contract proof with injected supervisor operations, not a new hosted +Claude qualification. + +Late review also reproduces a fresh remote claim whose lease is already 30 +seconds expired because cleanup/candidate work consumed the captured batch time. +Claims now calculate their deadline with the database clock in the existing +UPDATE, after the owner row is locked; no query is added. The real-Postgres +regression passes together with three natural renewal/preparation cases, +including the 90-second pool-delay case. Server typecheck passes. These are +deadline and ownership tests, not a simulated deployment qualification. + +Both Agent Detail variants reproduce a disabled history-paging button after +visibility aborts an in-flight manual page read. Resetting that read's loading +state preserves the retained cursor and rejects its late response. Both +regressions fail first; all 28 affected history/visibility tests and UI token +gates pass. CI attempts on the preceding heads also lose several runner services +at the same timestamp; those shutdown/cancellation logs remain in the evidence +and are not reported as assertion failures or passing jobs. + +The shared outage selector subsequently reproduces a dedicated-suite regression: +it excludes `daytona-restart-continuity`, suppressing its strict original-process +checks. The everyday default also requests one second despite its catalog's +125-second declaration. Both suites now derive that default from catalog +metadata; the dedicated suite rejects a shorter override. Explicit short +everyday experiments remain available with their actual duration recorded. The +catalog definition advances to version 10. Both new policy cases fail first; +all 149 relevant qualification tests and the runner E2E typecheck pass. + +A broader catalog invocation additionally finds existing model-account coverage +drift already present in master `663fe667d6`, and a report CLI fixture that did +not produce its output in the sandbox. Those failures remain in the evidence; +the permitted report-catalog +retry passes all 15 cases. They do not alter the continuity acceptance contract. + +Transcript rows now use their run/sequence identity in both Agent Detail +variants. Two regressions first reproduce collisions between database IDs and +synthetic live IDs, then pass without duplicate keys or missing rows. All 30 +history/visibility cases pass. This is a presentation fix and changes neither +canonical logs nor checkpoint writes. + +### Normal-operation hosted comparison and merge update (2026-10-27) + +The hosted release comparison uses Core-only baseline +`5823e9a6646b1624416a1e33f6016037cb88c1a3` and candidate +`445a76563c444188d35ddb4f219ddee8292d4f73`, each with two app CPUs, +1,500,000,000 bytes of app memory, Neon 0.25–1 CU and 2,000 synthetic historical +tasks. The source range includes other Core changes, so it does not isolate the +causal cost of recovery. + +A baseline-only 50-read/10-write-per-second warmup overloaded and was stopped: +1,016 offered requests produced 652 successes and 364 client timeouts. The +bottleneck remains incompletely localized. The original ten-agent ABBA, burst +and viewer acceptance profile has not passed. + +A separately declared API-only pair at 10 reads and two writes per second uses +120 seconds of warmup and 120 seconds of measurement. Both arms complete all +1,440 requests without errors, with no active agents or viewers. Baseline and +candidate raw Postgres transactions are 430.77/s and 430.56/s, including observer +and background work. App CPU is 12.70% and 13.04% of the two-CPU allocation. Read +p99 is 3,120 ms and 1,979 ms; write p99 is 3,359 ms and 2,384 ms. Both fail the +absolute latency gates. Peak cgroup memory is 1,136.5 MiB and 1,180.8 MiB; the +candidate exceeds the declared 80% memory gate at 82.5%. Main-process peak RSS +is 841.3 MiB and 822.4 MiB. A short ordered pair cannot establish leak behavior, +sustained capacity or active-run checkpoint cost. No checkpoint mutations occur +in this API-only workload. Actual database CPU and statement counts are not +available; transaction counts are not statement counts. + +One native Codex/Daytona pilot per arm completes 20 steps, with independent +hash-chain and exclusive-effect file checks. The candidate retains 80 encrypted +checkpoint parts totaling 466,260 bytes. This is not a growth-rate measurement. +Both test companies subsequently pause because gateway model usage is unpriced +to Paperclip. The scaled native comparison remains unexecuted; budget enforcement +is not qualified by these pilots. The immutable report SHA-256 is +`8ccde5cd71fd1076c9046135fdb9a46d7256c863672f63dee3b614b123615bc6`; +the metrics-only evidence archive SHA-256 is +`8c650ba41c0b8f8140a8f654840feb6dc165879068e92f576534a0dc995ee49e`. + +The master reconciliation preserves published Copilot v66 and the prior +hot-upgrade candidate identities, and generates v68 for the combined sources. +The focused runner lane passes 290 cases and typecheck. Its first sandboxed +invocation cannot open loopback listeners; the permitted retry passes. The +server now resolves the encryption key once per checkpoint batch and removes +two unused execution-lease declarations. Six key batching/provider cases, server +typecheck, migration safety and boundary/token checks pass. The checkpoint +migration is regenerated as 0385 after master's newer migrations and retains +preview idempotence. Fresh CI and review are required for the updated heads. +These code checks do not close the remaining different-artifact, hosted rollout, +additional-harness or complete active-agent/viewer qualification gates. + +Fresh review reproduces a public-reader race during external checkpoint writes. +Lifecycle events, result proposals, progress and command outcomes are visible to +transport polling before their write succeeds. Four new reader cases fail before +the correction; all held/rejected-write variants pass afterward. The controller +now publishes committed metadata and command outcomes, retains immutable event +references and reuses unchanged command decoding. Streaming remains independent +of checkpoint writes, and the last command result flushes without waiting for a +next command. The first combined repeat passes 556 cases and fails two overlap +fixtures that wait for the old provisional public view. Those fixtures now wait +for entry into the persistence callback and require the public event view to stay +empty. The full controller repeat then passes all 168 cases. The other four +combined files pass 390 cases. Runner TypeScript build and typecheck pass. + +Fresh qualification review also reproduces three rejected supervisor requests: +the Claude everyday cell and both dedicated continuity cells are selected by the +shared outage policy but rejected by a stale Codex-only guard. The existing file +reader is extracted as the supervisor's request boundary and uses that same +policy. Real request-file tests cover all four supported cells, unsupported cells +and conflicting fault requests. All 93 request/outage/continuity/process-owner +cases and runner E2E typecheck pass. This verifies the supervisor request path, +not a hosted Claude upgrade. + +The first master-reconciled CI attempt loses seven runner services together at +21:40:37 UTC. Their retained logs report shutdown and cancellation without test +assertion failures. The subsequent source corrections require fresh CI; no +successful run is inferred from the interrupted attempt or an older head. + +A further server review finds the expired-owner inventory query truncating its +known active-run IDs at 100. Repeated blocked issue locks can therefore starve +later recoverable runs. The query now returns all IDs from the caller's finite +active inventory in the same scoped query; empty inventory still makes no SQL +request. A 101-run real-Postgres regression retains 100 blocked owners and claims +the original remaining remote execution. This host's Postgres support gate skips +the fixture, so its execution in Linux CI is required before server merge. Server +typecheck and the independent zero-query empty-inventory case pass locally. + +The additional standalone forbidden-import checker reports 11 escaped imports in +existing controlled-eval, CI helper and instruction-test files. All those files +and the checker are unchanged from master09d4ca533e. This pre-existing checker +failure remains recorded; package-cycle boundaries, protocol manifest, module +boundaries and UI token gates pass. Final combined workspace typecheck and +production build pass after the committed-view and supervisor request fixes. + +The runner contract lands in master at `0a08d37aa0` after exact-head CI and a +fresh 5/5 review. Integrating that squash result keeps the server PR's remaining +diff scoped to its own behavior. Master has meanwhile added migrations 0385 and +0386, so the checkpoint schema is regenerated as 0387 with identical idempotent +DDL. The final retained snapshots are 0383–0387. Migration numbering/safety and +server typecheck pass. This ancestry/migration reconciliation requires fresh +checks on the remaining three PRs and adds no hosted qualification evidence. + +### Merge follow-up: fresh claim contention and transcript authority + +Review correction `69e6d415c0` starts fresh execution leases from the database +wall clock in the final claim update, after run locks and lineage reads. The new +Postgres variant observes the actual claim blocked on a run-row lock and requires +a full 60-second deadline after release. It changes no new lease timestamp. The +existing startup-renewal case and server typecheck pass locally. The host skips +the integration suite, including the permitted repeat, so Linux CI must actually +execute both the contention variant and the 101-run expired-owner case. + +The full executor file additionally reports 921 passes and 16 failures: all +16 fail at embedded-Postgres startup, before their database assertions. The local +database limitation remains a failure, not a full executor pass. Source CI is +required. No paid harness test or deployed artifact is involved in these checks. + +Transcript correction `84e9bc45c8` adds the current HTTP company-scope permission +to full WebSocket event/log delivery alongside existing task access. New real-key +Postgres/WebSocket cases cover both negotiation modes, allowed task access under +low-trust policy, blocked native and legacy content, and policy grant/restriction +on the same connection. The first local attempt fails listener sandboxing; its +permitted repeat fails embedded-Postgres startup and skips all five cases. These +new cases require actual passing Linux execution before the transcript merge. +Server typecheck and 27 existing delivery/WebSocket/toast cases pass locally. + +The qualification instructions now state the actual 125-second remote native +Codex/Claude default and the explicit 1000 ms override for a short experiment. +Master then adds its own migration 0387. Reconciliation `96d07efb0f` preserves all +master SQL/journal entries byte-for-byte and appends checkpoint migration 0388 +with the same idempotent DDL. Snapshots 0384–0388, numbering/safety and server +typecheck pass. Each updated candidate still needs fresh CI and exact-head +review. Hosted performance, different-artifact upgrade and rollback gates remain +open; merging source does not qualify them. + +The replacement-smoke driver also now guards database setup. Its real executable +previously exits without a saved report for both an invalid database URL and a +refused loopback connection. Both cases fail before the correction and pass +afterward, alongside the three existing cleanup cases. Reports distinguish +whether a database or wrapper was created, and cleanup touches only registered +resources. Initial typechecking finds a query-result inference error; an explicit +result type corrects it and the E2E typecheck passes. This experiment starts no +server and runs no provider turn; it qualifies failure evidence, not continuity. + +Transcript-head `405bedd189` has a serialized CI shard failure during the +unchanged scheduled-retry route fixture's teardown: a late wakeup references an +agent between deletion of wakeups and agents. The shard reports 283 passes and +one failure. The same serialized shard passes at server-head `d1513f8311` with the +same route test. The failure remains retained; the forthcoming qualification +correction changes the successor head, so the superseded job is not retried. +Fresh current-head CI remains mandatory; this is not a transcript assertion pass +or a waived gate. + +The Linux Runner lane at server-head `d1513f8311` executes and passes the +101-run expired-owner regression (2.803 seconds) and the 63.625-second fresh +preparation renewal case. It reports 114 passes, one failure and five optional +skips: the claim-lock variant fails during setup because the parameterized cases +reuse a uniquely indexed environment name. Correction `227dc068a8` gives each +case its own environment and provider-lease identity. This is a fixture repair, +not a retry of the failed head. Actual claim-lock execution and fresh full CI +remain required; the overall earlier Runner lane was not green. + +Transcript shared-stream recovery tests move unchanged into their own focused +file. The original is 803 lines and the new file 321. The first split leaves one +helper behind and fails one of 39 cases plus typechecking; restoring that fixture +helper yields all 39 passes, UI typecheck and clean token gates. These are hook +fixtures; they do not establish hosted viewer performance. + +Linux general-server shard 1 at transcript-head `405bedd189` executes all five +live-event run-read tests successfully, including both new transcript-permission +variants (2.346 seconds for the file). This is source-bound permission evidence; +that head's other two CI failures remain failures. Master then adds its own +migration 0388. Integration `e5acdc53c6` appends the unchanged idempotent checkpoint +DDL as 0389, preserving every master SQL file and journal entry. The retained +snapshots are 0385–0389; migration checks and server typecheck pass. This integration +requires fresh current-head CI and review and adds no hosted qualification. + +### Review follow-up: shutdown and qualification evidence boundaries + +Server correction `9bb9d3417e` fences checkpoint-pool creation even when shutdown +precedes first use. The real lazy-client regression fails before correction and +passes afterward; all 12 checkpoint-lifetime/options cases and database typecheck +pass without opening a database connection. Existing created pools stay ended, +other owners remain usable, and test doubles retain their fallback behavior. + +The qualification review batch reproduces 14 failures: absent-cache retirement, +missing or mismatched expected artifact pairs, malformed database history, and +invalid outage/manifest configuration without saved evidence. After correction, +all 91 focused outage, observer, executable smoke, cleanup, remote-restart and +artifact cases pass with E2E typecheck. Artifact smoke checks both revision and +frozen tree against the requested A/B/A transition. An absent old cache is safe; +a missing destination parent with a still-present cache remains a failure. +Malformed history fails before checkpoint loading. Seed evidence now precedes +configuration reads, so invalid settings and manifests retain failed reports. + +The reported wait on an already-completed original run is already handled by +`waitForRemoteRestartBoundary` and `pollUntil`'s immediate rejection. Existing +cases cover missing, succeeded, failed, cancelled and timed-out runs in one read, +with no clock advance. This remains an unexercised boundary; it does not authorize +a replacement task or claim active continuity after completion. These checks add +no real harness turns, deployed artifacts, or hosted upgrade acceptance. + + +### Partial outage receipts survive cleanup (October 27 review) + +The shared case-evidence collector now copies sanitized supervisor outage +receipts from the temporary control directory into each case's snapshots before +normal cleanup. This is independent of successful restart acknowledgements. +Startup, HTTP readiness, and authenticated recovery failures therefore preserve +the last recorded phase. Restart request files and temporary receipt writes are +not copied. Ordinary cells with no outage retain their existing evidence path. + +Three filesystem regressions fail before the collector change. They use the +normal replacement and evidence-packaging interfaces, delete the temporary root, +and then read the published receipt. After the change, all 95 focused +qualification cases and E2E typecheck pass. The tests also confirm redaction and +existing log collection. This is evidence-retention proof, not hosted upgrade +continuity or final performance qualification. + + +### Current Linux recovery proof and fourth master integration + +Server source `9bb9d3417e` executes the previously missing contention case in +Linux job 122521800018: fresh preparation passes in 62.808 seconds, genuine +claim-lock contention passes in 1.227 seconds, and the 101-run expired-owner +case passes in 3.828 seconds. That recovery lane finishes with 115 passes and +five optional skips. These are source-attributed correctness receipts, not +hosted upgrade or production load evidence. + +Master `3421366903` adds migration 0389. Integration `9c59bde1a9` preserves +every master SQL file and journal entry, then appends the unchanged idempotent +checkpoint DDL as 0390. Retained snapshots are 0386–0390. Migration checks and +server typecheck pass. The local database owner/lifetime batch initially has +two loopback-listener EPERM failures and 14 passes; with loopback permission +all 16 cases pass. A new exact-head CI and review gate remains required. + + +### Ordinary warm admission with external checkpoints + +The actual external-persistence callback leaves a marker instead of a legacy +full journal. A regression through the public server executor reproduces +`durable_identity_unreadable` on the second completed warm turn. The correction +restores the prior run's opaque checkpoint before identity and reuse checks. It +uses the retained owner's run/issue or a validated native-session checkpoint, +since an original marker can precede later warm identities. Three completed +turns now pass, including selection of the second run's saved checkpoint for +the third turn. The backend and database-load seam are scripted in this proof; +the control plane, persistence callback, identity rotation and filesystem are +real. This does not qualify an underlying harness. + +All 288 provider-wiring cases and server typecheck pass. The initial wider +focused run reports two failures: the scripted runner projection had not been +rotated for the third turn, and a legacy missing-checkpoint denial changed. +The fixture now rotates the runner identity and legacy file-backed roots retain +their existing checks. Initial typecheck also reports nullable fixture session +IDs; the fixture is corrected before the passing typecheck. Real-Postgres cases +are skipped by this Mac's support gate, including the new check that a foreign +normalized session fails before any restored directory is created. Require +current Linux CI execution before merge. No new streaming writes or idle SQL +are added; reconstruction is limited to warm admission with an external marker +or an older local projection. + + +Master `e9e84bb8a1` then adds migration 0390. The fifth reconciliation generates +checkpoint migration 0391 with the identical prior idempotent DDL, preserving +every master SQL file and journal entry. The five snapshots are 0387–0391. +Migration checks, server typecheck and all 13 warm-admission/configuration +variants pass after integration. Current-head CI and review remain required; +prior source checks are not carried forward as a new-head pass. + + +### Per-replacement identity and malformed receipt retention + +The last qualification review reproduces two gaps. An intermediate rollback +step could cite another runner or native session for the same run ID; only the +final step checked the full recovery identity. Every step now compares run, +company, issue, runner and native session against its own pre-fault boundary. +Negative cases change an intermediate receipt while retaining a valid final +step, so final success cannot hide the mismatch. + +A malformed receipt also threw from the shared collector before evidence could +be packaged. Parse failures now produce an explicit invalid-receipt JSON record +with byte count and digest plus sanitized diagnostic text. The parser's message +is not copied because it can quote private bytes. Other receipts and shared +logs still package after temporary-root deletion; malformed evidence cannot +satisfy an outage grade. Two regressions fail before correction (38 other cases +pass). Afterward all 96 focused qualification cases and E2E typecheck pass. +These are qualification-evidence fixes, with no deployment or provider turn. + +### Sixth master integration and completed diagnostic retry + +The one diagnostic retry of server job 122535673841 passed as job +122541891915 at 2026-10-28 00:01:54 UTC: all 20 expiry cases and 3,877 shard +cases pass, with one skip. This is recovery evidence, not proof of the exact +timing race; the original failure remains recorded above. No further unchanged +retry is allowed for that head and job. + +Master `2adcde162e` adds migration 0391. The checkpoint migration is regenerated +as `0392_daffy_guardsmen.sql`, with the same preview-idempotent DDL. Every +master SQL file and journal entry is unchanged. Snapshots 0388–0392 remain. +Migration checks pass after the sandbox denied TSX IPC on the first attempt. +Server typecheck and all 13 warm-owner variants pass after integration. An +initial test command used the wrong path and ran no tests; only the corrected +invocation supplies the 13-case receipt. Fresh exact-head CI and review are +still required. No hosted fixtures, access, budgets or deployment changed. + +All 118 cases in the nine-file qualification support lane and E2E typecheck +pass after integration. The initial sandbox run had 117 passes and one cleanup +failure because process identity inspection was denied. The permitted repeat +passes the real inert process-tree case as well; a subsequent targeted process +check finds no remaining fixture daemon. This is local process evidence, not a +hosted harness or distinct deployed artifact upgrade. + +### Seventh master integration and fresh Linux source receipts + +Before this integration, server `1ff7bb44f4` passes all Actions checks and +Greptile 5/5; its external review is still queued. Qualification `8dc921f773` +and transcripts `49a8783e9b` pass all checks, have exact-head Greptile 5/5 and +no open feedback. These heads cannot substitute for the new integrated heads. + +At server `1ff7bb44f4`, job 122549307215 passes all 938 native-executor cases +(65.497 seconds). Job 122549306469 passes the native recovery lane with 115 +passes and five container-only skips. The fresh preparation, claim-lock and +101-owner cases pass in 63.664, 3.961 and 3.824 seconds. The encrypted restore +session guard is included in that suite. At transcripts `49a8783e9b`, job +122549913643 passes all five run-read permission cases, including same-socket +grant and revocation. The current duration partition places this file in +server shard 11; the old shard-1 receipt does not prove it. + +Master `553a004926` adds migration 0392. Regeneration produces checkpoint +`0393_elite_angel.sql` with unchanged idempotent DDL. Every master SQL file +and journal entry is preserved; snapshots 0389–0393 remain. Migration checks, +server typecheck and all 13 warm-owner variants pass. No runtime recovery +behavior changes in this integration, and fresh exact-head CI and reviews +are required. No hosted fixture, deployment, budget or access changes. + +### Eighth master integration after reviewer continuation + +Master `2cd747d1b6` is integrated in server `051769955e`. It includes the +published Copilot v70 profiles and new upstream Runner fixes. Those profile +files and every published migration remain byte-identical to master. The +generated checkpoint migration is `0394_conscious_whizzer.sql`, with the same +idempotent DDL; snapshots 0390–0394 remain. Migration checks, the Runner vendor +rebuild and replay parity, server typecheck and all 13 warm-owner variants +pass. The warm cases also pass after the vendor rebuild finishes. Master's +report-lock patch keeps its required context whitespace; the feature delta +passes whitespace checks. + +Before integration, server `f2db8a6568` passes all Actions checks and Greptile +5/5. Its workspace setup job 122555672441 fails the 30-second health deadline +while the CLI reports doctor startup, before its server-start step. UI 4,169 +cases and CLI 285 cases pass, with one skip. Identical CLI/server/database +source on qualification `a866619d7b` passes the setup and all 286 CLI cases. +The single diagnostic retry, job 122561057225, passes the setup in 33.048 +seconds and all 286 CLI cases at 00:49:47 UTC on October 28. This retains the +original failure; its exact timing cause remains unproven. No unchanged retry +may repeat. That old head's Linux native-executor suite passes 938 cases; +native recovery passes 115 with five container-only skips, including fresh +preparation (64.160 s), claim-lock (2.693 s) and 101-owner fairness (3.352 s). +These receipts do not qualify the newly integrated source or a hosted harness. + +The existing external server review is resumed after explicit authorization. +It must submit a validated assessment of the current head. New exact-head CI, +Greptile, feedback resolution and normal eligibility remain required. No +deployment, paid harness experiment, hosted fixture or access change is made. + +After integration, all 146 cases across 12 qualification support, report-lock +and workflow suites pass, with E2E typecheck. These use local controlled faults, +configuration subprocesses, filesystem packaging and inert process fixtures. + + +### Project receipt recovery review correction + +Server source `5de803c2cd` fixes recovery of a committed `create_project` result +whose HTTP response was lost. Recovery uses the existing project route and audit +receipt owner with receipt-only semantics, current authorization after the receipt +lock, and unchanged input normalization and fingerprint checks. An absent receipt +returns no result before any mutation. Normal execution adds no persistence work. + +The transport regression first has two failures and five passes. After correction +and fixing the test's incorrect expectation about schema defaults, all seven +transport cases pass and server typecheck passes. The new real-Postgres/API lost +response, concurrent replay, input conflict, company isolation, cancellation and +no-effect cases belong to the existing chat-project-tools suite. All 20 cases in +that file skip on this Mac. Their Linux CI result is required; a local transport +pass does not prove the database or route behavior. + +On prior qualification source `57d3767eac`, jobs 122590643944, 122590643983 and +122590644161 end after runner shutdown/cancellation at 02:18:59–02:19:11 UTC. +They contain no final Vitest result. The host shutdown cause is unproven. No retry +was issued before the necessary server fix superseded that source. These are +retained incomplete/failed attempts, not passing evidence. All new heads require +fresh CI and review. No new hosted deployment, paid harness turn, active-agent +comparison, or A/B upgrade qualification was run. + + +### Server slice merged; qualification retargeted to master + +PR #17439 source `5de803c2cdbdd24d88e411add590edc21954516a` merged into +master as `09ad9adce2cf4742944353a3707a2f908e9bb35e` after all 60 current-head +checks completed cleanly, legacy status passed, Greptile completed at 5/5, the +formal Paperclip review completed at 5/5, and actionable feedback was resolved. +Qualification now targets master. Its diff contains only qualification code and +this evidence document; the server implementation is not repeated in its scope. + +Current-source Linux job 122593415890 executes all 20 project-tool tests +successfully (9.480 seconds), including the committed-project lost-response +regression; the shard has 1,627 passes and three unrelated skips. Job +122593415907 executes all 940 native-session-executor tests successfully (61.431 +seconds); its shard has 2,358 passes and one skip. The earlier macOS skips remain +local evidence limits and do not replace this actual database execution. + +Runner job 122593415475 passes 2,215 cases with one skip in its first lane and +115 native cases with five container-only skips. It includes the encrypted +checkpoint binding guard, fresh remote preparation (63.189 seconds), blocked +claim lock (1.226 seconds), and recovery beyond 101 blocked owners (3.897 +seconds). Its real-Postgres SQL cost sample reports zero idle SQL and zero +streaming SQL for both output volumes. These are component correctness and cost +receipts; they do not establish production capacity or hosted active-agent and +A/B upgrade acceptance. All previously retained failures and hosted limits still +apply. The retargeted qualification head needs fresh CI and reviews. + + +The first post-retarget local batch has four failures and 133 passes across +12 files. All four are in the inherited report-lock test: master #17811 commits +the exact already-approved output lock (`c7f230…b798`), while the fixture insists +HEAD must equal the approved input (`893496…bab4d`). The production helper already +accepts that output unchanged; its policy and patch need no change. The fixture +now checks current HEAD against the two pinned hashes, exercises the real +committed-lock path, and reconstructs the pinned input with the reviewed reverse +patch so its existing unknown-lock, changed-patch and wrong-output failures still +execute. No production validation, dependency, pinned hash or patch changes. +The initial batch selected the one-case process-preservation fixture instead of +the deterministic preservation suite. The corrected batch explicitly includes +both preservation suites and passes all 158 cases across 13 files. The earlier +146-case receipt retains its own source and selection; these counts do not imply +identical test inventories. E2E typecheck also passes after master integration. diff --git a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md index a9e395f53bb..dbb11f6b884 100644 --- a/tests/runner-e2e/EVERYDAY-WORKFLOWS.md +++ b/tests/runner-e2e/EVERYDAY-WORKFLOWS.md @@ -1,5 +1,43 @@ # Everyday Paperclip workflow evals +## Main-server outage qualification + +The remote native Codex and Claude `recover-controller` cells default to SIGKILL, +an observed 125-second offline interval and retirement of the old controller cache. They use +natural ownership timing. The wrapper emits `controller-outage.json`; the grader +requires distinct server processes, confirmed HTTP loss, cache retirement and the +requested interval, in addition to same-run/provider-turn and finalization proof. +Partial fault receipts remain in the attempt's private `control/` directory on +failure. A successful task without the fault receipt cannot qualify. + +Use `PAPERCLIP_RUNNER_E2E_CONTROLLER_STOP_SIGNAL=SIGTERM` for graceful shutdown, +`PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS=1000` for an explicit one-second +experiment, or `PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS=300000` for the +five-minute contract. These settings are recorded; compare only matching configurations. The configured +duration is a minimum hold after confirmed HTTP loss. Shutdown and startup add +to total recovery time; they do not extend the runner's seven-minute grace or the +workflow's attempt deadline. `recover-controller` permits no automatic retries. + +```sh +PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS=300000 pnpm test:e2e:runner -- --id everyday-workflows.runner-codex.daytona.recover-controller --max-automatic-retries 0 +``` + +The credential-free wrapper smoke starts the actual main server and a separate +disposable Postgres, kills/replaces the server, checks that Postgres never +restarted, reads existing data, accepts a new HTTP mutation, and confirms cleanup: + +```sh +node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/controller-replacement-smoke.ts +``` + +It prints a private JSON evidence path. It uses no model and **does not qualify +active-agent continuity, queued launches, or an A-to-B version upgrade**. See the +[qualification plan](../../doc/plans/2026-10-22-hot-upgrade-qualification.md) for the +separate gates and outstanding work. Hosted qualification still requires a valid +pinned Daytona image and the existing credential/budget admission. + +## Workflow suite + This manual suite tests useful work through the production browser, public API, native runner, and normal agent instructions. It complements the tightly scripted runner contract fixtures. It does not add a scheduled or default paid @@ -53,6 +91,12 @@ for 125 seconds after its health listener stops, then start its replacement. Before and after restart each pins the same run, normalized session, Runner identity, PID/start time, and active sandbox lease. The original receipt prefix must remain unchanged. A run that +uses this native Codex remote cell also loses the old controller runtime cache +between stopping and starting the server. The harness moves it into private +evidence outside product lookup paths and requires an acknowledgement of that +fault. Postgres and the instance encryption key survive. This checks deployment +recovery without relying on the old controller journal or local warm-session cache. +A run that ends before either active observation leaves the fault boundary unexercised; it cannot pass from the final ZIP alone. The cell no longer exempts an original Codex remote run that fails with `process_lost` and is replaced. @@ -290,3 +334,24 @@ consumed response cannot extend that wait. `declineGradeEvidence` retain the exact cloned assertion input and original checks. Later final-state or cleanup observations cannot overwrite that input. Earlier campaign grades remain unchanged when the evaluator is corrected. +# Upgrade qualification receipts + +`PAPERCLIP_RUNNER_E2E_UPGRADE_ARTIFACTS` selects frozen installed A/B artifacts +and an alternating A/B/A sequence. The remote controller-recovery workflow +executes every requested replacement on the original active run. If that run +ends before a later fault can be injected, that boundary is unexercised; starting +a replacement task does not qualify it. + +The supervisor reads existing Postgres ownership and Runner-owned checkpoint +evidence. It records HTTP readiness, natural lease takeover, a new authenticated +connection, and ingestion past the stopped controller's saved event cursor. +Final native and workspace receipts establish backlog settlement. No lease +timestamps are edited. Authentication and processing must be observed within +the original 420-second grace, including artifact verification and startup. + +For a credential-free startup/migration experiment, run +`controller-replacement-smoke.ts` with the manifest and an explicit disposable +`PAPERCLIP_TEST_DATABASE_URL`. The driver creates and drops a unique empty +database; it does not change the supplied database. This smoke qualifies zero +agent turns. The [qualification record](../../doc/plans/2026-10-22-hot-upgrade-qualification.md) +is the acceptance ledger for exact artifacts, attempts, and remaining gates. diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index 9ba8341a49c..8ced2fc5b5d 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -958,7 +958,7 @@ The trusted report job applies a reviewed `report-lock.patch` before its frozen install. `report-lock.json` pins the input lock, patch and resulting lock hashes. This updates existing patch identities and the already declared yaml dependency without registry resolution. The worker target and its lock artifact never -control report dependencies. An unknown committed lock fails before install; +control report dependencies. An already-reviewed output lock is used unchanged. An unknown committed lock fails before install; update and review this correction when dependency inputs change. The original failed run remains failed when retained evidence gets a new presentation. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 5ef908f975f..e0ed06db889 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1282,7 +1282,7 @@ stay unchanged. The correction adds exactly yaml@2.9.1 with pinned integrity. Th trusted authorization job rejects unknown locks before paid work. Manual history publication uses the same reviewed correction. Qualification image builds apply it only to a target equal to the workflow revision; other targets keep their own -lock. Unknown lock contents fail before installation. The +lock. An already-reviewed output lock is used unchanged. Unknown lock contents fail before installation. The workflow definition, runner-group permission, and protected-environment deployment still come from the default branch. Do not select the target branch in GitHub's **Use workflow from** control. diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 46960fc0e69..0f49c6e653e 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1516,7 +1516,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ groups: ["native"], profiles: [...everydayProfiles, productionStoryProfile(runnerProfiles.find(profile => profile.id === "runner-opencode")!), ], environments: [localEnvironment, daytonaWarmEnvironment], - tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback"].includes(task.id) + tasks: everydayTasks.map(task => ["hire-reuse", "delegate-feedback", "recover-controller"].includes(task.id) ? { ...task, automaticRetryPolicy: "single_attempt" as const } : task), expectedMatrixSize: 52, excludedExecutionIds: [...everydayProfiles.flatMap(profile => everydayTasks .filter(task => !["build-revise", "delegate-feedback", "recover-controller", "create-skill-studio"].includes(task.id)) @@ -1526,9 +1526,9 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ...(["hire-reuse", "delegate-feedback"].includes(task.id) ? [] : [`everyday-workflows.runner-opencode.local.${task.id}`]), ]), ], - definitionMetadata: { version: 9, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", - remoteControllerDowntimeMs: 125_000, - remoteCodexRestart: "same-active-process-lease-native-prefix-and-workspace-receipt", + definitionMetadata: { version: 10, instructions: "production", grading: "outcome-and-invariants", scheduling: "explicit-only", + controllerDowntimeMs: 125_000, + remoteCodexRestart: "bounded-main-server-outage-receipt-empty-cache-same-active-process-and-finalization", procedureCases: { ids: ["hire-reuse", "delegate-feedback"], maximumAttemptsPerCell: 1, companyAndLeadBudgetCents: 1_000 } }, }, { diff --git a/tests/runner-e2e/controller-database.test.ts b/tests/runner-e2e/controller-database.test.ts new file mode 100644 index 00000000000..72e9360c9a2 --- /dev/null +++ b/tests/runner-e2e/controller-database.test.ts @@ -0,0 +1,79 @@ +import { spawn } from "node:child_process"; +import { once } from "node:events"; +import { mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { createRequire } from "node:module"; +import path from "node:path"; +import { expect, it } from "vitest"; +import { createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "../../packages/db/src/index.js"; +import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js"; +import { prepareControllerDatabase } from "./controller-database.js"; +import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js"; + +it("keeps an unmigrated fixture database alive when its serving client dies", async () => { + const root = await createRunnerE2ETemporaryRoot(); + const configPath = path.join(root, "instance", "config.json"); + let database: Awaited> | undefined; + let db: ReturnType | undefined; + let child: ReturnType | undefined; + try { + await (await prepareRunnerE2EServerConfig({ temporaryRoot: root, configPath, serverPort: 3100 }))!.close(); + database = await prepareControllerDatabase(configPath, root); + const config = JSON.parse(await readFile(configPath, "utf8")); + expect(config.database.mode).toBe("postgres"); + await mkdir(path.dirname(config.secrets.localEncrypted.keyFilePath), { recursive: true }); + await writeFile(config.secrets.localEncrypted.keyFilePath, "fixture-key"); + await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined(); + db = createDb(config.database.connectionString); + const before = await db.$client`select pg_postmaster_start_time()::text as started`; + expect(await db.$client`select tablename from pg_tables where schemaname = 'public'`).toEqual([]); + const require = createRequire(new URL("../../packages/db/package.json", import.meta.url)); + child = spawn(process.execPath, ["--input-type=module", "-e", ` + const { default: postgres } = await import(process.argv[1]); + const sql = postgres(process.argv[2]); + await sql.unsafe('select 1'); + process.stdout.write('connected'); + setInterval(() => {}, 1000); + `, require.resolve("postgres"), config.database.connectionString], { stdio: ["ignore", "pipe", "pipe"] }); + let errors = ""; + child.stderr!.on("data", chunk => { errors += chunk; }); + const connected = await Promise.race([ + once(child.stdout!, "data").then(([chunk]) => String(chunk)), + once(child, "exit").then(() => { throw new Error(`Database client exited before connection: ${errors}`); }), + ]); + expect(connected).toBe("connected"); + const stopped = once(child, "exit"); + child.kill("SIGKILL"); + await stopped; + expect(await db.$client`select pg_postmaster_start_time()::text as started`).toEqual(before); + await db.$client`create table replacement_probe (id integer primary key)`; + await db.$client`insert into replacement_probe values (1)`; + expect(await db.$client`select count(*)::integer as count from replacement_probe`).toEqual([{ count: 1 }]); + await db.$client.end({ timeout: 1 }); db = undefined; + await database.cleanup(); database = undefined; + // The launcher grades isolation after the supervisor has shut down. + await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined(); + } finally { + if (child && child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + await rm(root, { recursive: true, force: true }); + } +}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + +it("rejects database directories outside the fixture and leaves external databases alone", async () => { + const root = await createRunnerE2ETemporaryRoot(); + const configPath = path.join(root, "config.json"); + try { + await (await prepareRunnerE2EServerConfig({ temporaryRoot: root, configPath, serverPort: 3100 }))!.close(); + const config = JSON.parse(await readFile(configPath, "utf8")); + config.database.embeddedPostgresDataDir = path.join(root, "..", "unrelated"); + await writeFile(configPath, JSON.stringify(config)); + await expect(prepareControllerDatabase(configPath, root)).rejects.toThrow("must belong to its qualification fixture"); + config.database.mode = "postgres"; + config.database.connectionString = "postgres://unused@127.0.0.1:1/never-connect"; + const encoded = JSON.stringify(config); + await writeFile(configPath, encoded); + await (await prepareControllerDatabase(configPath, root)).cleanup(); + expect(await readFile(configPath, "utf8")).toBe(encoded); + } finally { await rm(root, { recursive: true, force: true }); } +}); diff --git a/tests/runner-e2e/controller-database.ts b/tests/runner-e2e/controller-database.ts new file mode 100644 index 00000000000..eafd12d0c6e --- /dev/null +++ b/tests/runner-e2e/controller-database.ts @@ -0,0 +1,43 @@ +/** Qualification owns Postgres outside the server process tree. No migrations + * run here: each serving artifact must apply its own schema on startup. */ +import { existsSync } from "node:fs"; +import { readFile, writeFile } from "node:fs/promises"; +import { createRequire } from "node:module"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { createEmbeddedPostgresLogBuffer, ensurePostgresDatabase, loadWithoutEmbeddedPostgresExitHooks, + prepareEmbeddedPostgresNativeRuntime } from "../../packages/db/src/index.js"; +import { recordControllerDatabaseOwnership } from "./instance-isolation.js"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; + +export async function prepareControllerDatabase(configPath: string, temporaryRoot: string) { + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8"))); + if (config.database.mode === "postgres") return { cleanup: async () => {} }; + const directory = path.resolve(config.database.embeddedPostgresDataDir); + const relative = path.relative(path.resolve(temporaryRoot), directory); + if (!relative || relative.startsWith("..") || path.isAbsolute(relative)) throw new Error("Controller database must belong to its qualification fixture"); + const require = createRequire(new URL("../../packages/db/package.json", import.meta.url)); + const module = await loadWithoutEmbeddedPostgresExitHooks(() => import(pathToFileURL(require.resolve("embedded-postgres")).href)); + await prepareEmbeddedPostgresNativeRuntime(); + const logs = createEmbeddedPostgresLogBuffer(); + const database = new module.default({ databaseDir: directory, user: "paperclip", password: "paperclip", + port: config.database.embeddedPostgresPort, persistent: true, + initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"], + onLog: (message: unknown) => logs.append(message), onError: (message: unknown) => logs.append(message) }); + let started = false; + try { + if (!existsSync(path.join(directory, "PG_VERSION"))) await database.initialise(); + await database.start(); + started = true; + const base = `postgres://paperclip:paperclip@127.0.0.1:${config.database.embeddedPostgresPort}`; + await ensurePostgresDatabase(`${base}/postgres`, "paperclip"); + await recordControllerDatabaseOwnership({ configPath, temporaryRoot, dataDirectory: directory, + connectionString: `${base}/paperclip` }); + config.database = { ...config.database, mode: "postgres", connectionString: `${base}/paperclip` }; + await writeFile(configPath, `${JSON.stringify(config, null, 2)}\n`, { mode: 0o600 }); + return { cleanup: async () => { await database.stop(); } }; + } catch (error) { + if (started) await database.stop().catch(() => undefined); + throw new Error(`Qualification database startup failed: ${logs.getRecentLogs().join(" | ")}`, { cause: error }); + } +} diff --git a/tests/runner-e2e/controller-outage.test.ts b/tests/runner-e2e/controller-outage.test.ts new file mode 100644 index 00000000000..d8bb79b9743 --- /dev/null +++ b/tests/runner-e2e/controller-outage.test.ts @@ -0,0 +1,141 @@ +import { mkdtemp, mkdir, writeFile, readFile, rm, lstat } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import { configuredControllerOutage, controllerOutageForExecution, parseControllerOutage, replaceController, retireControllerCache, validControllerOutageReceipt } from "./controller-outage.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; + +function fixture() { + const order: string[] = []; + const receipts: unknown[] = []; + return { order, receipts, input: { + requestId: "calibration", previousPid: 100, outage: { signal: "SIGKILL" as const, holdMs: 20 }, + stop: async (signal: string) => { order.push(signal); }, + waitForUnhealthy: async () => { order.push("http-down"); }, + discardCache: async () => { order.push("empty-cache"); }, + start: vi.fn(async () => { order.push("start"); return { pid: 200 }; }), + waitForReady: async () => { order.push("ready"); }, cancelled: () => false, + record: async (receipt: unknown) => { receipts.push(receipt); }, + } }; +} + +describe("controller outage qualification", () => { + it("keeps original-execution fault evidence enabled for both dedicated continuity cells", async () => { + const executions = selectRunnerExecutions(parseRunnerSelectors(["--suite", "daytona-restart-continuity"])); + expect(executions).toHaveLength(2); + for (const execution of executions) { + const outage = controllerOutageForExecution(execution, {}); + expect(outage).toEqual({ signal: "SIGKILL", holdMs: 125_000 }); + expect(controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" })) + .toEqual({ signal: "SIGKILL", holdMs: 300_000 }); + expect(() => controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" })) + .toThrow("shorter than the dedicated continuity contract"); + } + }); + + it("uses the everyday suite's declared default while retaining explicit short profiles", () => { + const [execution] = selectRunnerExecutions(parseRunnerSelectors([ + "--id", "everyday-workflows.runner-codex.daytona.recover-controller", + ])); + expect(controllerOutageForExecution(execution!, {})).toEqual({ signal: "SIGKILL", holdMs: 125_000 }); + expect(controllerOutageForExecution(execution!, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" })) + .toEqual({ signal: "SIGKILL", holdMs: 1_000 }); + }); + + it.each(["runner-codex", "runner-acpx-claude"])("supplies replacement fault evidence to the %s Daytona restart cell", async profile => { + const [execution] = selectRunnerExecutions(parseRunnerSelectors([ + "--id", `everyday-workflows.${profile}.daytona.recover-controller`, + ])); + expect(execution).toBeDefined(); + const f = fixture(); + const outage = controllerOutageForExecution(execution!, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "20" }); + const receipt = outage ? await replaceController({ ...f.input, outage }) : undefined; + expect(validControllerOutageReceipt(receipt, { signal: "SIGKILL", holdMs: 20 })).toBe(true); + expect(receipt).toMatchObject({ previousPid: 100, replacementPid: 200, cacheDiscarded: true }); + expect(f.order).toEqual(["SIGKILL", "http-down", "empty-cache", "start", "ready"]); + for (const other of [ + { ...execution!, environment: { ...execution!.environment, id: "local" as const } }, + { ...execution!, profile: { ...execution!.profile, generation: "legacy" as const } }, + { ...execution!, task: { ...execution!.task, id: "stop-redirect" } }, + ]) expect(controllerOutageForExecution(other, {})).toBeUndefined(); + }); + it("records an actual bounded outage before starting the successor", async () => { + const f = fixture(); + const receipt = await replaceController(f.input); + expect(f.order).toEqual(["SIGKILL", "http-down", "empty-cache", "start", "ready"]); + expect(receipt.replacementStartedAfterMs! - receipt.stoppedAfterMs!).toBeGreaterThanOrEqual(20); + expect(validControllerOutageReceipt(receipt, f.input.outage)).toBe(true); + expect(validControllerOutageReceipt(receipt, { signal: "SIGTERM", holdMs: 20 })).toBe(false); + expect(validControllerOutageReceipt(receipt, { signal: "SIGKILL", holdMs: 300_000 })).toBe(false); + expect(validControllerOutageReceipt(receipt, f.input.outage, "another-attempt")).toBe(false); + expect(f.receipts[0]).not.toHaveProperty("readyAfterMs"); + expect(f.receipts.at(-1)).toEqual(receipt); + }); + + it.each(["stop", "waitForUnhealthy", "discardCache", "start", "waitForReady"] as const)("retains incomplete evidence when %s fails", async stage => { + const f = fixture(); + f.input[stage] = vi.fn(async () => { throw new Error(stage); }); + await expect(replaceController(f.input)).rejects.toThrow(stage); + expect(f.receipts.length).toBeGreaterThan(0); + expect(f.receipts.every(receipt => !validControllerOutageReceipt(receipt))).toBe(true); + }); + + it("cancels during the offline interval without launching another server", async () => { + const f = fixture(); + f.input.cancelled = () => f.order.includes("empty-cache"); + await expect(replaceController(f.input)).rejects.toThrow("shutdown interrupted"); + expect(f.input.start).not.toHaveBeenCalled(); + }); + + it.each([-1, 300001, 0.5, NaN, Infinity, "300000", null])("rejects invalid duration %s before signaling", holdMs => { + expect(() => parseControllerOutage({ signal: "SIGKILL", holdMs })).toThrow(); + }); + it.each(["", "-1", "1e3", "NaN", " 1000", "300001"])("rejects invalid environment duration %s", value => { + expect(() => configuredControllerOutage({ PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: value })).toThrow(); + }); + it("accepts the five-minute contract without extending the runner grace", () => { + expect(configuredControllerOutage({ PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" })) + .toEqual({ signal: "SIGKILL", holdMs: 300000 }); + expect(validControllerOutageReceipt({ schema: "paperclip.e2e.controller-outage/v1", requestId: "late", + signal: "SIGKILL", holdMs: 300000, previousPid: 1, replacementPid: 2, startedAt: new Date().toISOString(), + stoppedAfterMs: 0, cacheDiscarded: true, replacementStartedAfterMs: 300000, readyAfterMs: 420000 })).toBe(false); + }); +}); + +it.each(["absent", "present"])("retires a %s old controller cache", async state => { + const root = await mkdtemp(join(tmpdir(), "controller-cache-")); + const cache = join(root, "cache"), retired = join(root, "retired"); + try { + if (state === "present") { await mkdir(cache); await writeFile(join(cache, "evidence"), "saved"); } + await retireControllerCache(cache, retired); + await expect(lstat(cache)).rejects.toMatchObject({ code: "ENOENT" }); + if (state === "present") expect(await readFile(join(retired, "evidence"), "utf8")).toBe("saved"); + } finally { await rm(root, { recursive: true, force: true }); } +}); +it("does not mistake a missing retirement parent for an already absent cache", async () => { + const root = await mkdtemp(join(tmpdir(), "controller-cache-")), cache = join(root, "cache"); + try { + await mkdir(cache); await writeFile(join(cache, "evidence"), "saved"); + await expect(retireControllerCache(cache, join(root, "missing", "retired"))).rejects.toMatchObject({ code: "ENOENT" }); + expect(await readFile(join(cache, "evidence"), "utf8")).toBe("saved"); + } finally { await rm(root, { recursive: true, force: true }); } +}); +it.each(["missing", "previous-revision", "previous-tree", "replacement-revision", "replacement-tree", "reversed"])( + "requires the requested upgrade artifact pair when evidence is %s", problem => { + const A = { revision: "a".repeat(40), treeSha256: "a".repeat(64), version: "1", evidence: "development-build" as const }; + const B = { ...A, revision: "b".repeat(40), treeSha256: "b".repeat(64) }; + const receipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "upgrade", previousPid: 1, replacementPid: 2, + startedAt: new Date().toISOString(), signal: "SIGKILL", holdMs: 0, stoppedAfterMs: 0, + replacementStartedAfterMs: 1, readyAfterMs: 2, cacheDiscarded: true, + previousArtifact: { ...A }, replacementArtifact: { ...B } }; + expect(validControllerOutageReceipt(receipt, undefined, "upgrade", { previous: A, replacement: B })).toBe(true); + expect(validControllerOutageReceipt({ ...receipt, previousArtifact: B, replacementArtifact: A }, undefined, + "upgrade", { previous: B, replacement: A })).toBe(true); + if (problem === "missing") Object.assign(receipt, { previousArtifact: undefined, replacementArtifact: undefined }); + if (problem === "previous-revision") receipt.previousArtifact.revision = "c".repeat(40); + if (problem === "previous-tree") receipt.previousArtifact.treeSha256 = "c".repeat(64); + if (problem === "replacement-revision") receipt.replacementArtifact.revision = "c".repeat(40); + if (problem === "replacement-tree") receipt.replacementArtifact.treeSha256 = "c".repeat(64); + if (problem === "reversed") Object.assign(receipt, { previousArtifact: B, replacementArtifact: A }); + expect(validControllerOutageReceipt(receipt, undefined, "upgrade", { previous: A, replacement: B })).toBe(false); + }); diff --git a/tests/runner-e2e/controller-outage.ts b/tests/runner-e2e/controller-outage.ts new file mode 100644 index 00000000000..b50045829ae --- /dev/null +++ b/tests/runner-e2e/controller-outage.ts @@ -0,0 +1,159 @@ +import { lstat, rename } from "node:fs/promises"; +import type { ServingArtifact } from "./upgrade-artifacts.js"; +import type { ControllerRecoveryReceipt } from "./controller-recovery-observer.js"; +import { setTimeout as delay } from "node:timers/promises"; +import type { MatrixExecution } from "./types.js"; + +export interface ControllerOutage { + signal: "SIGTERM" | "SIGKILL"; + holdMs: number; +} + +export interface ControllerOutageReceipt extends ControllerOutage { + schema: "paperclip.e2e.controller-outage/v1"; + requestId: string; + previousPid: number; + replacementPid?: number; + startedAt: string; + stoppedAfterMs?: number; + cacheDiscarded?: boolean; + replacementStartedAfterMs?: number; + readyAfterMs?: number; + previousArtifact?: ServingArtifact; + replacementArtifact?: ServingArtifact; + recovery?: ControllerRecoveryReceipt; + recoveryProgress?: Partial; +} + +export function parseControllerOutage(value: unknown): ControllerOutage { + if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("Invalid controller outage"); + const { signal, holdMs } = value as Record; + if ((signal !== "SIGTERM" && signal !== "SIGKILL") || !Number.isSafeInteger(holdMs) + || Number(holdMs) < 0 || Number(holdMs) > 300_000) throw new Error("Controller outage requires SIGTERM/SIGKILL and 0..300000 integer milliseconds"); + return { signal, holdMs: Number(holdMs) }; +} + +export function configuredControllerOutage(env: NodeJS.ProcessEnv, defaultHoldMs = 1_000): ControllerOutage { + const raw = env.PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS ?? String(defaultHoldMs); + if (!/^\d+$/.test(raw)) throw new Error("Invalid PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS"); + return parseControllerOutage({ signal: env.PAPERCLIP_RUNNER_E2E_CONTROLLER_STOP_SIGNAL ?? "SIGKILL", holdMs: Number(raw) }); +} + +export function controllerOutageForExecution(execution: MatrixExecution, env: NodeJS.ProcessEnv): ControllerOutage | undefined { + if (execution.task.flow !== "everyday_workflow" || execution.task.id !== "recover-controller" + || execution.environment.id !== "daytona" || execution.profile.generation !== "native" + || (execution.profile.provider !== "codex" && execution.profile.provider !== "acpx")) return undefined; + const declaredHoldMs = execution.suite.definitionMetadata?.controllerDowntimeMs; + if (typeof declaredHoldMs !== "number") throw new Error("Controller recovery suite must declare its outage duration"); + const outage = configuredControllerOutage(env, declaredHoldMs); + if (execution.suite.id === "daytona-restart-continuity" && outage.holdMs < declaredHoldMs) { + throw new Error("Controller outage is shorter than the dedicated continuity contract"); + } + return outage; +} + +/** A fault receipt is required in addition to the runner's continuity receipts. */ +export function validControllerOutageReceipt(value: unknown, expected?: ControllerOutage, requestId?: string, + expectedArtifacts?: { previous: Pick; + replacement: Pick }): value is ControllerOutageReceipt { + if (!value || typeof value !== "object" || Array.isArray(value)) return false; + const r = value as ControllerOutageReceipt; + try { parseControllerOutage(r); } catch { return false; } + const durations = [r.stoppedAfterMs, r.replacementStartedAfterMs, r.readyAfterMs]; + return r.schema === "paperclip.e2e.controller-outage/v1" + && typeof r.requestId === "string" && /^[A-Za-z0-9._:-]{1,200}$/.test(r.requestId) + && typeof r.startedAt === "string" && Number.isFinite(Date.parse(r.startedAt)) + && Number.isSafeInteger(r.previousPid) && r.previousPid > 0 + && Number.isSafeInteger(r.replacementPid) && Number(r.replacementPid) > 0 && r.previousPid !== r.replacementPid + && r.cacheDiscarded === true + && durations.every(ms => typeof ms === "number" && Number.isFinite(ms) && ms >= 0) + && r.replacementStartedAfterMs! - r.stoppedAfterMs! >= r.holdMs + && r.readyAfterMs! >= r.replacementStartedAfterMs! + && r.readyAfterMs! < 420_000 + && ((!r.previousArtifact && !r.replacementArtifact) || Boolean(r.previousArtifact && r.replacementArtifact + && [r.previousArtifact, r.replacementArtifact].every(a => /^[a-f0-9]{40}$/.test(a.revision) + && /^[a-f0-9]{64}$/.test(a.treeSha256) && typeof a.version === "string" + && ["released", "development-build"].includes(a.evidence)) + && r.previousArtifact.revision !== r.replacementArtifact.revision + && r.previousArtifact.treeSha256 !== r.replacementArtifact.treeSha256)) + && (!expectedArtifacts || Boolean(r.previousArtifact && r.replacementArtifact + && r.previousArtifact.revision === expectedArtifacts.previous.revision + && r.previousArtifact.treeSha256 === expectedArtifacts.previous.treeSha256 + && r.replacementArtifact.revision === expectedArtifacts.replacement.revision + && r.replacementArtifact.treeSha256 === expectedArtifacts.replacement.treeSha256)) + && (requestId === undefined || r.requestId === requestId) + && (!expected || (r.signal === expected.signal && r.holdMs === expected.holdMs)); +} + +/** Test harness only. Owners supply process cleanup, HTTP probing and cache paths. + * No database mutation, runner command or production recovery hook is involved. + */ +export async function replaceController(input: { + requestId: string; previousPid: number; outage: ControllerOutage; + stop: (signal: ControllerOutage["signal"]) => Promise; + waitForUnhealthy: () => Promise; + discardCache: () => Promise; + start: () => Promise; + waitForReady: (replacement: T) => Promise; + cancelled: () => boolean; + record: (receipt: ControllerOutageReceipt) => Promise; + previousArtifact?: ServingArtifact; + replacementArtifact?: () => ServingArtifact | undefined; + afterStopped?: (receipt: ControllerOutageReceipt) => Promise; + afterReady?: (receipt: ControllerOutageReceipt, record: () => Promise) => Promise; +}): Promise { + const outage = parseControllerOutage(input.outage); + const started = performance.now(); + const receipt: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outage/v1", + requestId: input.requestId, previousPid: input.previousPid, startedAt: new Date().toISOString(), ...outage, + ...(input.previousArtifact ? { previousArtifact: input.previousArtifact } : {}) }; + const check = () => { if (input.cancelled()) throw new Error("Wrapper shutdown interrupted the controller outage"); }; + const record = () => input.record({ ...receipt }); + check(); + await record(); // Failed attempts retain the boundary they actually reached. + await input.stop(outage.signal); + await input.waitForUnhealthy(); + receipt.stoppedAfterMs = performance.now() - started; + await input.afterStopped?.(receipt); + await record(); + await input.discardCache(); + receipt.cacheDiscarded = true; + await record(); + while (performance.now() - started - receipt.stoppedAfterMs < outage.holdMs) { + check(); + await delay(Math.min(100, outage.holdMs - (performance.now() - started - receipt.stoppedAfterMs))); + } + check(); + receipt.replacementStartedAfterMs = performance.now() - started; + await record(); + const replacement = await input.start(); + receipt.replacementPid = replacement.pid; + await record(); + await input.waitForReady(replacement); + check(); + receipt.readyAfterMs = performance.now() - started; + if (input.previousArtifact) { + receipt.replacementArtifact = input.replacementArtifact?.(); + if (!receipt.replacementArtifact) throw new Error("Replacement has no verified serving artifact"); + } + await record(); + if (!validControllerOutageReceipt(receipt, outage, input.requestId)) throw new Error("Controller outage did not produce a qualifying replacement receipt"); + await input.afterReady?.(receipt, record); + return receipt; +} + +/** Retire only the controller-owned cache; this is not Runner state. */ +export async function retireControllerCache(cache: string, retired: string): Promise { + try { await rename(cache, retired); } + catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + // ENOENT can also mean the destination parent is missing. Only absence of + // the old cache satisfies retirement; preserve every other failure. + try { await lstat(cache); } + catch (missing) { + if ((missing as NodeJS.ErrnoException).code === "ENOENT") return; + throw missing; + } + throw error; + } +} diff --git a/tests/runner-e2e/controller-recovery-observer.integration.test.ts b/tests/runner-e2e/controller-recovery-observer.integration.test.ts new file mode 100644 index 00000000000..4009a85fdcf --- /dev/null +++ b/tests/runner-e2e/controller-recovery-observer.integration.test.ts @@ -0,0 +1,84 @@ +import { randomUUID } from "node:crypto"; +import path from "node:path"; +import { execFile } from "node:child_process"; +import { writeFile } from "node:fs/promises"; +import { promisify } from "node:util"; +import { pathToFileURL } from "node:url"; +import { afterEach, expect, it, vi } from "vitest"; +import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, nativeRunFinalizations, + nativeRunnerCheckpoints, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "../../packages/db/src/index.js"; +import { DurablePrpControlPlane } from "../../server/src/vendor/paperclip-runner/index.js"; +import { registerNativeRunnerCheckpoint } from "../../server/src/services/native-runtime/native-runner-checkpoint.js"; +import { recoveryPostgresSupport, startRecoveryPostgres } from "../../server/src/__tests__/helpers/recovery-postgres.js"; +import { createIsolatedTestHome } from "../../server/src/__tests__/helpers/isolated-test-home.js"; +import { readControllerRecoverySnapshot } from "./controller-recovery-observer.js"; +import { buildRunnerE2EProcessEnvironment } from "./harness-env.js"; +import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; + +afterEach(() => vi.unstubAllEnvs()); +(recoveryPostgresSupport.supported ? it : it.skip)("reads real UUID runner ownership and text event identity in one read-only snapshot", async () => { + const database = await startRecoveryPostgres("controller-observer-"); + const db = createDb(database.connectionString); + const home = createIsolatedTestHome(); + vi.stubEnv("PAPERCLIP_HOME", home.home); + vi.stubEnv("PAPERCLIP_INSTANCE_ID", "observer-fixture"); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", undefined); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY_FILE", undefined); + const binding = { companyId: randomUUID(), issueId: randomUUID(), runId: randomUUID() }; + const agentId = randomUUID(), runnerInstanceId = randomUUID(), normalizedSessionId = randomUUID(); + const authority = new DurablePrpControlPlane({ stateDirectory: path.join(home.home, "authority"), + identity: { runnerInstanceId, normalizedSessionId, runId: binding.runId, + environmentLeaseId: randomUUID(), turnId: randomUUID(), itemId: randomUUID() }, + expectedRunnerVersion: "observer-fixture", expectedRunnerDigest: `sha256:${"a".repeat(64)}` }); + try { + await db.insert(companies).values({ id: binding.companyId, name: "Qualification observer" }); + await db.insert(agents).values({ id: agentId, companyId: binding.companyId, name: "Observer fixture", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: binding.issueId, companyId: binding.companyId, title: "Observe original execution" }); + await db.insert(heartbeatRuns).values({ id: binding.runId, companyId: binding.companyId, agentId, + invocationSource: "manual", status: "running", runtimeMode: "native", nativeIssueId: binding.issueId, + runnerInstanceId, nativeSessionId: normalizedSessionId }); + await db.insert(nativeRunFinalizations).values({ ...binding, phase: "observed", controllerBootId: "observer-controller", + controllerGeneration: 1, leaseOwner: "observer-owner", leaseExpiresAt: new Date(Date.now() + 60_000) }); + await registerNativeRunnerCheckpoint({ db, binding, authority, bootId: "observer-controller" }); + await db.insert(heartbeatRunEvents).values([runnerInstanceId, randomUUID()].map((sourceInstanceId, index) => ({ + companyId: binding.companyId, runId: binding.runId, agentId, seq: index + 1, + eventType: "fixture", sourceSeq: index === 0 ? 7 : 99, sourceInstanceId, + }))); + const checkpointBefore = await db.select().from(nativeRunnerCheckpoints); + expect(await readControllerRecoverySnapshot(db, binding)).toMatchObject({ + runId: binding.runId, runnerInstanceId, nativeSessionId: normalizedSessionId, + bootId: "observer-controller", generation: 1, sourceSeq: 7, authenticatedConnectionCount: 0, + }); + expect(await db.select().from(nativeRunnerCheckpoints)).toEqual(checkpointBefore); + // Exercise the actual launcher boundary in a separate process. Neither an + // inline shell key nor a shell key file may replace this fixture's key. + const repositoryRoot = path.resolve(import.meta.dirname, "../.."); + const childScript = path.join(home.home, "observe.mts"); + const wrongKeyFile = path.join(home.home, "ambient-master.key"); + await writeFile(wrongKeyFile, "b".repeat(64), { mode: 0o600 }); + await writeFile(childScript, ` + import { createDb } from ${JSON.stringify(pathToFileURL(path.join(repositoryRoot, "packages/db/src/index.ts")).href)}; + import { readControllerRecoverySnapshot } from ${JSON.stringify(pathToFileURL(path.join(repositoryRoot, "tests/runner-e2e/controller-recovery-observer.ts")).href)}; + const db = createDb(process.env.OBSERVER_TEST_DATABASE_URL); + try { console.log(JSON.stringify(await readControllerRecoverySnapshot(db, ${JSON.stringify(binding)}))); } + finally { await db.$client.end(); } + `, { mode: 0o600 }); + for (const override of [{ PAPERCLIP_SECRETS_MASTER_KEY: "a".repeat(64) }, + { PAPERCLIP_SECRETS_MASTER_KEY_FILE: wrongKeyFile }]) { + const { stdout } = await promisify(execFile)(process.execPath, + runnerE2ETypeScriptProcessArgs(repositoryRoot, childScript), { + env: buildRunnerE2EProcessEnvironment({ ...process.env, ...override, + OBSERVER_TEST_DATABASE_URL: database.connectionString }, []), timeout: 20_000, + }); + expect(JSON.parse(stdout)).toMatchObject({ runId: binding.runId, sourceSeq: 7, + runnerInstanceId, authenticatedConnectionCount: 0 }); + } + expect(await db.select().from(nativeRunnerCheckpoints)).toEqual(checkpointBefore); + await expect(readControllerRecoverySnapshot(db, { ...binding, companyId: randomUUID() })) + .rejects.toThrow("no admitted native owner"); + } finally { + await authority.stop(); + await database.cleanup(); + home.cleanup(); + } +}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); diff --git a/tests/runner-e2e/controller-recovery-observer.test.ts b/tests/runner-e2e/controller-recovery-observer.test.ts new file mode 100644 index 00000000000..19bcdcd9b97 --- /dev/null +++ b/tests/runner-e2e/controller-recovery-observer.test.ts @@ -0,0 +1,46 @@ +import type { Db } from "../../packages/db/src/client.js"; +import { describe, expect, it } from "vitest"; +import { parseRecoveryBinding, readControllerRecoverySnapshot, recoveryObservation, validControllerRecoveryReceipt, type RecoverySnapshot } from "./controller-recovery-observer.js"; +import type { ControllerOutageReceipt } from "./controller-outage.js"; + +const binding = { companyId: "10000000-0000-0000-0000-000000000001", issueId: "10000000-0000-0000-0000-000000000002", runId: "10000000-0000-0000-0000-000000000003" }; +const before: RecoverySnapshot = { runId: binding.runId, runnerInstanceId: "runner", nativeSessionId: "session", + bootId: "old", generation: 1, attempt: 1, leaseExpiresAt: "2026-10-24T00:01:00.000Z", sourceSeq: 42, history: [], + authenticatedConnectionCount: 1, checkpointSourceSeq: 42 }; +const fault: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "restart", previousPid: 1, + replacementPid: 2, signal: "SIGKILL", holdMs: 1000, startedAt: "2026-10-24T00:00:00.000Z", stoppedAfterMs: 1, + replacementStartedAfterMs: 1001, readyAfterMs: 2000, cacheDiscarded: true }; +const after = (): RecoverySnapshot => ({ ...before, bootId: "new", generation: 2, sourceSeq: 47, authenticatedConnectionCount: 2, + history: [{ controllerBootId: "new", controllerGeneration: 2, disposition: "reattach_remote_runner", at: "2026-10-24T00:01:00.001Z" }] }); +describe("read-only controller recovery evidence", () => { + it("requires natural ownership and new original-run receipts after the stopped-process cursor", () => { + expect(recoveryObservation(before, before, fault, binding, 2500)).toBeNull(); + expect(recoveryObservation(before, { ...after(), sourceSeq: 42 }, fault, binding, 61_000)).toBeNull(); + expect(recoveryObservation(before, { ...after(), authenticatedConnectionCount: 1 }, fault, binding, 61_000)).toBeNull(); + const recovery = recoveryObservation(before, after(), fault, binding, 62_000)!; + expect(validControllerRecoveryReceipt({ ...fault, recovery })).toBe(true); + expect(recovery.authenticatedAttachmentObservedAfterMs).toBe(62_000); + expect(recovery.ownershipAcquiredAt).toBe("2026-10-24T00:01:00.001Z"); + expect(validControllerRecoveryReceipt(fault)).toBe(false); + expect(validControllerRecoveryReceipt({ ...fault, recovery: { ...recovery, resumedProcessingObservedAfterMs: 420_000 } })).toBe(false); + }); + it.each(["runId", "runnerInstanceId", "nativeSessionId", "attempt"] as const)("rejects replacement of %s", field => { + expect(() => recoveryObservation(before, { ...after(), [field]: field === "attempt" ? 2 : "replacement" }, fault, binding, 62_000)) + .toThrow("changed the admitted execution"); + }); + it("rejects shortened leases, replacement launches and recovery nudges", () => { + for (const change of [{ at: "2026-10-24T00:00:30.000Z" }, { disposition: "resume_dead_runner" }, { disposition: "bootstrap_incomplete" }]) { + const current = after(); Object.assign(current.history[0]!, change); + expect(() => recoveryObservation(before, current, fault, binding, 62_000)).toThrow(); + } + }); + it("validates the exact fixture binding", () => { + expect(parseRecoveryBinding(binding)).toEqual(binding); + for (const invalid of [null, {}, { ...binding, runId: "other" }, { ...binding, extra: true }]) expect(() => parseRecoveryBinding(invalid)).toThrow(); + }); +}); + +it.each([null, {}, "not-history", [null], [42]])("rejects malformed database recovery history %j before reading checkpoints", async history => { + const db = { transaction: async (work: (tx: unknown) => Promise) => work({ execute: async () => [{ ...before, history }] }) } as unknown as Db; + await expect(readControllerRecoverySnapshot(db, binding)).rejects.toThrow("Recovery history must be an array of records"); +}); diff --git a/tests/runner-e2e/controller-recovery-observer.ts b/tests/runner-e2e/controller-recovery-observer.ts new file mode 100644 index 00000000000..0ee66823a66 --- /dev/null +++ b/tests/runner-e2e/controller-recovery-observer.ts @@ -0,0 +1,169 @@ +/** Read-only qualification observer. This never calls recovery, renews a lease, + * parses checkpoint records, or interprets an underlying harness protocol. */ +import { readFile } from "node:fs/promises"; +import { setTimeout as delay } from "node:timers/promises"; +import { createRequire } from "node:module"; +const { sql }: typeof import("../../packages/db/node_modules/drizzle-orm/index.js") = + createRequire(new URL("../../packages/db/package.json", import.meta.url))("drizzle-orm"); +import { createDb, type Db } from "../../packages/db/src/index.js"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; +import { loadNativeRunnerCheckpoint } from "../../server/src/services/native-runtime/native-runner-checkpoint.js"; +import type { ControllerOutageReceipt } from "./controller-outage.js"; + +export interface RecoveryBinding { companyId: string; issueId: string; runId: string } +export interface RecoverySnapshot { + runId: string; runnerInstanceId: string; nativeSessionId: string; + bootId: string; generation: number; attempt: number; leaseExpiresAt: string | null; + sourceSeq: number; history: Array>; + authenticatedConnectionCount: number; + checkpointSourceSeq: number; +} +export interface ControllerRecoveryReceipt { + binding: RecoveryBinding; + runnerInstanceId: string; + nativeSessionId: string; + previousGeneration: number; + generation: number; + previousBootId: string; + bootId: string; + previousLeaseExpiresAt: string | null; + ownershipAcquiredAt: string; + ownershipObservedAfterMs: number; + authenticatedAttachmentObservedAfterMs: number; + resumedProcessingObservedAfterMs: number; + runnerSourceSeqBefore: number; + runnerSourceSeqAfter: number; + previousAuthenticatedConnectionCount: number; + authenticatedConnectionCount: number; + attachmentEvidence: "runner_checkpoint_authenticated_connection"; +} +export function parseRecoveryBinding(value: unknown): RecoveryBinding { + if (!value || typeof value !== "object" || Array.isArray(value)) throw new Error("Invalid controller recovery binding"); + const row = value as Record; + if (Object.keys(row).length !== 3 || !["companyId", "issueId", "runId"].every(key => + typeof row[key] === "string" && /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/i.test(row[key] as string))) { + throw new Error("Invalid controller recovery binding"); + } + return row as unknown as RecoveryBinding; +} + +export function recoveryObservation(before: RecoverySnapshot, after: RecoverySnapshot, fault: ControllerOutageReceipt, + binding: RecoveryBinding, observedAfterMs: number): ControllerRecoveryReceipt | null { + if (before.runId !== binding.runId || after.runId !== before.runId || after.runnerInstanceId !== before.runnerInstanceId + || after.nativeSessionId !== before.nativeSessionId || after.attempt !== before.attempt) { + throw new Error("Controller replacement changed the admitted execution"); + } + if (after.bootId === before.bootId || after.generation <= before.generation) return null; + const claim = after.history.findLast(entry => entry.controllerBootId === after.bootId + && entry.controllerGeneration === after.generation); + if (!claim || claim.disposition !== "reattach_remote_runner" || typeof claim.at !== "string") { + throw new Error("Replacement did not naturally adopt the original remote runner"); + } + const claimedAt = Date.parse(claim.at); + if (!Number.isFinite(claimedAt) || claimedAt < Date.parse(fault.startedAt) + || (fault.signal === "SIGKILL" && before.leaseExpiresAt && claimedAt < Date.parse(before.leaseExpiresAt))) { + throw new Error("Replacement bypassed natural controller lease expiry"); + } + if (after.authenticatedConnectionCount <= before.authenticatedConnectionCount || after.sourceSeq <= before.sourceSeq) return null; + // The runner-owned projection counts successful authentications. New canonical + // receipts separately prove ingestion beyond both the saved checkpoint and DB + // cursor. Replaying an old saved event alone cannot pass either boundary. + return { binding, runnerInstanceId: before.runnerInstanceId, nativeSessionId: before.nativeSessionId, + previousGeneration: before.generation, generation: after.generation, previousBootId: before.bootId, + bootId: after.bootId, previousLeaseExpiresAt: before.leaseExpiresAt, ownershipAcquiredAt: claim.at, + ownershipObservedAfterMs: observedAfterMs, authenticatedAttachmentObservedAfterMs: observedAfterMs, + resumedProcessingObservedAfterMs: observedAfterMs, runnerSourceSeqBefore: before.sourceSeq, + runnerSourceSeqAfter: after.sourceSeq, previousAuthenticatedConnectionCount: before.authenticatedConnectionCount, + authenticatedConnectionCount: after.authenticatedConnectionCount, attachmentEvidence: "runner_checkpoint_authenticated_connection" }; +} + +export function validControllerRecoveryReceipt(fault: ControllerOutageReceipt | undefined): boolean { + const r = fault?.recovery; + return Boolean(r && r.bootId !== r.previousBootId && r.generation > r.previousGeneration + && r.runnerSourceSeqAfter > r.runnerSourceSeqBefore && r.authenticatedConnectionCount > r.previousAuthenticatedConnectionCount + && r.attachmentEvidence === "runner_checkpoint_authenticated_connection" + && [r.ownershipObservedAfterMs, r.authenticatedAttachmentObservedAfterMs, r.resumedProcessingObservedAfterMs] + .every(ms => Number.isFinite(ms) && ms >= fault!.readyAfterMs! && ms < 420_000)); +} + +export async function readControllerRecoverySnapshot(db: Db, binding: RecoveryBinding): Promise { + return db.transaction(async tx => { + const [row] = await tx.execute(sql`select r.id as "runId", r.runner_instance_id as "runnerInstanceId", + r.native_session_id as "nativeSessionId", f.controller_boot_id as "bootId", f.controller_generation as generation, + f.attempt, f.lease_expires_at::text as "leaseExpiresAt", f.recovery_history as history, + coalesce((select max(e.source_seq) from heartbeat_run_events e where e.run_id = r.id + and e.company_id = r.company_id and e.source_instance_id = r.runner_instance_id::text), 0)::float8 as "sourceSeq" + from heartbeat_runs r join native_run_finalizations f on f.run_id = r.id and f.company_id = r.company_id + where r.id = ${binding.runId} and r.company_id = ${binding.companyId} and r.native_issue_id = ${binding.issueId}`); + if (!row?.runnerInstanceId || !row.nativeSessionId || !row.bootId) throw new Error("Recovery binding has no admitted native owner"); + if (!Array.isArray(row.history) || row.history.some(entry => entry === null + || typeof entry !== "object" || Array.isArray(entry))) { + throw new Error("Recovery history must be an array of records"); + } + // Use the existing server loader and Runner's public diagnostic projection. + // The qualification code never reads encrypted material or private formats. + const checkpoint = await loadNativeRunnerCheckpoint(tx as unknown as Db, binding); + const evidence = checkpoint?.recoveryEvidence; + if (!evidence || evidence.runnerInstanceId !== row.runnerInstanceId || evidence.normalizedSessionId !== row.nativeSessionId) { + throw new Error("Recovery checkpoint has no matching runner evidence"); + } + return { ...row, authenticatedConnectionCount: evidence.authenticatedConnectionCount, + checkpointSourceSeq: evidence.highestRecordedSourceSeq } as RecoverySnapshot; + }, { isolationLevel: "repeatable read", accessMode: "read only" }); +} + +export async function createControllerRecoveryObserver(configPath: string, binding: RecoveryBinding) { + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8"))); + const url = config.database.mode === "postgres" ? config.database.connectionString + : `postgres://paperclip:paperclip@127.0.0.1:${config.database.embeddedPostgresPort}/paperclip`; + if (!url) throw new Error("Qualification database is unavailable"); + const db = createDb(url); + let before: RecoverySnapshot | undefined; + const read = () => readControllerRecoverySnapshot(db, binding); + return { + // Called only after the old server is confirmed dead. This also excludes + // receipts committed while it was stopping from replacement evidence. + stopped: async (fault: ControllerOutageReceipt) => { + before = await read(); + before.sourceSeq = Math.max(before.sourceSeq, before.checkpointSourceSeq); + fault.recoveryProgress = { binding, previousBootId: before.bootId, previousGeneration: before.generation, + previousLeaseExpiresAt: before.leaseExpiresAt, runnerSourceSeqBefore: before.sourceSeq, + previousAuthenticatedConnectionCount: before.authenticatedConnectionCount, + runnerInstanceId: before.runnerInstanceId, nativeSessionId: before.nativeSessionId }; + }, + recovered: async (fault: ControllerOutageReceipt, record: () => Promise, cancelled: () => boolean) => { + if (!before) throw new Error("Missing stopped-controller observation"); + const deadline = Date.parse(fault.startedAt) + 420_000; + let ownershipObservedAfterMs: number | undefined; + let authenticatedObservedAfterMs: number | undefined; + while (Date.now() < deadline && !cancelled()) { + const current = await read(); + const elapsed = Date.now() - Date.parse(fault.startedAt); + if (current.bootId !== before.bootId && current.generation > before.generation && ownershipObservedAfterMs === undefined) { + ownershipObservedAfterMs = elapsed; + fault.recoveryProgress = { ...fault.recoveryProgress, bootId: current.bootId, generation: current.generation, + ownershipObservedAfterMs }; + await record(); + } + if (ownershipObservedAfterMs !== undefined && current.authenticatedConnectionCount > before.authenticatedConnectionCount + && authenticatedObservedAfterMs === undefined) { + authenticatedObservedAfterMs = elapsed; + fault.recoveryProgress = { ...fault.recoveryProgress, authenticatedConnectionCount: current.authenticatedConnectionCount, + authenticatedAttachmentObservedAfterMs: elapsed }; + await record(); + } + const recovered = recoveryObservation(before, current, fault, binding, elapsed); + if (recovered) { + recovered.ownershipObservedAfterMs = ownershipObservedAfterMs!; + recovered.authenticatedAttachmentObservedAfterMs = authenticatedObservedAfterMs!; + fault.recovery = recovered; + await record(); + return; + } + await delay(250); + } + throw new Error("Original runner did not reattach and resume ingestion within its original 420-second grace"); + }, + close: () => db.$client.end(), + }; +} diff --git a/tests/runner-e2e/controller-replacement-smoke.test.ts b/tests/runner-e2e/controller-replacement-smoke.test.ts new file mode 100644 index 00000000000..fb0a423b699 --- /dev/null +++ b/tests/runner-e2e/controller-replacement-smoke.test.ts @@ -0,0 +1,51 @@ +import { tmpdir } from "node:os"; +import { execFile } from "node:child_process"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { promisify } from "node:util"; +import { expect, it } from "vitest"; +import { upgradeArtifactKey } from "./upgrade-artifacts.js"; +import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; + +const execute = promisify(execFile); +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); + +it.each(["invalid-database-url", "postgres://fixture:fixture@127.0.0.1:1/fixture?connect_timeout=1", "invalid-outage", "invalid-manifest"])( + "retains failed replacement evidence when setup rejects %s", async connectionString => { + const configRoot = await mkdtemp(path.join(tmpdir(), "replacement-config-")); + const manifest = path.join(configRoot, "malformed.json"); + await writeFile(manifest, "{broken JSON"); + const fixtureEnv: NodeJS.ProcessEnv = { ...process.env, PAPERCLIP_TEST_DATABASE_URL: connectionString, + [upgradeArtifactKey]: undefined }; + if (connectionString === "invalid-outage") fixtureEnv.PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS = "invalid"; + if (connectionString === "invalid-manifest") fixtureEnv[upgradeArtifactKey] = manifest; + let output = "", exitCode: unknown; + try { + await execute(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, + path.join(import.meta.dirname, "controller-replacement-smoke.ts")), { + cwd: repositoryRoot, timeout: 20_000, + env: fixtureEnv, + }); + } catch (error) { + const failure = error as Error & { stdout: string; code: unknown }; + output = failure.stdout; exitCode = failure.code; + } + await rm(configRoot, { recursive: true, force: true }); + const started = output.split("\n").filter(line => line.startsWith("{")).map(line => JSON.parse(line)) + .find(value => value.started === true); + expect(started?.evidence).toEqual(expect.any(String)); + const temporaryRoot = path.dirname(started.evidence); + expect(path.basename(temporaryRoot)).toMatch(/^paperclip-runner-e2e-/); + try { + expect(exitCode).toBe(1); + const report = JSON.parse(await readFile(started.evidence, "utf8")); + expect(report).toMatchObject({ schema: "paperclip.e2e.controller-replacement-smoke/v1", + passed: false, providerTurns: 0, activeAgentContinuityQualified: false, + databaseCreated: false, wrapperStarted: false, cleanupConfirmed: true, + error: expect.any(String) }); + if (connectionString === "invalid-outage") expect(report.error).toContain("Invalid PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS"); + if (connectionString === "invalid-manifest") expect(report.error).toMatch(/JSON/); + expect(report.replacements).toBeUndefined(); + expect(output).toContain('"passed":false'); + } finally { await rm(temporaryRoot, { recursive: true, force: true }); } + }); diff --git a/tests/runner-e2e/controller-replacement-smoke.ts b/tests/runner-e2e/controller-replacement-smoke.ts new file mode 100644 index 00000000000..b937dcc3850 --- /dev/null +++ b/tests/runner-e2e/controller-replacement-smoke.ts @@ -0,0 +1,171 @@ +import { readUpgradeArtifacts, upgradeArtifactKey } from "./upgrade-artifacts.js"; +/** Credential-free main-server replacement. This proves the wrapper/startup/DB + * boundary; only the hosted workflow can qualify an active Daytona agent. + * Run: node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/controller-replacement-smoke.ts + */ +import assert from "node:assert/strict"; +import { execFileSync, spawn } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { createWriteStream } from "node:fs"; +import { mkdir, readFile, rename, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { createDb, startEmbeddedPostgresTestDatabase } from "../../packages/db/src/index.js"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; +import { configuredControllerOutage, validControllerOutageReceipt, type ControllerOutage } from "./controller-outage.js"; +import { runnerE2EServerControlPaths } from "./harness-env.js"; +import { reserveRunnerE2EServerPort } from "./ports.js"; +import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js"; +import { createRunnerE2EServerStopper } from "./server-stop.js"; +import { cleanupControllerSmokeDatabase } from "./controller-smoke-cleanup.js"; +import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; + +let outage: ControllerOutage; +let upgrades: Awaited> = null; +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); +const temporaryRoot = await createRunnerE2ETemporaryRoot(); +const home = path.join(temporaryRoot, "home"); +const configPath = path.join(home, "instances", "default", "config.json"); +const control = runnerE2EServerControlPaths(temporaryRoot); +const reportPath = path.join(temporaryRoot, "replacement-evidence.json"); +console.info(JSON.stringify({ started: true, evidence: reportPath })); +const report: Record = { schema: "paperclip.e2e.controller-replacement-smoke/v1", + providerTurns: 0, activeAgentContinuityQualified: false, cleanupConfirmed: false, + databaseCreated: false, wrapperStarted: false }; +// Register resources only after creation so setup failures still have evidence +// and cleanup never assumes a database or client exists. +let database: { connectionString: string; cleanup(): Promise } | undefined; +let db: ReturnType | undefined; +const log = createWriteStream(path.join(temporaryRoot, "wrapper.log"), { mode: 0o600 }); +const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: 35_000, forcedTimeoutMs: 5_000, + hasSpawnError: () => false, markExpectedStop: () => {}, log: message => log.write(message) }); +let wrapper: ReturnType | undefined; +let failure: unknown; +let ownedPostgresProcessId: number | null | undefined = process.env.PAPERCLIP_TEST_DATABASE_URL ? null : undefined; +report.databaseCleanupScope = ownedPostgresProcessId === null ? "owned_database" : "owned_cluster_process"; +try { + report.sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8" }).trim(); + report.sourceDirty = execFileSync("git", ["status", "--porcelain"], { cwd: repositoryRoot, encoding: "utf8" }).trim().length > 0; + outage = configuredControllerOutage(process.env); + report.outage = outage; + upgrades = await readUpgradeArtifacts(process.env[upgradeArtifactKey]); + const port = await reserveRunnerE2EServerPort(); + // The database belongs to this driver, never to either server process tree. + const external = process.env.PAPERCLIP_TEST_DATABASE_URL; + if (!external) { + if (upgrades) throw new Error("Artifact migration proof requires PAPERCLIP_TEST_DATABASE_URL; the embedded test helper pre-migrates its database"); + database = await startEmbeddedPostgresTestDatabase("controller-replacement-smoke-"); + } else { + // Create an empty database. Artifact A applies its own migrations before B. + // Record ownership before closing the admin client, which can itself fail. + const name = `upgrade_${randomUUID().replaceAll("-", "")}`; + const url = new URL(external); url.pathname = `/${name}`; + const admin = createDb(external); + try { + await admin.$client.unsafe(`create database "${name}"`); + database = { connectionString: url.toString(), cleanup: async () => { + const cleanup = createDb(external); + try { await cleanup.$client.unsafe(`drop database "${name}"`); } + finally { await cleanup.$client.end(); } + } }; + report.databaseCreated = true; + } finally { await admin.$client.end(); } + } + report.databaseCreated = true; + db = createDb(database.connectionString); + if (ownedPostgresProcessId !== null) { + const [owner] = await db.$client`select split_part(pg_read_file('postmaster.pid', 0, 64), chr(10), 1)::int as pid`; + const pid = Number(owner?.pid); + if (!Number.isSafeInteger(pid) || pid <= 0) throw new Error("owned_postgres_identity_unavailable"); + ownedPostgresProcessId = pid; + } + const reservation = await prepareRunnerE2EServerConfig({ temporaryRoot, configPath, serverPort: port }); + await reservation?.close(); + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8"))); + config.database = { ...config.database, mode: "postgres", connectionString: database.connectionString }; + await writeFile(configPath, JSON.stringify(config), { mode: 0o600 }); + const source = { PATH: process.env.PATH, HOME: home, TMPDIR: process.env.TMPDIR, + ...(upgrades ? { ...upgrades.artifacts.A.pins, [upgradeArtifactKey]: process.env[upgradeArtifactKey] } : {}), + NODE_ENV: "test", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, PAPERCLIP_INSTANCE_ID: "default", + PAPERCLIP_AGENT_JWT_SECRET: randomUUID(), PAPERCLIP_DECISION_SIGNING_SECRET: randomUUID(), + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomUUID(), BETTER_AUTH_SECRET: randomUUID(), + PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(temporaryRoot, "server.log"), + PAPERCLIP_RUNNER_E2E_TEMP_ROOT: temporaryRoot, PAPERCLIP_RUNNER_E2E_PORT: String(port), + PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify(["everyday-workflows.runner-codex.daytona.recover-controller"]) }; + wrapper = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(import.meta.dirname, "server.ts")), + { cwd: repositoryRoot, env: source, detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"] }); + wrapper.once("spawn", () => { report.wrapperStarted = true; }); + wrapper.stdout!.pipe(log, { end: false }); wrapper.stderr!.pipe(log, { end: false }); + const stopped = new Promise((resolve, reject) => { wrapper!.once("error", reject); wrapper!.once("close", () => resolve()); }); + void stopped.catch(() => {}); + await stopper.watch(wrapper); + async function poll(load: () => Promise, accept: (value: T) => boolean, timeoutMs: number): Promise { + const deadline = Date.now() + timeoutMs; + while (Date.now() < deadline) { + if (wrapper!.exitCode !== null || wrapper!.signalCode !== null) throw new Error("Server supervisor exited; inspect private logs"); + const value = await load(); + if (accept(value)) return value; + await delay(200); + } + throw new Error("Main-server replacement smoke timed out; inspect private logs"); + } + const base = `http://127.0.0.1:${port}`; + async function api(route: string, body?: object) { + const response = await fetch(`${base}/api${route}`, { method: body ? "POST" : "GET", + headers: { "content-type": "application/json", origin: base }, body: body ? JSON.stringify(body) : undefined, + signal: AbortSignal.timeout(5_000) }); + if (!response.ok) throw new Error(`Smoke API ${route} returned ${response.status}`); + return response.json() as Promise>; + } + await poll(() => api("/health").catch(() => null), health => health?.status === "ok", 180_000); + const company = await api("/companies", { name: `Restart smoke ${randomUUID()}` }); + const [before] = await db.$client<{ started: string }[]>`select pg_postmaster_start_time()::text as started`; + const receipts: unknown[] = []; + for (let index = 1; index < (upgrades?.sequence.length ?? 2); index += 1) { + // Empty fixture cache exercises retirement without pretending a runner exists. + const cache = path.join(home, "instances", "default", "runtime", "paperclip-runner"); + await mkdir(cache, { recursive: true }); + const requestId = `smoke-${index}-${randomUUID()}`; + await writeFile(`${control.restartRequestPath}.tmp`, JSON.stringify({ requestId, discardControllerCache: true, controllerOutage: outage }), { mode: 0o600 }); + await rename(`${control.restartRequestPath}.tmp`, control.restartRequestPath); + const ack = await poll(async () => { + const value = await readFile(control.restartAcknowledgementPath, "utf8").then(text => JSON.parse(text) as Record) + .catch(error => { if (error.code === "ENOENT") return null; throw error; }); + if (value?.requestId === requestId && value.status === "failed") throw new Error(String(value.message)); + return value; + }, value => value?.requestId === requestId && value.status === "ready", outage.holdMs + 180_000); + assert(validControllerOutageReceipt(ack!.controllerOutage, outage, requestId, upgrades ? { + previous: upgrades.artifacts[upgrades.sequence[index - 1]!], + replacement: upgrades.artifacts[upgrades.sequence[index]!], + } : undefined), "Invalid replacement fault or requested artifact receipt"); + receipts.push(ack!.controllerOutage); + report.replacements = receipts; + await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 }); + assert.equal((await api(`/companies/${company.id}`)).id, company.id); + const after: { started: string } = (await db.$client<{ started: string }[]>`select pg_postmaster_start_time()::text as started`)[0]!; + assert.equal(after.started, before.started, "Postgres must remain alive across the web-server replacement"); + await api("/companies", { name: `New admission after restart ${randomUUID()}` }); + } + Object.assign(report, { postgresStayedAlive: true, existingDataReadable: true, newMutationAccepted: true, passed: true }); + await stopper.stopAll(); + await stopped; +} catch (error) { + failure = error; + Object.assign(report, { passed: false, error: error instanceof Error ? error.message : String(error) }); +} finally { + const cleanupErrors: string[] = []; + const cleanups: Array<() => Promise> = []; + if (wrapper) cleanups.push(() => stopper.stopAll()); + if (db) { const client = db.$client; cleanups.push(() => client.end()); } + if (database) { const owned = database; cleanups.push(() => cleanupControllerSmokeDatabase(owned, ownedPostgresProcessId)); } + for (const cleanup of cleanups) { + try { await cleanup(); } + catch (error) { failure ??= error; cleanupErrors.push(error instanceof Error ? error.message : String(error)); } + } + report.cleanupConfirmed = cleanupErrors.length === 0; + if (cleanupErrors.length) { report.cleanupErrors = cleanupErrors; report.passed = false; } + await new Promise(resolve => log.end(resolve)); + await writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 }); + console.info(JSON.stringify({ passed: report.passed, evidence: reportPath, cleanupConfirmed: report.cleanupConfirmed })); +} +if (failure) throw failure; diff --git a/tests/runner-e2e/controller-smoke-cleanup.test.ts b/tests/runner-e2e/controller-smoke-cleanup.test.ts new file mode 100644 index 00000000000..ed95e117d0a --- /dev/null +++ b/tests/runner-e2e/controller-smoke-cleanup.test.ts @@ -0,0 +1,34 @@ +import { spawn } from "node:child_process"; +import { once } from "node:events"; +import { expect, it, vi } from "vitest"; +import { cleanupControllerSmokeDatabase } from "./controller-smoke-cleanup.js"; + +it("does not certify cleanup while the owned database process is still alive", async () => { + const child = spawn(process.execPath, ["-e", "setInterval(() => {}, 1000)"], { stdio: "ignore" }); + await once(child, "spawn"); + const cleanup = vi.fn(async () => {}); + try { + await expect(cleanupControllerSmokeDatabase({ cleanup }, child.pid!, 30)).rejects.toThrow("owned_postgres_exit_unconfirmed"); + expect(cleanup).toHaveBeenCalledTimes(1); + expect(() => process.kill(child.pid!, 0)).not.toThrow(); + const exited = once(child, "exit"); child.kill("SIGTERM"); await exited; + await expect(cleanupControllerSmokeDatabase({ cleanup }, child.pid!, 30)).resolves.toBeUndefined(); + } finally { + if (child.exitCode === null && child.signalCode === null) { + const exited = once(child, "exit"); child.kill("SIGTERM"); await exited; + } + } +}); + +it("preserves cleanup failures and refuses an unknown embedded process identity", async () => { + await expect(cleanupControllerSmokeDatabase({ cleanup: async () => { throw new Error("drop_failed"); } }, null)) + .rejects.toThrow("drop_failed"); + await expect(cleanupControllerSmokeDatabase({ cleanup: async () => {} }, undefined)) + .rejects.toThrow("owned_postgres_identity_unavailable"); +}); + +it("accepts confirmed external database deletion without stopping its shared cluster", async () => { + const cleanup = vi.fn(async () => {}); + await expect(cleanupControllerSmokeDatabase({ cleanup }, null)).resolves.toBeUndefined(); + expect(cleanup).toHaveBeenCalledTimes(1); +}); diff --git a/tests/runner-e2e/controller-smoke-cleanup.ts b/tests/runner-e2e/controller-smoke-cleanup.ts new file mode 100644 index 00000000000..3e0bde12490 --- /dev/null +++ b/tests/runner-e2e/controller-smoke-cleanup.ts @@ -0,0 +1,25 @@ +import { setTimeout as delay } from "node:timers/promises"; + +export async function cleanupControllerSmokeDatabase( + database: { cleanup(): Promise }, + ownedProcessId: number | null | undefined, + timeoutMs = 10_000, +): Promise { + await database.cleanup(); + // An external fixture owns a database, not its shared Postgres process. Its + // cleanup awaits DROP DATABASE. Embedded cleanup can return before exit. + if (ownedProcessId === null) return; + if (typeof ownedProcessId !== "number" || !Number.isSafeInteger(ownedProcessId) || ownedProcessId <= 0) { + throw new Error("owned_postgres_identity_unavailable"); + } + const deadline = performance.now() + timeoutMs; + while (true) { + try { process.kill(ownedProcessId, 0); } + catch (error) { + if ((error as NodeJS.ErrnoException).code === "ESRCH") return; + throw error; + } + if (performance.now() >= deadline) throw new Error("owned_postgres_exit_unconfirmed"); + await delay(Math.min(25, deadline - performance.now())); + } +} diff --git a/tests/runner-e2e/daytona-image-content.ts b/tests/runner-e2e/daytona-image-content.ts index 917bcd3d592..0dc464750fc 100644 --- a/tests/runner-e2e/daytona-image-content.ts +++ b/tests/runner-e2e/daytona-image-content.ts @@ -63,11 +63,15 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [ "packages/paperclip-runner/test/fixtures/copilot-profile-v52-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v54-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v55-identity.json", + "packages/paperclip-runner/test/fixtures/copilot-profile-v56-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v57-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json", + "packages/paperclip-runner/test/fixtures/copilot-profile-v59-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v65-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json", + "packages/paperclip-runner/test/fixtures/copilot-profile-v67-identity.json", + "packages/paperclip-runner/test/fixtures/copilot-profile-v68-identity.json", "packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs", "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs", "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs", diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index bebc81eacff..9cffbbe678c 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -21,6 +21,7 @@ import { waitForTaskChatRendered, waitForTaskIdentity } from "./continuation-scr import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interruption.js"; import { collectRunEvents } from "./run-observations.js"; import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js"; +import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js"; import { setupAggregatorFixture } from "./aggregator-fixture.js"; import { gradeProviderChoice, gradeProviderOutcome, requireProviderAccessCard } from "./connection-routing-evidence.js"; import { setupConnectionReview } from "./connection-reviews.js"; @@ -89,7 +90,8 @@ interface Input { workspacePath: string; privateDir: string; deadlineAt: number; - restart(controllerDowntimeMs?: number): Promise; + restartCount?: number; + restart(binding?: import("./controller-recovery-observer.js").RecoveryBinding, controllerDowntimeMs?: number): Promise; observe(issue: StoryIssue, runs: StoryRun[]): void; capture(id: string, label: string, file: string): Promise; evidence(name: string, value: unknown): Promise; @@ -168,9 +170,7 @@ export async function runEverydayFlow(input: Input) { let review: Awaited> | undefined; let project = fixtures.project; const caseId = execution.task.id; - const requireRemoteRestart = caseId === "recover-controller" && execution.environment.id === "daytona" - && execution.profile.generation === "native" && (execution.profile.provider === "codex" - || execution.profile.provider === "acpx"); + const requireRemoteRestart = controllerOutageForExecution(execution, process.env) !== undefined; const providerChoice = caseId === "provider-decline" || caseId === "provider-second"; const nativeProviderCase = caseId === "provider-native"; let aggregatorFixture: Awaited> | undefined; @@ -550,9 +550,9 @@ export async function runEverydayFlow(input: Input) { async function sourceReady() { await recordSource("source-saved-before-interruption", "source-before-interruption.json", true); } - async function remoteRestartBoundary(runId?: string) { + async function remoteRestartBoundary(runId?: string, recovered = false) { await refresh(); - const active = ev.runs.filter(run => run.status === "running" && run.nativeIssueId === parent!.id + const active = ev.runs.filter(run => (run.status === "running" || (recovered && run.status === "succeeded")) && run.nativeIssueId === parent!.id && (!runId || run.id === runId)); if (active.length !== 1) throw new Error("Remote restart boundary unexercised: the original run is not active"); const run = active[0]!; @@ -561,7 +561,7 @@ export async function runEverydayFlow(input: Input) { api.get(`/api/environments/${fixtures.environment.id}/leases`), ]); return observeRemoteRestartBoundary(run, rows, leases, { companyId: fixtures.company.id, - agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id }); + agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id }, recovered); } async function prepareStopBoundary() { // Providers can finish a short first turn before the browser can click @@ -596,6 +596,11 @@ export async function runEverydayFlow(input: Input) { "everyday-delivery.ts", "everyday-observations.ts", "everyday-remote-restart.ts", + "controller-outage.ts", + "controller-recovery-observer.ts", + "upgrade-artifacts.ts", + "server.ts", + "server-stop.ts", "native-event-envelope.ts", "run-observations.ts", "everyday-artifact.py", @@ -937,27 +942,34 @@ export async function runEverydayFlow(input: Input) { } else { await reply(LATE_REQUIREMENT); note("followup-submitted-before-interruption"); - if (requireRemoteRestart) { - const before = await waitForRemoteRestartBoundary({ - deadlineAt: input.deadlineAt, load: async () => { - await refresh(); - const original = ev.runs.find(run => run.id === active.id); - if (!original || original.status !== "running") return { run: original }; - const rows = await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${active.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); - if (!rows.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed")) return { run: original }; - return { run: original, boundary: await remoteRestartBoundary(active.id) }; - } }); - ev.remoteRestart = { before }; - await input.evidence("remote-restart-before.json", before); - } else ev.allowedInterruptedRuns.push(active.id); - await input.restart(requireRemoteRestart ? 125_000 : undefined); - note("controller-restarted"); - await openParent(); - if (requireRemoteRestart) { - const after = await remoteRestartBoundary(active.id); - ev.remoteRestart!.after = after; - await input.evidence("remote-restart-after.json", after); - if (!gradeRemoteRestart(ev.remoteRestart)[0]!.passed) throw new Error("Remote restart changed the original native process, lease or receipt prefix"); + for (let restartIndex = 0; restartIndex < (requireRemoteRestart ? input.restartCount ?? 1 : 1); restartIndex += 1) { + let currentBefore: RemoteRestartEvidence["before"]; + if (requireRemoteRestart) { + const before = await waitForRemoteRestartBoundary({ + deadlineAt: input.deadlineAt, load: async () => { + await refresh(); + const original = ev.runs.find(run => run.id === active.id); + if (!original || original.status !== "running") return { run: original }; + const rows = await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${active.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); + if (!rows.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed")) return { run: original }; + return { run: original, boundary: await remoteRestartBoundary(active.id) }; + } }); + currentBefore = before; + ev.remoteRestart ??= { before, expectedReplacements: input.restartCount ?? 1, replacements: [] }; + await input.evidence(`remote-restart-${restartIndex + 1}-before.json`, before); + } else ev.allowedInterruptedRuns.push(active.id); + const outage = await input.restart(requireRemoteRestart ? { companyId: fixtures.company.id, issueId: parent!.id, runId: active.id } : undefined, requireRemoteRestart ? 125_000 : undefined); + if (requireRemoteRestart) ev.remoteRestart!.outage = outage || undefined; + note("controller-restarted"); + await openParent(); + if (requireRemoteRestart) { + const after = await remoteRestartBoundary(active.id, true); + ev.remoteRestart!.after = after; + if (!outage || !currentBefore) throw new Error("Missing replacement fault evidence"); + ev.remoteRestart!.replacements!.push({ before: currentBefore, after, outage }); + await input.evidence(`remote-restart-${restartIndex + 1}-after.json`, { after, outage }); + if (!gradeRemoteRestart(ev.remoteRestart)[0]!.passed) throw new Error("Remote restart changed the original native process, lease or receipt prefix"); + } } } } @@ -1636,7 +1648,7 @@ export async function runEverydayFlow(input: Input) { api.get(`/api/issues/${parent!.id}`), ]); if (!Array.isArray(operationRows) || !Array.isArray(recovery.actions) || !("active" in recovery)) throw new Error("Remote restart finalization evidence is incomplete"); - const final = { run: projectRemoteRestartRun(run), events: projectRemoteRestartEvents(run, rows), + const final = { observedAt: new Date().toISOString(), run: projectRemoteRestartRun(run), events: projectRemoteRestartEvents(run, rows), operations: operationRows.map(operation => ({ companyId: operation.companyId, heartbeatRunId: operation.heartbeatRunId, issueId: operation.issueId, executionWorkspaceId: operation.executionWorkspaceId, phase: operation.phase, status: operation.status })), scheduledRetry: issue.scheduledRetry, diff --git a/tests/runner-e2e/everyday-remote-restart.test.ts b/tests/runner-e2e/everyday-remote-restart.test.ts index c492e4f9e72..43e41fd6b54 100644 --- a/tests/runner-e2e/everyday-remote-restart.test.ts +++ b/tests/runner-e2e/everyday-remote-restart.test.ts @@ -1,4 +1,5 @@ import { afterEach, describe, expect, it, vi } from "vitest"; +import type { ControllerOutageReceipt } from "./controller-outage.js"; import { storyLifecycleChecks, type StoryRun } from "./everyday-observations.js"; import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary } from "./everyday-remote-restart.js"; @@ -31,10 +32,20 @@ function fixture() { } }; const finalRows = [...afterRows, event(6, "run.result.proposed"), event(7, "turn.completed"), event(8, "run.result.accepted", {}, "control_plane", 1), event(9, "run.terminal", {}, "control_plane", 2)]; - const final = { run: finished, events: projectRemoteRestartEvents(finished, finalRows), + const final = { observedAt: "2026-10-22T00:02:00.000Z", run: finished, events: projectRemoteRestartEvents(finished, finalRows), operations: [{ companyId: "company", heartbeatRunId: "run", issueId: "task", executionWorkspaceId: "workspace", phase: "workspace_finalize", status: "succeeded" }], scheduledRetry: null as unknown, recoveryActions: [] as Record[] }; - return { active, scope, leases, event, rows, afterRows, finalRows, evidence: { before, after, final } }; + const outage: ControllerOutageReceipt = { schema: "paperclip.e2e.controller-outage/v1", requestId: "story-test", + signal: "SIGKILL", holdMs: 1000, previousPid: 100, replacementPid: 200, + startedAt: "2026-10-22T00:00:00.000Z", stoppedAfterMs: 100, cacheDiscarded: true, + replacementStartedAfterMs: 1100, readyAfterMs: 4000, + recovery: { binding: { companyId: "company", issueId: "task", runId: "run" }, runnerInstanceId: "runner", nativeSessionId: "session", + previousGeneration: 1, generation: 2, previousBootId: "old", bootId: "new", previousLeaseExpiresAt: "2026-10-22T00:01:00.000Z", + ownershipAcquiredAt: "2026-10-22T00:01:00.001Z", ownershipObservedAfterMs: 60_010, + authenticatedAttachmentObservedAfterMs: 61_000, resumedProcessingObservedAfterMs: 61_000, + runnerSourceSeqBefore: 4, runnerSourceSeqAfter: 5, previousAuthenticatedConnectionCount: 1, + authenticatedConnectionCount: 2, attachmentEvidence: "runner_checkpoint_authenticated_connection" } }; + return { active, scope, leases, event, rows, afterRows, finalRows, evidence: { before, after, final, outage } }; } describe("remote restart boundary polling", () => { @@ -66,6 +77,64 @@ describe("remote restart boundary polling", () => { }); describe("remote restart lifecycle evidence", () => { + it("accepts a plugin task with no controller PID and permits completion during the outage", () => { + const f = fixture(); + const task = { runnerId: "runner", leaseId: "lease", runId: "run", sessionId: "session", turnId: "turn", itemId: "item" }; + const lease = { ...f.leases[0]!, provider: "plugin", metadata: { runnerTaskDispatched: true, runnerTaskIdentity: task } }; + const run = { ...f.active, processPid: null, processStartedAt: null }; + const before = observeRemoteRestartBoundary(run, f.rows, [lease], f.scope); + const completedLease = { ...lease, status: "released", metadata: { + ...lease.metadata, runnerTaskDispatched: false, runnerTaskExecutionStopped: true, + } }; + const after = observeRemoteRestartBoundary({ ...run, status: "succeeded" }, f.finalRows, [completedLease], f.scope, true); + expect(before.identity).toEqual(after.identity); + expect(before.identity.processPid).toBeNull(); + expect(before.identity.taskIdentitySha256).toMatch(/^[a-f0-9]{64}$/); + expect(() => observeRemoteRestartBoundary({ ...run, processPid: 123 }, f.rows, [lease], f.scope)).toThrow("not the controller PID"); + for (const runnerTaskExecutionStopped of [undefined, false]) { + const unconfirmed = { ...completedLease, metadata: { ...completedLease.metadata, runnerTaskExecutionStopped } }; + expect(() => observeRemoteRestartBoundary({ ...run, status: "succeeded" }, f.finalRows, [unconfirmed], f.scope, true)) + .toThrow("admitted plugin task identity"); + } + const stoppedActiveLease = { ...lease, metadata: completedLease.metadata }; + expect(() => observeRemoteRestartBoundary(run, f.rows, [stoppedActiveLease], f.scope, true)) + .toThrow("admitted plugin task identity"); + const foreignTask = { ...completedLease, metadata: { ...completedLease.metadata, runnerTaskIdentity: { ...task, runId: "replacement" } } }; + expect(() => observeRemoteRestartBoundary({ ...run, status: "succeeded" }, f.finalRows, [foreignTask], f.scope, true)) + .toThrow("admitted plugin task identity"); + }); + it("does not confuse unchanged metadata or HTTP readiness with authenticated recovery", () => { + const { evidence } = fixture(); + delete evidence.outage.recovery; + expect(gradeRemoteRestart(evidence).find(check => check.id === "remote-restart-authenticated-recovery")?.passed).toBe(false); + }); + it("requires every requested immutable replacement instead of counting one restart as rollback", () => { + const { evidence } = fixture(); + const A = { revision: "a".repeat(40), treeSha256: "a".repeat(64), version: "1.0.0", evidence: "development-build" as const }; + const B = { ...A, revision: "b".repeat(40), treeSha256: "b".repeat(64) }; + const first = { before: evidence.before, after: evidence.after, outage: { ...evidence.outage, previousArtifact: A, replacementArtifact: B } }; + const second = { before: evidence.after, after: evidence.after, outage: { ...evidence.outage, previousArtifact: B, replacementArtifact: A } }; + const grade = (replacements: typeof first[]) => gradeRemoteRestart({ ...evidence, expectedReplacements: 2, replacements }) + .find(check => check.id === "remote-restart-all-replacements")!.passed; + expect(grade([first])).toBe(false); + expect(grade([first, second])).toBe(true); + for (const field of ["runnerInstanceId", "nativeSessionId"] as const) { + expect(grade([{ ...first, outage: { ...first.outage, recovery: { ...first.outage.recovery!, [field]: "another" } } }, second])).toBe(false); + } + for (const field of ["companyId", "issueId"] as const) { + expect(grade([{ ...first, outage: { ...first.outage, recovery: { ...first.outage.recovery!, + binding: { ...first.outage.recovery!.binding, [field]: "another" } } } }, second])).toBe(false); + } + expect(grade([first, { ...second, outage: { ...second.outage, recovery: undefined } }])).toBe(false); + expect(grade([first, { ...second, outage: { ...second.outage, previousArtifact: A } }])).toBe(false); + }); + it("rejects otherwise successful recovery without a complete actual-server fault receipt", () => { + const { evidence } = fixture(); + for (const outage of [undefined, { ...evidence.outage, replacementPid: 100 }, + { ...evidence.outage, replacementStartedAfterMs: 1099 }, { ...evidence.outage, cacheDiscarded: false }]) { + expect(gradeRemoteRestart({ ...evidence, outage }).every(check => check.passed)).toBe(false); + } + }); it("does not accept a successful replacement as proof that the original remote runner recovered", () => { const { evidence } = fixture(); const replacement = { ...evidence.final.run, id: "replacement", runnerInstanceId: "replacement-runner" }; diff --git a/tests/runner-e2e/everyday-remote-restart.ts b/tests/runner-e2e/everyday-remote-restart.ts index 9b50934eec6..cbc4a87548c 100644 --- a/tests/runner-e2e/everyday-remote-restart.ts +++ b/tests/runner-e2e/everyday-remote-restart.ts @@ -3,6 +3,8 @@ import { canonicalJson } from "../../packages/shared/src/portability-hash.js"; import { isValidNativePrpEnvelope } from "./native-event-envelope.js"; import type { StoryCheck, StoryRun } from "./everyday-observations.js"; import { pollUntil } from "./api.js"; +import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js"; +import { validControllerRecoveryReceipt } from "./controller-recovery-observer.js"; type Row = Record; const record = (value: unknown): Row => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Row : {}; @@ -17,15 +19,19 @@ export interface RemoteRestartEvent { } export interface RemoteRestartBoundary { identity: { runId: string; companyId: string; agentId: string; issueId: string; - runnerInstanceId: string; nativeSessionId: string; processPid: number; processStartedAt: string }; + runnerInstanceId: string; nativeSessionId: string; processPid: number | null; processStartedAt: string | null; + taskIdentitySha256?: string }; lease: { id: string; environmentId: string; executionWorkspaceId: string | null; providerLeaseSha256: string }; events: RemoteRestartEvent[]; } export interface RemoteRestartEvidence { + expectedReplacements?: number; + replacements?: Array<{ before: RemoteRestartBoundary; after: RemoteRestartBoundary; outage: ControllerOutageReceipt }>; + outage?: ControllerOutageReceipt; before?: RemoteRestartBoundary; after?: RemoteRestartBoundary; final?: { run: StoryRun; events: RemoteRestartEvent[]; operations: Row[]; - scheduledRetry: unknown; recoveryActions: Row[] }; + scheduledRetry: unknown; recoveryActions: Row[]; observedAt?: string }; } export async function waitForRemoteRestartBoundary(input: { @@ -87,30 +93,49 @@ export function projectRemoteRestartEvents(run: StoryRun, rows: readonly Row[]): export function observeRemoteRestartBoundary(run: StoryRun, rows: readonly Row[], leases: readonly Row[], scope: { companyId: string; agentId: string; issueId: string; environmentId: string; -}): RemoteRestartBoundary { +}, recovered = false): RemoteRestartBoundary { if (run.companyId !== scope.companyId || run.agentId !== scope.agentId || run.nativeIssueId !== scope.issueId - || run.status !== "running" || run.runtimeMode !== "native" || !id(run.id) || !id(run.runnerInstanceId) - || !id(run.nativeSessionId) || !Number.isSafeInteger(run.processPid) || Number(run.processPid) <= 0 - || !id(run.processStartedAt) || !Number.isFinite(Date.parse(run.processStartedAt))) { + || !(run.status === "running" || (recovered && run.status === "succeeded")) + || run.runtimeMode !== "native" || !id(run.id) || !id(run.runnerInstanceId) || !id(run.nativeSessionId)) { throw new Error("Remote restart boundary requires the original live native process"); } const owned = leases.filter(lease => lease.companyId === scope.companyId && lease.environmentId === scope.environmentId - && lease.issueId === scope.issueId && lease.heartbeatRunId === run.id && lease.provider === "daytona" - && lease.status === "active" && lease.releasedAt == null && id(lease.id) && id(lease.providerLeaseId)); + && lease.issueId === scope.issueId && lease.heartbeatRunId === run.id + && (lease.provider === "daytona" || lease.provider === "plugin" || String(lease.provider).startsWith("plugin:")) + && ((lease.status === "active" && lease.releasedAt == null) || (recovered && run.status === "succeeded")) + && id(lease.id) && id(lease.providerLeaseId)); if (owned.length !== 1) throw new Error("Remote restart boundary requires one exact active sandbox lease"); const lease = owned[0]!; + const metadata = record(lease.metadata), task = record(metadata.runnerTaskIdentity); + const plugin = lease.provider === "plugin" || String(lease.provider).startsWith("plugin:"); + if (plugin) { + // The normal task authority clears dispatch only after provider status confirms + // execution stopped. Completion during the outage must retain that proof. + const settled = recovered && run.status === "succeeded" + && metadata.runnerTaskDispatched === false && metadata.runnerTaskExecutionStopped === true; + if (run.processPid != null || run.processStartedAt != null || (!settled && metadata.runnerTaskDispatched !== true) + || task.runnerId !== run.runnerInstanceId || task.runId !== run.id || task.sessionId !== run.nativeSessionId + || !id(task.leaseId) || !id(task.turnId) || !id(task.itemId)) { + throw new Error("Remote restart requires the admitted plugin task identity, not the controller PID"); + } + } else if (!Number.isSafeInteger(run.processPid) || Number(run.processPid) <= 0 + || !id(run.processStartedAt) || !Number.isFinite(Date.parse(run.processStartedAt))) { + throw new Error("Remote restart boundary requires the original live native process"); + } const events = projectRemoteRestartEvents(run, rows); const starts = events.filter(event => event.eventType === "turn.started" && event.sourceKind === "runner"); const completed = events.filter(event => event.toolCompleted && event.sourceKind === "runner" && event.toolExecutionId); if (starts.length !== 1 || !id(starts[0]!.turnId) || completed.length === 0 || completed.some(event => event.turnId !== starts[0]!.turnId) - || events.some(event => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted", "run.terminal"].includes(event.eventType))) { + || (!recovered && events.some(event => ["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted", "run.terminal"].includes(event.eventType)))) { throw new Error("Remote restart boundary has no completed tool in an active original provider turn"); } return { identity: { runId: run.id, companyId: scope.companyId, agentId: scope.agentId, issueId: scope.issueId, runnerInstanceId: run.runnerInstanceId, nativeSessionId: run.nativeSessionId, - processPid: Number(run.processPid), processStartedAt: run.processStartedAt }, + processPid: plugin ? null : Number(run.processPid), processStartedAt: plugin ? null : run.processStartedAt!, + ...(plugin ? { taskIdentitySha256: digest({ runnerId: task.runnerId, leaseId: task.leaseId, + runId: task.runId, sessionId: task.sessionId, turnId: task.turnId, itemId: task.itemId }) } : {}) }, lease: { id: String(lease.id), environmentId: scope.environmentId, executionWorkspaceId: typeof lease.executionWorkspaceId === "string" ? lease.executionWorkspaceId : null, providerLeaseSha256: digest(lease.providerLeaseId) }, @@ -161,9 +186,40 @@ export function gradeRemoteRestart(evidence: RemoteRestartEvidence | undefined): && operation.executionWorkspaceId === before?.lease.executionWorkspaceId) && !operations.some(operation => ["pending", "queued", "running"].includes(String(operation.status))) && final?.scheduledRetry == null && !final?.recoveryActions.some(action => ["active", "pending"].includes(String(action.status))); + const expectedReplacements = evidence?.expectedReplacements ?? 1; + const replacements = evidence?.replacements; + const allReplacements = replacements === undefined ? expectedReplacements === 1 + : replacements.length === expectedReplacements && replacements.every((step, index, all) => { + const previous = all[index - 1]; + const preservesIdentity = digest(step.before.identity) === digest(step.after.identity) + && digest(step.before.lease) === digest(step.after.lease) + && retainsPrefix(step.before.events, step.after.events); + const authenticated = validControllerOutageReceipt(step.outage) && validControllerRecoveryReceipt(step.outage) + && step.outage.recovery?.binding.runId === step.before.identity.runId + && step.outage.recovery.binding.companyId === step.before.identity.companyId + && step.outage.recovery.binding.issueId === step.before.identity.issueId + && step.outage.recovery.runnerInstanceId === step.before.identity.runnerInstanceId + && step.outage.recovery.nativeSessionId === step.before.identity.nativeSessionId; + const artifactsPresent = expectedReplacements <= 1 || Boolean(step.outage.previousArtifact && step.outage.replacementArtifact); + const chained = !previous || (retainsPrefix(previous.after.events, step.before.events) + && (!step.outage.previousArtifact || Boolean(previous.outage.replacementArtifact + && digest(previous.outage.replacementArtifact) === digest(step.outage.previousArtifact)))); + return preservesIdentity && authenticated && artifactsPresent && chained; + }); return [ - check("remote-restart-same-process-and-lease", sameIdentity, "The same run, native session, Runner process and sandbox lease remain live after restart."), + check("remote-restart-same-execution-and-lease", sameIdentity, "The original run, native session, admitted runner identity and sandbox lease survive replacement."), check("remote-restart-native-completion-once", nativeCompletion, "The original provider turn completes once and retains its native receipt prefix."), check("remote-restart-files-finalized", Boolean(saved), "The original run commits its workspace receipt and leaves no active operation or recovery."), + check("remote-restart-fault-exercised", validControllerOutageReceipt(evidence?.outage), "The main server exited, its cache was retired, and a distinct replacement became ready after the requested outage."), + check("remote-restart-authenticated-recovery", validControllerRecoveryReceipt(evidence?.outage) + && evidence?.outage?.recovery?.binding.runId === identity?.runId + && evidence?.outage?.recovery?.runnerInstanceId === identity?.runnerInstanceId + && evidence?.outage?.recovery?.nativeSessionId === identity?.nativeSessionId, + "Natural ownership acquisition and new authenticated runner receipts were observed within the original 420-second grace."), + check("remote-restart-backlog-settled", Boolean(saved && nativeCompletion && final?.observedAt + && Date.parse(final.observedAt) >= Date.parse(evidence?.outage?.startedAt ?? "") + (evidence?.outage?.recovery?.resumedProcessingObservedAfterMs ?? Infinity)), + "Canonical terminal and workspace receipts settle the original backlog after reattachment."), + check("remote-restart-all-replacements", allReplacements, + "Every requested upgrade and rollback exercised the original running execution and retained its receipts."), ]; } diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index 9aa865b0c1f..b6814d877f1 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -135,6 +135,10 @@ export function buildRunnerE2EProcessEnvironment( const installed = usesInstalledCli(source); if (installed) assertInstalledCliSelection(source, executions); const result = { ...source }; + // The supervisor's read-only recovery observer must use the same disposable + // instance key as the serving child, never a launching shell's vault key. + delete result.PAPERCLIP_SECRETS_MASTER_KEY; + delete result.PAPERCLIP_SECRETS_MASTER_KEY_FILE; delete result.PAPERCLIP_E2E_GATEWAY_PHASE_PATH; delete result.PAPERCLIP_E2E_OTLP_RELAY_URL; delete result.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN; @@ -211,7 +215,7 @@ export function buildPaperclipServerEnvironment( ]) { delete result[key]; } - for (const key of [...GENERATED_SERVER_SECRET_KEYS, ...installedCliKeys, ...installedDaytonaPluginKeys]) delete result[key]; + for (const key of [...GENERATED_SERVER_SECRET_KEYS, ...installedCliKeys, ...installedDaytonaPluginKeys, "PAPERCLIP_RUNNER_E2E_UPGRADE_ARTIFACTS"]) delete result[key]; Object.assign(result, overrides); return result; } diff --git a/tests/runner-e2e/installed-cli.test.ts b/tests/runner-e2e/installed-cli.test.ts index 83770cd98ac..7b917353ba3 100644 --- a/tests/runner-e2e/installed-cli.test.ts +++ b/tests/runner-e2e/installed-cli.test.ts @@ -32,7 +32,7 @@ it("admits all26 current Pi cases through pinned installed public CLI/server wit }); it("admits existing Codex onboarding, useful-file revision and restart journeys without broadening other cells", async () => { const { env, server } = await fixture(); - for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart"]) { + for (const id of ["first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", "everyday-workflows.runner-codex.daytona.recover-controller"]) { const cell = runnerMatrix.find(e => e.id === id)!; expect(cell).toBeDefined(); expect(await verifyInstalledCli(env, [cell])).toMatchObject({ serverRoot: server, defaultRuntimeResolution: true, qualificationOverride: false }); diff --git a/tests/runner-e2e/installed-cli.ts b/tests/runner-e2e/installed-cli.ts index 2fc4aa5c797..0e5fe3c55c5 100644 --- a/tests/runner-e2e/installed-cli.ts +++ b/tests/runner-e2e/installed-cli.ts @@ -16,6 +16,7 @@ const installedCodexCells = new Set([ "first-task.runner-codex.local.task-card-accept", "everyday-workflows.runner-codex.local.build-revise", "agent-chat.runner-codex.local.continuity-restart", + "everyday-workflows.runner-codex.daytona.recover-controller", ]); const installedCopilotCells = new Set([ "copilot-protection.runner-acpx-copilot.daytona.attached-async-settlement", @@ -46,7 +47,7 @@ export function assertInstalledCliSelection(env: NodeJS.ProcessEnv, executions: if (!usesInstalledCli(env)) return; if (!installedCliKeys.every(key => Boolean(env[key])) || executions.length === 0 || executions.some(e => !( (e.profile.id === "runner-acpx-pi" && e.profile.qualificationCandidate === "pi" && e.suite.manualOnly && suites.has(e.suite.id)) || - (e.profile.id === "runner-codex" && e.profile.provider === "codex" && e.environment.id === "local" && installedCodexCells.has(e.id)) || + (e.profile.id === "runner-codex" && e.profile.provider === "codex" && installedCodexCells.has(e.id)) || (e.profile.id === "runner-acpx-copilot" && e.profile.qualificationCandidate === "copilot" && e.suite.manualOnly && installedCopilotCells.has(e.id)) ))) throw new Error("Installed CLI proof requires complete pins and explicit supported Pi, Codex, or Copilot cases"); for (const key of overrideKeys) if (env[key] !== undefined) throw new Error(`Installed CLI proof forbids ${key}`); diff --git a/tests/runner-e2e/instance-isolation.test.ts b/tests/runner-e2e/instance-isolation.test.ts new file mode 100644 index 00000000000..5923b79ef16 --- /dev/null +++ b/tests/runner-e2e/instance-isolation.test.ts @@ -0,0 +1,40 @@ +import path from "node:path"; +import { mkdir, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import { expect, it } from "vitest"; +import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js"; +import { createRunnerE2ETemporaryRoot, prepareRunnerE2EServerConfig } from "./server-config.js"; + +it.each(["owned", "missing-receipt", "foreign-connection", "wrong-port", "foreign-owner", "escaped-database"])( + "checks the supervisor-owned database after the serving process exits: %s", async variant => { + const root = await createRunnerE2ETemporaryRoot(); + const configPath = path.join(root, "instance", "config.json"); + const outside = await createRunnerE2ETemporaryRoot(); + try { + await (await prepareRunnerE2EServerConfig({ temporaryRoot: root, configPath, serverPort: 3100 }))!.close(); + const config = JSON.parse(await readFile(configPath, "utf8")); + const directory = config.database.embeddedPostgresDataDir; + await mkdir(directory, { recursive: true }); + await writeFile(path.join(directory, "PG_VERSION"), "18\n"); + await mkdir(path.dirname(config.secrets.localEncrypted.keyFilePath), { recursive: true }); + await writeFile(config.secrets.localEncrypted.keyFilePath, "fixture-key"); + config.database.mode = "postgres"; + config.database.connectionString = `postgres://paperclip:paperclip@127.0.0.1:${config.database.embeddedPostgresPort}/paperclip`; + const receipt = { version: 1, configPath, temporaryRoot: root, dataDirectory: directory, + connectionString: config.database.connectionString }; + if (variant === "foreign-owner") receipt.temporaryRoot = outside; + if (variant !== "missing-receipt") await writeFile(path.join(directory, ".runner-e2e-controller-owner.json"), JSON.stringify(receipt)); + if (variant === "foreign-connection") config.database.connectionString = "postgres://paperclip:paperclip@db.example.invalid:5432/paperclip"; + if (variant === "wrong-port") config.database.embeddedPostgresPort += 1; + if (variant === "escaped-database") { + await rm(directory, { recursive: true }); + await symlink(outside, directory, "dir"); + } + await writeFile(configPath, JSON.stringify(config)); + if (variant === "owned") await expect(assertEmbeddedDatabaseIsolation(configPath, root)).resolves.toBeUndefined(); + else await expect(assertEmbeddedDatabaseIsolation(configPath, root)).rejects.toThrow(/fixture-owned|escaped|embedded database/); + } finally { + await rm(root, { recursive: true, force: true }); + await rm(outside, { recursive: true, force: true }); + } + }, +); diff --git a/tests/runner-e2e/instance-isolation.ts b/tests/runner-e2e/instance-isolation.ts index 46540f5a9ed..9bd98b60a3e 100644 --- a/tests/runner-e2e/instance-isolation.ts +++ b/tests/runner-e2e/instance-isolation.ts @@ -1,5 +1,22 @@ import path from "node:path"; -import { readFile, stat } from "node:fs/promises"; +import { readFile, realpath, stat, writeFile } from "node:fs/promises"; + +const CONTROLLER_DATABASE_OWNER_FILE = ".runner-e2e-controller-owner.json"; + +/** Written only after the supervisor has started this fixture's Postgres. */ +export async function recordControllerDatabaseOwnership(input: { + configPath: string; temporaryRoot: string; dataDirectory: string; connectionString: string; +}) { + const temporaryRoot = await realpath(input.temporaryRoot); + const dataDirectory = await realpath(input.dataDirectory); + if (!dataDirectory.startsWith(`${temporaryRoot}${path.sep}`)) { + throw new Error("Controller database path escaped its fixture-owned root"); + } + await writeFile(path.join(dataDirectory, CONTROLLER_DATABASE_OWNER_FILE), JSON.stringify({ + version: 1, configPath: await realpath(input.configPath), temporaryRoot, + dataDirectory, connectionString: input.connectionString, + }), { mode: 0o600 }); +} export async function assertEmbeddedDatabaseIsolation( configPath: string, @@ -11,6 +28,7 @@ export async function assertEmbeddedDatabaseIsolation( mode?: string; connectionString?: string; embeddedPostgresDataDir?: string; + embeddedPostgresPort?: number; backup?: { dir?: string }; }; logging?: { logDir?: string }; @@ -24,13 +42,10 @@ export async function assertEmbeddedDatabaseIsolation( localEncrypted?: { keyFilePath?: string }; }; }; - if ( - config.database?.mode !== "embedded-postgres" || - config.database.connectionString - ) { - throw new Error( - "Runner E2E Paperclip instance did not use its embedded database", - ); + const supervised = config.database?.mode === "postgres"; + if (!config.database || (!supervised && + (config.database.mode !== "embedded-postgres" || config.database.connectionString))) { + throw new Error("Runner E2E Paperclip instance did not use its fixture-owned database"); } if ( config.storage?.provider !== "local_disk" || @@ -60,6 +75,25 @@ export async function assertEmbeddedDatabaseIsolation( } } const databasePath = path.resolve(isolatedPaths.database!); + if (supervised) { + try { + const root = await realpath(temporaryRoot); + const directory = await realpath(databasePath); + if (!directory.startsWith(`${root}${path.sep}`)) throw new Error("Database escaped fixture"); + const owner = JSON.parse(await readFile(path.join(directory, CONTROLLER_DATABASE_OWNER_FILE), "utf8")); + const port = config.database.embeddedPostgresPort; + const expectedConnection = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`; + if (!Number.isInteger(port) || Number(port) < 1 || Number(port) > 65535 + || config.database.connectionString !== expectedConnection + || owner.version !== 1 || owner.temporaryRoot !== root || owner.dataDirectory !== directory + || owner.configPath !== await realpath(configPath) || owner.connectionString !== expectedConnection + || !(await stat(path.join(directory, "PG_VERSION"))).isFile()) { + throw new Error("Supervisor database ownership mismatch"); + } + } catch (cause) { + throw new Error("Runner E2E Paperclip instance did not use its fixture-owned database", { cause }); + } + } const databaseStat = await stat(databasePath); if (!databaseStat.isDirectory()) throw new Error("Embedded database path is not a directory"); diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index 74ec9179866..7694feb483a 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -1,3 +1,4 @@ +import { readUpgradeArtifacts, verifyUpgradeArtifact, upgradeArtifactKey } from "./upgrade-artifacts.js"; import { recoverWorkerTiming } from "./worker-timing.js"; import { openEvalOperations, type EvalOperations } from "./operations.js"; import { readRecoveredFeedback } from "./feedback-recovery.js"; @@ -32,6 +33,7 @@ import { import { isImmutableDaytonaImage, runnerMatrix } from "./catalog.js"; import { renderRunnerE2EDashboard } from "./dashboard.js"; import { packageEvidence } from "./evidence.js"; +import { copySharedEvidence } from "./shared-evidence.js"; import { classifyFailure } from "./failure-classifier.js"; import { mustPreserveRecoveryState, shouldKeepFailedDiagnostics } from "./cleanup-verification.js"; import { effectiveAutomaticRetryLimit, executeWithAutomaticRetry } from "./automatic-retry.js"; @@ -451,24 +453,6 @@ async function readResult(resultPath: string, fallback: RunnerE2EResult) { } } -async function copySharedEvidence(privateRoot: string, casePrivateDir: string) { - for (const relative of [ - "server.log", - "playwright.log", - "junit.xml", - "html-report", - "blob-report", - "playwright-output", - ]) { - await cp( - path.join(privateRoot, relative), - path.join(casePrivateDir, relative), - { recursive: true, force: true }, - ).catch((error: NodeJS.ErrnoException) => { - if (error.code !== "ENOENT") throw error; - }); - } -} async function runAttempt(input: { executions: readonly MatrixExecution[]; @@ -494,6 +478,15 @@ async function runAttempt(input: { "One isolated runner E2E harness cannot mix profiles or environments", ); } + const upgrades = await readUpgradeArtifacts(process.env[upgradeArtifactKey]); + if (upgrades) { + // All attempts and the browser use A's installed fixture admission. The + // supervisor changes only the server artifact; the original sandbox lives on. + Object.assign(process.env, upgrades.artifacts.A.pins); + for (const artifact of [upgrades.artifacts.A, upgrades.artifacts.B]) { + await verifyUpgradeArtifact(artifact, process.env, executions); + } + } assertInstalledCliSelection(process.env, executions); const installedCli = await verifyInstalledCli(process.env, executions); const installedPlugin = await verifyInstalledDaytonaPlugin(process.env, executions); @@ -816,7 +809,7 @@ async function runAttempt(input: { `attempt-${attempt}`, ); await mkdir(casePrivateDir, { recursive: true }); - await copySharedEvidence(privateDir, casePrivateDir); + await copySharedEvidence(privateDir, casePrivateDir, temporaryRoot, credentials); if (persistedStateError && persistenceCheckedExecutions.includes(candidate)) { const snapshots = path.join(casePrivateDir, "snapshots"); await mkdir(snapshots, { recursive: true }); diff --git a/tests/runner-e2e/prepare-report-lock.test.ts b/tests/runner-e2e/prepare-report-lock.test.ts index d56c0d263d0..a13b064ec48 100644 --- a/tests/runner-e2e/prepare-report-lock.test.ts +++ b/tests/runner-e2e/prepare-report-lock.test.ts @@ -14,10 +14,22 @@ it.each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])( try { await mkdir(relative, { recursive: true }); for (const file of ['report-lock.json', 'report-lock.patch']) await copyFile(path.join(here, file), path.join(relative, file)); - const input = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') }); + const committed = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') }); const policy = JSON.parse(await readFile(path.join(relative, 'report-lock.json'), 'utf8')); - expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256); + const committedHash = createHash('sha256').update(committed).digest('hex'); + expect([policy.inputSha256, policy.outputSha256]).toContain(committedHash); const lock = path.join(root, 'pnpm-lock.yaml'); + await writeFile(lock, committed); + if (variant === 'valid') { + expect(await prepareReportLock(root)).toBe(committedHash === policy.outputSha256 ? 'already-reviewed' : 'reviewed-correction'); + expect(createHash('sha256').update(await readFile(lock)).digest('hex')).toBe(policy.outputSha256); + await writeFile(lock, committed); + } + if (committedHash === policy.outputSha256) { + execFileSync('git', ['apply', '--reverse', path.join(relative, 'report-lock.patch')], { cwd: root }); + } + const input = await readFile(lock); + expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256); await writeFile(lock, variant === 'unknown-lock' ? 'unknown\n' : input); if (variant === 'changed-patch') await writeFile(path.join(relative, 'report-lock.patch'), 'changed'); if (variant === 'wrong-output') { policy.outputSha256 = '0'.repeat(64); await writeFile(path.join(relative, 'report-lock.json'), JSON.stringify(policy)); } diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 1584725be40..0f2fe4c9b94 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -2,6 +2,7 @@ import { createWorkerTiming } from "./worker-timing.js"; import { FEEDBACK_PERSISTENCE_SUITE, gradeFeedbackPersistence } from "./feedback-persistence.js"; import { EvalAttemptTrace, evalBraintrustConfig, EVAL_OBSERVABILITY_SECRET_NAMES } from "./braintrust.js"; import { captureFollowupStopFailureApiState } from "./followup-stop-evidence.js"; +import { controllerOutageForExecution, validControllerOutageReceipt, type ControllerOutage, type ControllerOutageReceipt } from "./controller-outage.js"; import { runFollowupStopResumeFlow } from "./followup-stop-resume-flow.js"; import { runFinalizationRecoveryFlow } from "./finalization-recovery-flow.js"; import type { FinalizationCrashBinding } from "./finalization-crash-boundary.js"; @@ -263,8 +264,12 @@ async function restartIsolatedPaperclipServer(input: { requestId: string; deadlineAt: number; preserveRunner?: RestartRunnerIdentity; + discardControllerCache?: boolean; crashAfterResult?: FinalizationCrashBinding; controllerDowntimeMs?: number; + controllerOutage?: ControllerOutage; + recoveryBinding?: import("./controller-recovery-observer.js").RecoveryBinding; + onControllerOutage?: (receipt: ControllerOutageReceipt) => Promise; }): Promise { const { controlDirectory, @@ -275,14 +280,16 @@ async function restartIsolatedPaperclipServer(input: { const temporaryRequestPath = `${requestPath}.${process.pid}.${input.requestId}.tmp`; await writeFile( temporaryRequestPath, - JSON.stringify({ requestId: input.requestId, ...(input.controllerDowntimeMs !== undefined ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), + JSON.stringify({ requestId: input.requestId, ...(input.controllerDowntimeMs !== undefined && !input.controllerOutage ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), discardControllerCache: input.discardControllerCache === true, + ...(input.controllerOutage ? { controllerOutage: input.controllerOutage } : {}), + ...(input.recoveryBinding ? { recoveryBinding: input.recoveryBinding } : {}), ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}), ...(input.crashAfterResult ? { crashAfterResult: input.crashAfterResult } : {}) }), { encoding: "utf8", mode: 0o600 }, ); await rename(temporaryRequestPath, requestPath); - await pollUntil({ + const acknowledgement = await pollUntil({ label: `isolated server restart ${input.requestId}`, timeoutFailureClass: "transient_infrastructure", deadlineAt: input.deadlineAt, @@ -298,7 +305,9 @@ async function restartIsolatedPaperclipServer(input: { }, accept: (acknowledgement) => acknowledgement.requestId === input.requestId && - acknowledgement.status === "ready", + acknowledgement.status === "ready" && + (!input.discardControllerCache || acknowledgement.controllerCacheDiscarded === true) && + (!input.controllerOutage || validControllerOutageReceipt(acknowledgement.controllerOutage, input.controllerOutage, input.requestId)), reject: (acknowledgement) => acknowledgement.requestId === input.requestId && acknowledgement.status === "failed" @@ -313,6 +322,7 @@ async function restartIsolatedPaperclipServer(input: { load: () => input.api.get>("/api/health"), accept: (health) => Boolean(health), }); + if (input.controllerOutage) await input.onControllerOutage?.(acknowledgement.controllerOutage as ControllerOutageReceipt); } const executionIds = (() => { @@ -580,6 +590,7 @@ async function expectPlanStageVisible( } for (const execution of executions) { + const controllerOutage = controllerOutageForExecution(execution, process.env); const privateDir = path.join(privateRoot, "cases", execution.task.id); const resultPath = path.join(privateDir, "result.json"); const snapshotsDir = path.join(privateDir, "snapshots"); @@ -1239,10 +1250,25 @@ for (const execution of executions) { }); issue = continuation.issue as IssueRecord; selectedRuns = continuation.runs as RunRecord[]; } else if (execution.task.flow === "everyday_workflow") { + const { readUpgradeArtifacts, upgradeArtifactKey } = await import("./upgrade-artifacts.js"); + const upgradeManifest = controllerOutage ? await readUpgradeArtifacts(process.env[upgradeArtifactKey]) : null; + let restartIndex = 0; const story = await runEverydayFlow({ page, api, fixtures, execution, nonce, workspacePath, privateDir, deadlineAt: startedAtMs + deadlineMs, - restart: controllerDowntimeMs => restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}`, deadlineAt: startedAtMs + deadlineMs, controllerDowntimeMs }), + restartCount: upgradeManifest ? upgradeManifest.sequence.length - 1 : 1, + restart: async (recoveryBinding, controllerDowntimeMs) => { + let receipt: ControllerOutageReceipt | undefined; + restartIndex += 1; + await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs, + discardControllerCache: controllerOutage !== undefined, controllerOutage, controllerDowntimeMs, + ...(controllerOutage ? { recoveryBinding } : {}), + onControllerOutage: async value => { + receipt = value; + await writeSanitizedJson(snapshotsDir, `controller-outage-${restartIndex}.json`, value, secrets); + } }); + return receipt; + }, observe: (storyIssue, storyRuns) => { issue = storyIssue; selectedRuns = storyRuns; }, capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), diff --git a/tests/runner-e2e/server-restart-request.test.ts b/tests/runner-e2e/server-restart-request.test.ts new file mode 100644 index 00000000000..691b54c81cd --- /dev/null +++ b/tests/runner-e2e/server-restart-request.test.ts @@ -0,0 +1,60 @@ +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, expect, it } from "vitest"; +import { runnerMatrix } from "./catalog.js"; +import { controllerOutageForExecution } from "./controller-outage.js"; +import { readRestartRequest } from "./server-restart-request.js"; + +let root: string; +let requestPath: string; +beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), "paperclip-supervisor-request-")); + requestPath = join(root, "restart.json"); +}); +afterEach(async () => { await rm(root, { recursive: true, force: true }); }); + +const cells = ["everyday-workflows", "daytona-restart-continuity"].flatMap(suite => + ["codex", "acpx"].map(provider => { + const execution = runnerMatrix.find(row => row.suite.id === suite && row.task.id === "recover-controller" + && row.environment.id === "daytona" && row.profile.generation === "native" && row.profile.provider === provider); + if (!execution) throw new Error(`Missing real ${suite}/${provider} continuity cell`); + return execution; + })); + +it.each(cells)("accepts the real supervisor request file for $id", async execution => { + const outage = controllerOutageForExecution(execution, {})!; + await writeFile(requestPath, JSON.stringify({ requestId: "actual-restart-request", + discardControllerCache: true, controllerOutage: outage })); + await expect(readRestartRequest(requestPath, [execution.id])).resolves.toMatchObject({ + requestId: "actual-restart-request", discardControllerCache: true, controllerOutage: outage, + }); +}); + +it("rejects cache replacement for unselected, local and legacy cells", async () => { + await writeFile(requestPath, JSON.stringify({ requestId: "unsupported-restart", discardControllerCache: true })); + const local = runnerMatrix.find(row => row.environment.id === "local")!; + const legacy = runnerMatrix.find(row => row.profile.generation === "legacy")!; + expect(local).toBeDefined(); expect(legacy).toBeDefined(); + for (const ids of [[], ["not-a-cell"], [local.id], [legacy.id]]) { + await expect(readRestartRequest(requestPath, ids)).rejects.toThrow(/cache replacement requires/i); + } +}); + +it.each([ + { controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: false }, + { controllerOutage: { signal: "SIGKILL", holdMs: 300_001 }, discardControllerCache: true }, + { controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: true, preserveRunner: {} }, + { controllerOutage: { signal: "SIGKILL", holdMs: 125_000 }, discardControllerCache: true, crashAfterResult: {} }, + { recoveryBinding: {} }, + { controllerDowntimeMs: -1 }, +])("keeps invalid and conflicting fault boundaries rejected: %j", async fault => { + await writeFile(requestPath, JSON.stringify({ requestId: "conflicting-restart", ...fault })); + await expect(readRestartRequest(requestPath, [cells[0].id])).rejects.toThrow(); +}); + +it("ignores absent and incomplete atomic request files", async () => { + await expect(readRestartRequest(requestPath, [cells[0].id])).resolves.toBeNull(); + await writeFile(requestPath, "{"); + await expect(readRestartRequest(requestPath, [cells[0].id])).resolves.toBeNull(); +}); diff --git a/tests/runner-e2e/server-restart-request.ts b/tests/runner-e2e/server-restart-request.ts new file mode 100644 index 00000000000..ab38d2bc590 --- /dev/null +++ b/tests/runner-e2e/server-restart-request.ts @@ -0,0 +1,68 @@ +import { readFile } from "node:fs/promises"; +import { runnerMatrix } from "./catalog.js"; +import { controllerOutageForExecution, parseControllerOutage, type ControllerOutage } from "./controller-outage.js"; +import { parseRecoveryBinding, type RecoveryBinding } from "./controller-recovery-observer.js"; +import { parseFinalizationCrashBinding, type FinalizationCrashBinding } from "./finalization-crash-boundary.js"; +import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js"; + +export interface RestartRequest { + requestId: string; + controllerDowntimeMs?: number; + preserveRunner?: RestartRunnerIdentity; + discardControllerCache?: boolean; + crashAfterResult?: FinalizationCrashBinding; + controllerOutage?: ControllerOutage; + recoveryBinding?: RecoveryBinding; +} + +export async function readRestartRequest(restartRequestPath: string, executionIds: readonly string[]): Promise { + let encoded: string; + try { + encoded = await readFile(restartRequestPath, "utf8"); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; + throw error; + } + let value: unknown; + try { + value = JSON.parse(encoded); + } catch { + // The writer may not have completed its atomic replacement yet. + return null; + } + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const requestId = (value as { requestId?: unknown }).requestId; + if ( + typeof requestId !== "string" || + !/^[A-Za-z0-9._:-]{1,200}$/.test(requestId) + ) { + return null; + } + const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner; + const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs; + if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs) + || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) { + throw new Error("Controller downtime must be an integer between zero and five minutes"); + } + const discardControllerCache = (value as { discardControllerCache?: unknown }).discardControllerCache === true; + if (discardControllerCache && !executionIds.some(id => { + const execution = runnerMatrix.find(row => row.id === id); + return execution !== undefined && controllerOutageForExecution(execution, {}) !== undefined; + })) throw new Error("Controller cache replacement requires an explicit remote controller-outage cell"); + const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult; + const controllerOutage = (value as { controllerOutage?: unknown }).controllerOutage; + const recoveryBinding = (value as { recoveryBinding?: unknown }).recoveryBinding; + if (recoveryBinding !== undefined && controllerOutage === undefined) throw new Error("Recovery observation requires a controller outage"); + if (controllerOutage !== undefined && (!discardControllerCache || preserveRunner !== undefined || crashAfterResult !== undefined)) { + throw new Error("Controller outage requires the explicit remote cache-replacement cell without another crash boundary"); + } + if (crashAfterResult !== undefined && (preserveRunner !== undefined || discardControllerCache)) { + throw new Error("A finalization crash cannot preserve a running provider or discard its controller cache"); + } + return { requestId, discardControllerCache, ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), + ...(recoveryBinding === undefined ? {} : { recoveryBinding: parseRecoveryBinding(recoveryBinding) }), + ...(controllerOutage === undefined ? {} : { controllerOutage: parseControllerOutage(controllerOutage) }), + ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), + ...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) }; + +} diff --git a/tests/runner-e2e/server.ts b/tests/runner-e2e/server.ts index 451ed91e034..c67f29caa82 100644 --- a/tests/runner-e2e/server.ts +++ b/tests/runner-e2e/server.ts @@ -1,5 +1,9 @@ -import { parseFinalizationCrashBinding, readFinalizationCrashBoundary, type FinalizationCrashBinding } from "./finalization-crash-boundary.js"; -import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./process-tree-owner.js"; +import { readRestartRequest, type RestartRequest } from "./server-restart-request.js"; +import { prepareControllerDatabase } from "./controller-database.js"; +import { readUpgradeArtifacts, verifyUpgradeArtifact, assertServingArtifact, upgradeArtifactKey, type ServingArtifact } from "./upgrade-artifacts.js"; +import { createControllerRecoveryObserver } from "./controller-recovery-observer.js"; +import { readFinalizationCrashBoundary } from "./finalization-crash-boundary.js"; +import { replaceController, retireControllerCache, type ControllerOutageReceipt } from "./controller-outage.js"; import { recoverInterruptedFeedback } from "./feedback-recovery.js"; import { normalizedSecrets } from "./redaction.js"; import { CREDENTIAL_NAMES } from "./types.js"; @@ -36,7 +40,9 @@ const port = required("PAPERCLIP_RUNNER_E2E_PORT"); const repositoryRoot = path.resolve(import.meta.dirname, "../.."); const paperclipCli = path.join(repositoryRoot, "tests/runner-e2e/server-entry.ts"); const executionIds: string[] = JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]"); -const selectedExecutions = usesInstalledCli(process.env) ? executionIds.map(id => { +const upgradeArtifacts = await readUpgradeArtifacts(process.env[upgradeArtifactKey]); +let artifactIndex = 0; +const selectedExecutions = usesInstalledCli(process.env) || upgradeArtifacts ? executionIds.map(id => { const execution = runnerMatrix.find(row => row.id === id); if (!execution) throw new Error("Unknown installed CLI proof execution"); return execution; @@ -48,8 +54,9 @@ const { } = runnerE2EServerControlPaths(temporaryRoot); const restartTimeoutMs = 180_000; const gracefulStopTimeoutMs = 30_000; -const installedRelease = process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI - ? installedReleaseLaunch(process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI) : null; +const initialCli = upgradeArtifacts?.artifacts.A.pins.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI ?? process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI; +const installedRelease = initialCli ? installedReleaseLaunch(initialCli) : null; +const servingArtifacts = new WeakMap(); const serverEnvironment = buildPaperclipServerEnvironment(installedRelease ? installedReleaseEnvironment(process.env, repositoryRoot, path.join(temporaryRoot, "provider-bin")) : process.env, { @@ -104,7 +111,7 @@ const definedServerEnvironment = Object.fromEntries( ), ); -await prepareSourceServerDependencies(repositoryRoot, definedServerEnvironment, usesInstalledCli(process.env)); +await prepareSourceServerDependencies(repositoryRoot, definedServerEnvironment, Boolean(installedRelease)); await Promise.all([ mkdir(path.dirname(logPath), { recursive: true }), @@ -122,6 +129,7 @@ let unexpectedChildFailure: Error | null = null; let shutdownSignal: NodeJS.Signals | null = null; let activeRestartRequestId: string | null = null; let feedbackRecovery: Promise | undefined; +let controllerDatabase: Awaited> | undefined; function recoverFeedbackBeforeShutdown() { return feedbackRecovery ??= (async () => { @@ -183,13 +191,24 @@ async function startServer() { throw new Error("Refusing to start Paperclip after wrapper shutdown"); } // Recheck bytes and dependency resolution on every controller restart. - const installed = await verifyInstalledCli(process.env, selectedExecutions); + const selected = upgradeArtifacts?.sequence[artifactIndex]; + if (upgradeArtifacts && !selected) throw new Error("Upgrade artifact sequence exhausted"); + const verified = selected ? await verifyUpgradeArtifact(upgradeArtifacts!.artifacts[selected], process.env, selectedExecutions) : undefined; + const installed = verified?.installed ?? await verifyInstalledCli(process.env, selectedExecutions); + const launch = installed ? installedReleaseLaunch(installed.entry) : null; + // The serving process must report its baked-in revision, never a test-supplied SHA. + const environment = { ...definedServerEnvironment }; + if (upgradeArtifacts) { + delete environment.PAPERCLIP_BUILD_COMMIT; + delete environment[upgradeArtifactKey]; + environment.GIT_CEILING_DIRECTORIES = upgradeArtifacts.artifacts[selected!].root; + } const candidate = spawn( process.execPath, installed ? [installed.entry, "onboard", "--yes", "--run"] : runnerE2ETypeScriptProcessArgs(repositoryRoot, paperclipCli, ["onboard", "--yes", "--run"]), { - cwd: installedRelease?.cwd ?? repositoryRoot, - env: definedServerEnvironment, + cwd: launch?.cwd ?? repositoryRoot, + env: environment, stdio: ["ignore", "pipe", "pipe"], // Playwright signals the wrapper's group. Keep that signal from bypassing // our single graceful stop; the launcher still tracks descendant groups. @@ -197,6 +216,7 @@ async function startServer() { }, ); child = candidate; + if (verified) servingArtifacts.set(candidate, verified.receipt); candidate.stdout?.on("data", (chunk) => { process.stdout.write(chunk); @@ -261,8 +281,13 @@ async function waitForHealth(candidate: ChildProcess) { const response = await fetch(healthUrl, { signal: AbortSignal.timeout(1_000), }); - if (response.ok) return; - } catch { + if (response.ok) { + const artifact = servingArtifacts.get(candidate); + if (artifact) assertServingArtifact(await response.json(), artifact); + return; + } + } catch (error) { + if (error instanceof Error && error.message.includes("serving server revision")) throw error; // The replacement process may still be booting. } await delay(250); @@ -291,54 +316,14 @@ async function waitForHealthToStop() { ); } -interface RestartRequest { - requestId: string; - controllerDowntimeMs?: number; - preserveRunner?: RestartRunnerIdentity; - crashAfterResult?: FinalizationCrashBinding; -} - -async function readRestartRequest(): Promise { - let encoded: string; - try { - encoded = await readFile(restartRequestPath, "utf8"); - } catch (error) { - if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; - throw error; - } - let value: unknown; - try { - value = JSON.parse(encoded); - } catch { - // The writer may not have completed its atomic replacement yet. - return null; - } - if (!value || typeof value !== "object" || Array.isArray(value)) return null; - const requestId = (value as { requestId?: unknown }).requestId; - if ( - typeof requestId !== "string" || - !/^[A-Za-z0-9._:-]{1,200}$/.test(requestId) - ) { - return null; - } - const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner; - const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs; - if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs) - || Number(controllerDowntimeMs) < 0 || Number(controllerDowntimeMs) > 300_000)) { - throw new Error("Controller downtime must be an integer between zero and five minutes"); - } - const crashAfterResult = (value as { crashAfterResult?: unknown }).crashAfterResult; - if (crashAfterResult !== undefined && preserveRunner !== undefined) throw new Error("A finalization crash cannot preserve a running provider"); - return { requestId, ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), - ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), - ...(crashAfterResult === undefined ? {} : { crashAfterResult: parseFinalizationCrashBinding(crashAfterResult) }) }; -} async function writeRestartAck( requestId: string, status: "ready" | "failed", message?: string, + controllerCacheDiscarded = false, crashBoundary?: Awaited>, + controllerOutage?: ControllerOutageReceipt, ) { const temporaryAckPath = `${restartAckPath}.${process.pid}.tmp`; await writeFile( @@ -346,20 +331,55 @@ async function writeRestartAck( `${JSON.stringify({ requestId, status, + controllerCacheDiscarded, completedAt: new Date().toISOString(), ...(message ? { message } : {}), ...(crashBoundary ? { crashBoundary } : {}), + ...(controllerOutage ? { controllerOutage } : {}), })}\n`, { encoding: "utf8", mode: 0o600 }, ); await rename(temporaryAckPath, restartAckPath); } -async function restartServer({ requestId, preserveRunner, crashAfterResult, controllerDowntimeMs = 0 }: RestartRequest) { +async function restartServer({ requestId, preserveRunner, discardControllerCache, crashAfterResult, controllerOutage, recoveryBinding, controllerDowntimeMs = 0 }: RestartRequest) { activeRestartRequestId = requestId; appendLog(`\nRestart request ${requestId}: stopping Paperclip\n`); const previous = child; if (!previous) throw new Error("No Paperclip server is available to restart"); + const discardCache = async () => { + const cache = path.join(paperclipHome, "instances", required("PAPERCLIP_INSTANCE_ID"), "runtime", "paperclip-runner"); + await retireControllerCache(cache, path.join(controlDirectory, `retired-controller-cache-${requestId}`)); + appendLog(`Restart request ${requestId}: old controller cache made unavailable\n`); + }; + if (controllerOutage) { + if (upgradeArtifacts) artifactIndex += 1; + const previousArtifact = servingArtifacts.get(previous); + const observer = recoveryBinding ? await createControllerRecoveryObserver(configPath, recoveryBinding) : undefined; + try { + const receipt = await replaceController({ requestId, previousPid: previous.pid!, outage: controllerOutage, + stop: async signal => { + if (signal === "SIGKILL") await stopServer.crashForRestart(previous); + else await stopServer(previous); + if (child === previous) child = null; + }, + waitForUnhealthy: waitForHealthToStop, discardCache, start: startServer, waitForReady: waitForHealth, + ...(previousArtifact ? { previousArtifact, replacementArtifact: () => child && servingArtifacts.get(child) || undefined } : {}), + cancelled: shutdownRequested, + afterStopped: observer?.stopped, + afterReady: observer ? (receipt, record) => observer.recovered(receipt, record, shutdownRequested) : undefined, + record: async receipt => { + const target = path.join(controlDirectory, `controller-outage-${requestId}.json`); + await writeFile(`${target}.tmp`, `${JSON.stringify(receipt)}\n`, { mode: 0o600 }); + await rename(`${target}.tmp`, target); + }, + }); + await writeRestartAck(requestId, "ready", undefined, true, undefined, receipt); + appendLog(`Restart request ${requestId}: Paperclip is healthy after ${Math.round(receipt.readyAfterMs!)}ms\n`); + activeRestartRequestId = null; + return; + } finally { await observer?.close(); } + } const crashBoundary = crashAfterResult ? await readFinalizationCrashBoundary(async route => { const response = await fetch(`http://127.0.0.1:${port}${route}`, { signal: AbortSignal.timeout(5_000), cache: "no-store" }); if (!response.ok) throw new Error("Finalization crash boundary could not be read"); @@ -377,6 +397,9 @@ async function restartServer({ requestId, preserveRunner, crashAfterResult, cont if (shutdownRequested()) throw new Error("Wrapper shutdown interrupted controller downtime"); await delay(Math.min(250, downtimeDeadline - Date.now())); } + if (discardControllerCache) { + await discardCache(); + } if (shutdownRequested()) { throw new Error("Wrapper shutdown interrupted the Paperclip restart"); } @@ -387,7 +410,7 @@ async function restartServer({ requestId, preserveRunner, crashAfterResult, cont if (shutdownRequested()) { throw new Error("Wrapper shutdown interrupted the Paperclip restart"); } - await writeRestartAck(requestId, "ready", undefined, crashBoundary); + await writeRestartAck(requestId, "ready", undefined, discardControllerCache, crashBoundary); appendLog(`Restart request ${requestId}: Paperclip is healthy\n`); activeRestartRequestId = null; } @@ -413,11 +436,17 @@ async function supervise() { }); // Postgres needs the socket itself; release immediately before child spawn. await databaseReservation?.close(); - await startServer(); + if (upgradeArtifacts || executionIds.some(id => id.endsWith(".recover-controller"))) { + // The wrapper owns this sibling process. Killing the serving child must + // leave Postgres available to the stopped-controller observer and successor. + controllerDatabase = await prepareControllerDatabase(configPath, temporaryRoot); + } + const initial = await startServer(); + if (upgradeArtifacts) await waitForHealth(initial); let lastRestartRequestId: string | null = null; while (!shutdownRequested()) { if (unexpectedChildFailure) throw unexpectedChildFailure; - const request = await readRestartRequest(); + const request = await readRestartRequest(restartRequestPath, executionIds); if (request && request.requestId !== lastRestartRequestId) { lastRestartRequestId = request.requestId; await restartServer(request); @@ -453,5 +482,10 @@ try { } } +try { await controllerDatabase?.cleanup(); } +catch (error) { + exitCode = 1; + appendLog(`Qualification database cleanup failed: ${error instanceof Error ? error.message : String(error)}\n`); +} await new Promise((resolve) => log.end(resolve)); process.exitCode = exitCode; diff --git a/tests/runner-e2e/shared-evidence.test.ts b/tests/runner-e2e/shared-evidence.test.ts new file mode 100644 index 00000000000..8963b635bfd --- /dev/null +++ b/tests/runner-e2e/shared-evidence.test.ts @@ -0,0 +1,96 @@ +import { mkdtemp, mkdir, readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { expect, it } from "vitest"; +import { replaceController, validControllerOutageReceipt } from "./controller-outage.js"; +import { packageEvidence } from "./evidence.js"; +import { runnerE2EServerControlPaths } from "./harness-env.js"; +import { copySharedEvidence } from "./shared-evidence.js"; + +it.each(["start", "readiness", "recovery"])("publishes the last outage phase after %s fails and temporary state is deleted", async phase => { + const root = await mkdtemp(path.join(tmpdir(), "outage-evidence-")); + const temporaryRoot = path.join(root, "temporary"); + const privateRoot = path.join(temporaryRoot, "private"); + const caseDir = path.join(privateRoot, "cases", "recover-controller"); + const uploadDir = path.join(root, "published"); + const { controlDirectory } = runnerE2EServerControlPaths(temporaryRoot); + const secret = "synthetic-evidence-credential"; + try { + await mkdir(controlDirectory, { recursive: true }); + await mkdir(caseDir, { recursive: true }); + await writeFile(path.join(privateRoot, "server.log"), "server stopped\n"); + await writeFile(path.join(controlDirectory, "restart-request.json"), JSON.stringify({ token: secret })); + await expect(replaceController({ + requestId: "attempt-1", previousPid: 100, outage: { signal: "SIGKILL", holdMs: 0 }, + stop: async () => {}, waitForUnhealthy: async () => {}, discardCache: async () => {}, + start: async () => { if (phase === "start") throw new Error(phase); return { pid: 200 }; }, + waitForReady: async () => { if (phase === "readiness") throw new Error(phase); }, + afterReady: async (receipt, record) => { + receipt.recoveryProgress = { bootId: secret }; + await record(); + throw new Error(phase); + }, + cancelled: () => false, + record: async receipt => { + await writeFile(path.join(controlDirectory, "controller-outage-attempt-1.json"), JSON.stringify(receipt)); + }, + })).rejects.toThrow(phase); + await copySharedEvidence(privateRoot, caseDir, temporaryRoot, [secret]); + const evidence = await packageEvidence({ privateDir: caseDir, uploadDir, secrets: [secret], expectPassScreenshot: false }); + await rm(temporaryRoot, { recursive: true, force: true }); + expect(evidence.leaks).toEqual([]); + const relative = "snapshots/controller-outages/controller-outage-attempt-1.json"; + expect(evidence.files).toContain(relative); + const text = await readFile(path.join(uploadDir, relative), "utf8"); + const receipt = JSON.parse(text); + expect(text).not.toContain(secret); + expect(receipt).toMatchObject({ requestId: "attempt-1", cacheDiscarded: true }); + if (phase === "start") expect(receipt).not.toHaveProperty("replacementPid"); + else expect(receipt.replacementPid).toBe(200); + if (phase === "recovery") expect(receipt.recoveryProgress).toEqual({ bootId: "[REDACTED]" }); + else { + expect(receipt).not.toHaveProperty("readyAfterMs"); + expect(validControllerOutageReceipt(receipt)).toBe(false); + } + expect(await readdir(uploadDir)).not.toContain("control"); + expect(evidence.files.some(file => file.includes("restart-request"))).toBe(false); + expect(await readFile(path.join(uploadDir, "server.log"), "utf8")).toBe("server stopped\n"); + } finally { await rm(root, { recursive: true, force: true }); } +}); + +it("keeps ordinary evidence collection working when no outage was requested", async () => { + const root = await mkdtemp(path.join(tmpdir(), "ordinary-evidence-")); + try { + await writeFile(path.join(root, "server.log"), "ordinary run\n"); + const caseDir = path.join(root, "case"); + await mkdir(caseDir); + await copySharedEvidence(root, caseDir, root, []); + expect(await readdir(caseDir)).toEqual(["server.log"]); + } finally { await rm(root, { recursive: true, force: true }); } +}); + +it("retains sanitized corrupt receipt evidence without losing healthy receipts or logs", async () => { + const root = await mkdtemp(path.join(tmpdir(), "corrupt-outage-evidence-")); + const temporaryRoot = path.join(root, "temporary"), privateRoot = path.join(temporaryRoot, "private"); + const caseDir = path.join(privateRoot, "case"), uploadDir = path.join(root, "published"); + const { controlDirectory } = runnerE2EServerControlPaths(temporaryRoot); + const secret = "synthetic-corrupt-receipt-secret"; + try { + await mkdir(controlDirectory, { recursive: true }); await mkdir(caseDir, { recursive: true }); + await writeFile(path.join(privateRoot, "server.log"), "original failure retained\n"); + await writeFile(path.join(controlDirectory, "controller-outage-a-broken.json"), `{"note":"${secret}"`); + await writeFile(path.join(controlDirectory, "controller-outage-z-good.json"), JSON.stringify({ requestId: "good", stoppedAfterMs: 10 })); + await copySharedEvidence(privateRoot, caseDir, temporaryRoot, [secret]); + const result = await packageEvidence({ privateDir: caseDir, uploadDir, secrets: [secret], expectPassScreenshot: false }); + await rm(temporaryRoot, { recursive: true, force: true }); + expect(result.leaks).toEqual([]); + const directory = path.join(uploadDir, "snapshots/controller-outages"); + const broken = JSON.parse(await readFile(path.join(directory, "controller-outage-a-broken.json"), "utf8")); + expect(broken).toMatchObject({ error: "invalid_json", source: "controller-outage-a-broken.json" }); + expect(validControllerOutageReceipt(broken)).toBe(false); + const raw = await readFile(path.join(directory, "controller-outage-a-broken.json.invalid.txt"), "utf8"); + expect(raw).not.toContain(secret); expect(raw).toContain("[REDACTED]"); + expect(JSON.parse(await readFile(path.join(directory, "controller-outage-z-good.json"), "utf8"))).toMatchObject({ requestId: "good" }); + expect(await readFile(path.join(uploadDir, "server.log"), "utf8")).toBe("original failure retained\n"); + } finally { await rm(root, { recursive: true, force: true }); } +}); diff --git a/tests/runner-e2e/shared-evidence.ts b/tests/runner-e2e/shared-evidence.ts new file mode 100644 index 00000000000..6c16051025b --- /dev/null +++ b/tests/runner-e2e/shared-evidence.ts @@ -0,0 +1,47 @@ +import { cp, mkdir, readFile, readdir, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { createHash } from "node:crypto"; +import { runnerE2EServerControlPaths } from "./harness-env.js"; +import { redactText, sanitizeJson } from "./redaction.js"; + +export async function copySharedEvidence(privateRoot: string, casePrivateDir: string, + temporaryRoot: string, secrets: readonly string[]) { + for (const relative of [ + "server.log", + "playwright.log", + "junit.xml", + "html-report", + "blob-report", + "playwright-output", + ]) { + await cp( + path.join(privateRoot, relative), + path.join(casePrivateDir, relative), + { recursive: true, force: true }, + ).catch((error: NodeJS.ErrnoException) => { + if (error.code !== "ENOENT") throw error; + }); + } + // The supervisor records each phase before it can fail. Copy its receipts + // independently of the successful-restart acknowledgement before cleanup. + const { controlDirectory } = runnerE2EServerControlPaths(temporaryRoot); + const entries = await readdir(controlDirectory, { withFileTypes: true }).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return []; + throw error; + }); + for (const entry of entries) { + if (!entry.isFile() || !/^controller-outage-[A-Za-z0-9._:-]{1,200}\.json$/.test(entry.name)) continue; + const raw = await readFile(path.join(controlDirectory, entry.name), "utf8"); + const snapshots = path.join(casePrivateDir, "snapshots", "controller-outages"); + await mkdir(snapshots, { recursive: true }); + let receipt: unknown; + try { receipt = JSON.parse(raw); } + catch { + // Never include parser excerpts: they can quote private receipt bytes. + receipt = { schema: "paperclip.e2e.controller-outage-parse-error/v1", source: entry.name, + error: "invalid_json", byteLength: Buffer.byteLength(raw), sha256: createHash("sha256").update(raw).digest("hex") }; + await writeFile(path.join(snapshots, `${entry.name}.invalid.txt`), redactText(raw, secrets), { mode: 0o600 }); + } + await writeFile(path.join(snapshots, entry.name), `${JSON.stringify(sanitizeJson(receipt, secrets), null, 2)}\n`, { mode: 0o600 }); + } +} diff --git a/tests/runner-e2e/upgrade-artifacts.test.ts b/tests/runner-e2e/upgrade-artifacts.test.ts new file mode 100644 index 00000000000..ed0469f4f4a --- /dev/null +++ b/tests/runner-e2e/upgrade-artifacts.test.ts @@ -0,0 +1,61 @@ +import { createHash } from "node:crypto"; +import { mkdir, mkdtemp, readFile, realpath, rm, symlink, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, expect, it } from "vitest"; +import { runnerMatrix } from "./catalog.js"; +import { assertServingArtifact, installedTreeSha256, readUpgradeArtifacts, verifyUpgradeArtifact, type UpgradeArtifact } from "./upgrade-artifacts.js"; + +const roots: string[] = []; +afterEach(async () => { await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); +const cell = runnerMatrix.find(e => e.id === "everyday-workflows.runner-codex.daytona.recover-controller")!; +async function fixture(revision = "a".repeat(40)): Promise { + const root = await realpath(await mkdtemp(join(tmpdir(), "upgrade-artifact-"))); roots.push(root); + const cli = join(root, "node_modules/paperclipai"); + const server = join(root, "node_modules/@paperclipai/server"); + for (const directory of [cli, server]) await mkdir(join(directory, "dist"), { recursive: true }); + await writeFile(join(cli, "package.json"), JSON.stringify({ name: "paperclipai", version: "1.2.3", type: "module", bin: { paperclipai: "./dist/index.js" } })); + await writeFile(join(server, "package.json"), JSON.stringify({ name: "@paperclipai/server", version: "1.2.3", type: "module", exports: { ".": { import: "./dist/index.js" } } })); + await writeFile(join(cli, "dist/index.js"), "// public cli"); + await writeFile(join(server, "dist/index.js"), `// public server ${revision}`); + await writeFile(join(server, "dist/build-info.json"), JSON.stringify({ commit: revision })); + const hash = async (path: string) => createHash("sha256").update(await readFile(path)).digest("hex"); + return { root, revision, treeSha256: await installedTreeSha256(root), evidence: "development-build", pins: { + PAPERCLIP_RUNNER_E2E_INSTALLED_CLI: join(cli, "dist/index.js"), + PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256: await hash(join(cli, "dist/index.js")), + PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT: server, + PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256: await hash(join(server, "dist/index.js")), + } }; +} +it("verifies distinct A/B artifacts and a real serving revision at the same package version", async () => { + const A = await fixture(); const B = await fixture("b".repeat(40)); + const manifest = join(tmpdir(), `upgrade-${process.pid}-${Date.now()}.json`); + try { + await writeFile(manifest, JSON.stringify({ schema: "paperclip.e2e.upgrade-artifacts/v1", artifacts: { A, B }, sequence: ["A", "B", "A"] })); + expect((await readUpgradeArtifacts(manifest))!.sequence).toEqual(["A", "B", "A"]); + for (const artifact of [A, B, A]) { + const verified = await verifyUpgradeArtifact(artifact, {}, [cell]); + expect(verified.receipt.evidence).toBe("development-build"); + expect(() => assertServingArtifact({ status: "ok", commit: artifact.revision }, verified.receipt)).not.toThrow(); + expect(() => assertServingArtifact({ status: "ok" }, verified.receipt)).toThrow("serving server revision"); + expect(() => assertServingArtifact({ status: "ok", commit: "c".repeat(40) }, verified.receipt)).toThrow("serving server revision"); + } + await writeFile(manifest, JSON.stringify({ schema: "paperclip.e2e.upgrade-artifacts/v1", artifacts: { A, B: A }, sequence: ["A", "B", "A"] })); + await expect(readUpgradeArtifacts(manifest)).rejects.toThrow("genuinely different"); + } finally { await rm(manifest, { force: true }); } +}); +it("detects changed transitive files even when CLI and server entrypoint hashes are unchanged", async () => { + const artifact = await fixture(); + await writeFile(join(artifact.root, "node_modules/transitive.js"), "changed migration or runtime"); + await expect(verifyUpgradeArtifact(artifact, {}, [cell])).rejects.toThrow("frozen tree"); +}); +it("rejects source checkouts, escaping links and a false build stamp", async () => { + const artifact = await fixture(); const other = await fixture("b".repeat(40)); + await symlink(other.root, join(artifact.root, "foreign")); + await expect(installedTreeSha256(artifact.root)).rejects.toThrow("escapes"); + await rm(join(artifact.root, "foreign")); + await mkdir(join(artifact.root, ".git")); + await expect(installedTreeSha256(artifact.root)).rejects.toThrow("outside a source checkout"); + await rm(join(artifact.root, ".git"), { recursive: true }); + await expect(verifyUpgradeArtifact({ ...artifact, revision: other.revision }, {}, [cell])).rejects.toThrow("build stamp"); +}); diff --git a/tests/runner-e2e/upgrade-artifacts.ts b/tests/runner-e2e/upgrade-artifacts.ts new file mode 100644 index 00000000000..11400f6e1be --- /dev/null +++ b/tests/runner-e2e/upgrade-artifacts.ts @@ -0,0 +1,99 @@ +/** Qualification-only artifact selection. Production recovery remains in the server. */ +import { createHash } from "node:crypto"; +import { createReadStream } from "node:fs"; +import { lstat, readFile, readdir, readlink, realpath } from "node:fs/promises"; +import { isAbsolute, join, relative, resolve, sep } from "node:path"; +import { verifyInstalledCli, type installedCliKeys } from "./installed-cli.js"; +import type { MatrixExecution } from "./types.js"; + +export const upgradeArtifactKey = "PAPERCLIP_RUNNER_E2E_UPGRADE_ARTIFACTS"; +export interface UpgradeArtifact { + root: string; + revision: string; + treeSha256: string; + evidence: "development-build" | "released"; + pins: Record<(typeof installedCliKeys)[number], string>; +} +export interface UpgradeArtifacts { + schema: "paperclip.e2e.upgrade-artifacts/v1"; + artifacts: { A: UpgradeArtifact; B: UpgradeArtifact }; + sequence: Array<"A" | "B">; +} +export interface ServingArtifact { + revision: string; + treeSha256: string; + version: string; + evidence: UpgradeArtifact["evidence"]; +} +const hex = (value: unknown, length: number): value is string => typeof value === "string" && new RegExp(`^[a-f0-9]{${length}}$`).test(value); +const inside = (root: string, file: string) => { + const part = relative(root, file); + return part !== ".." && !part.startsWith(`..${sep}`) && !isAbsolute(part); +}; + +/** Hash the complete installed tree, including transitive packages and migrations. + * Links are hashed as links and must resolve within the frozen installation. + * Stream large binaries; never buffer a whole installation in memory. + */ +export async function installedTreeSha256(root: string): Promise { + if (!isAbsolute(root) || resolve(root) !== root || await realpath(root) !== root) throw new Error("Upgrade artifact root must be canonical"); + const hash = createHash("sha256"); + async function visit(file: string): Promise { + const before = await lstat(file, { bigint: true }); + const name = relative(root, file).split(sep).join("/"); + if (before.isSymbolicLink()) { + if (!inside(root, await realpath(file))) throw new Error("Upgrade artifact link escapes the installation"); + hash.update(JSON.stringify([name, "link", await readlink(file)]) + "\n"); + } else if (before.isDirectory()) { + hash.update(JSON.stringify([name, "directory"]) + "\n"); + for (const child of (await readdir(file)).sort()) { + if (child === ".git") throw new Error("Upgrade artifacts must be installed builds outside a source checkout"); + await visit(join(file, child)); + } + } else if (before.isFile()) { + hash.update(JSON.stringify([name, "file", String(before.size), Number(before.mode & 0o777n)]) + "\n"); + for await (const chunk of createReadStream(file)) hash.update(chunk); + } else throw new Error("Upgrade artifacts must contain only regular files, directories and contained links"); + const after = await lstat(file, { bigint: true }); + if (before.ino !== after.ino || before.dev !== after.dev || before.size !== after.size || before.mtimeNs !== after.mtimeNs || before.ctimeNs !== after.ctimeNs) { + throw new Error("Upgrade artifact changed during verification"); + } + } + await visit(root); + return hash.digest("hex"); +} + +export async function readUpgradeArtifacts(filename: string | undefined): Promise { + if (!filename) return null; + const bytes = await readFile(filename); + if (bytes.length > 64 * 1024) throw new Error("Upgrade artifact manifest is too large"); + const value = JSON.parse(bytes.toString()) as UpgradeArtifacts; + if (value?.schema !== "paperclip.e2e.upgrade-artifacts/v1" || !Array.isArray(value.sequence) + || value.sequence.length < 3 || value.sequence.length > 9 || value.sequence[0] !== "A" + || value.sequence[1] !== "B" || value.sequence[2] !== "A" + || value.sequence.some((id, index) => (id !== "A" && id !== "B") || id === value.sequence[index - 1])) throw new Error("Upgrade qualification requires an explicit alternating A, B, A sequence"); + for (const id of ["A", "B"] as const) { + const a = value.artifacts?.[id]; + if (!a || typeof a.root !== "string" || !hex(a.revision, 40) || !hex(a.treeSha256, 64) + || !["released", "development-build"].includes(a.evidence) || !a.pins || typeof a.pins !== "object") throw new Error("Invalid pinned upgrade artifact"); + } + if (value.artifacts.A.root === value.artifacts.B.root || value.artifacts.A.revision === value.artifacts.B.revision + || value.artifacts.A.treeSha256 === value.artifacts.B.treeSha256) throw new Error("A and B must be genuinely different immutable artifacts"); + return value; +} + +export async function verifyUpgradeArtifact(artifact: UpgradeArtifact, env: NodeJS.ProcessEnv, executions: readonly MatrixExecution[]) { + const installed = await verifyInstalledCli({ ...env, ...artifact.pins }, executions); + if (!installed || !inside(artifact.root, installed.cliRoot) || !inside(artifact.root, installed.serverRoot)) throw new Error("Upgrade packages must belong to their pinned installation"); + if (await installedTreeSha256(artifact.root) !== artifact.treeSha256) throw new Error("Upgrade installation differs from its frozen tree"); + const build = JSON.parse(await readFile(join(installed.serverRoot, "dist/build-info.json"), "utf8")); + if (build.commit !== artifact.revision) throw new Error("Upgrade build stamp differs from its reviewed revision"); + return { installed, receipt: { revision: artifact.revision, treeSha256: artifact.treeSha256, + version: installed.version, evidence: artifact.evidence } satisfies ServingArtifact }; +} + +export function assertServingArtifact(health: unknown, artifact: ServingArtifact): void { + if (!health || typeof health !== "object" || (health as { commit?: unknown }).commit !== artifact.revision) { + throw new Error("The serving server revision does not match the verified upgrade artifact"); + } +}