From a18f6b85a392d439746ae35c74f72846dd7a9833 Mon Sep 17 00:00:00 2001 From: Dotta Date: Sun, 1 Nov 2026 06:43:42 -0600 Subject: [PATCH] docs: record hosted enrollment and live runtime qualification Preserve failed attempts, current verified evidence and remaining acceptance gates. Co-Authored-By: Paperclip --- .../2026-10-31-zero-downtime-web-deploys.md | 202 +++++++++++++++++- 1 file changed, 198 insertions(+), 4 deletions(-) diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index afbbe4d8282..6ee184aca91 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -63,10 +63,11 @@ Measure one-time setup separately from repeated attempts. Keep failed evidence. | Item | State | Next action | | --- | --- | --- | -| Prior prerequisites | #18403 now merged; current source fetched | Characterize remaining local authority paths | -| Core implementation | `/private/tmp/paperclip-zero-downtime-20261031`, durable files, readiness/drain and migration/cleanup implemented in draft PR #18518 | Cross-process commands, browser/assets, active Runner qualification | -| Cloud implementation | `/private/tmp/paperclip-cloud-zero-downtime-20261031`, persisted ephemeral shape and image/variable gates implemented in draft PR #1096 | Complete lifecycle guards, test and deploy qualification | -| Staging / production | Not selected or deployed | Record exact target, owner, artifact and campaign before action | +| Prior prerequisites | #18403 and all five storage workstreams merged | Retain the ephemeral filesystem contract | +| Core implementation | #18518, #18526, #18529, #18530, #18531 and #18533 merged; local packaged migration, browser overlap, cold restoration and durable command proofs recorded below | Complete live remote execution, accounting and later continuation qualification | +| Cloud implementation | #1096, #1098, #1099, #1100, #1101 and #1102 merged | Await automatic staging recovery revision, then verify repeated Railway overlap behavior | +| Staging | Original attempt has migrated data and preserved the signing key; volume detached and retained; failed final boot held in maintenance | Resume the same boot attempt after #1102, then qualify ten app-only replacements and lifecycle | +| Production | No canary mutation performed | Proceed only after the staging and active-agent acceptance gates pass | Keep PRs, tested revisions, artifact identities, evidence paths, deployment targets and unresolved gates here. A green CI/build/deploy status alone is not acceptance. @@ -528,3 +529,196 @@ and after logs are `/private/tmp/paperclip-zdt-buffer-test-{before,after}-pg.log A first local attempt failed before tests because embedded PostgreSQL could not start; the existing external test database enabled the comparison. The failed post-merge shard was retried once while the assertion fix proceeds through review. + +### 2026-11-01 11:07 UTC package compatibility prerequisite + +The post-merge source retry passed. The final Cloud composition for Core +`1d348bbef0` is `cloud-41298718800-1`, digest +`sha256:800cb8578368c63e33bdd048786ec66b3e7a9902f0f221d7fe41ea3c35134020`. +It passed the real-volume migration and empty-home rehearsal with 121 reads, +25 saves and zero failures; receipt: +`/private/tmp/paperclip-zdt-container-drdrQS/evidence.json`. + +The forward/rollback manifest comparison exposed a real packaging blocker. +The Agent Computer bundle differed only in its embedded Runner source revision +and source map, despite unchanged Runner inputs, SDKs and dependency lock. +The exact-package gate correctly requires maintenance for those two images. +Do not use them as proof of an app-only overlap pair. + +The Cloud source-selection owner now selects the last first-parent Runner input +change when the older reviewed ROM pin differs, then proves complete input +equality. The plugin and ROM builder share that selection and keep temporary +history alive through archive verification. Actual clean checkouts of both +released Core commits select `66e8fab9447aa21350f87ddec79fcd027f3e23ad`. +A real shallow-checkout probe selected the same source in 25.6 seconds without +modifying the caller. The focused suite passed 32 cases; full Cloud tests passed +4,830 cases with 129 gated skips and no failures. Rebuilt package equality and +hosted overlap remain unverified. Private receipts are +`/private/tmp/paperclip-zdt-canonical-source-{proof,shallow-proof}.json`. + +Core's assertion fix is in #18533, reviewed at 5/5. One CI shard received a +runner shutdown signal, so that cancelled shard and its downstream aggregate +gate were retried. The original cancellation log remains at +`/private/tmp/paperclip-zdt-buffer-pr-cancelled-job.log`. +No hosted enrollment has occurred. After the controller became quiet, the +qualified maintenance image was added to the staging allowlist without an +automatic deployment. An explicit controller configuration restart was then +submitted with a mutation-confirmed deployment identity. The prior busy-campaign +and pending-default checks produced no mutations. Private receipt: +`/private/tmp/paperclip-zdt-controller-allowlist-RzRa7k/receipt.json`. +The paid-agent credential blocker and all remaining acceptance criteria remain. + + +### 2026-11-01 11:15 UTC hosted enrollment response correction + +Core #18533 merged as `247552fe0719dcc92f7b42c50b658be6f624986d`; Cloud +#1099 merged as `1c5c5133c2d3740fdb716b26c31f038add273b55`. Both latest +heads passed all checks and review at 5/5 with no unresolved findings. Real +ROM-input generation also verified the complete source archive and Linux +dependency lock in 53.1 seconds from a clean checkout. + +The staging controller configuration restart succeeded with unchanged source +`316163b28`. The first real enrollment request at 11:08 UTC failed before +provider mutation with `Enrollment requires the owned Railway service domain`. +The registry correctly retains maintenance routing. The original web deployment +and attached volume were not stopped or changed. Railway's scoped domain query +returns the matching service/environment but a null `projectId`. Reproducing +that response in the existing lifecycle test fails identically. The correction +accepts the nullable project field from the project-scoped query while rejecting +explicit project, service or environment mismatches before provider mutation. +The original attempt identity is retained for retry after reviewed deployment. +Private failure receipt: `/private/tmp/paperclip-zdt-enrollment-controller-logs.json`. +No hosted overlap deployment or production canary has passed yet. + + +### 2026-11-01 11:25 UTC exact release pair qualified + +Cloud #1100 merged as `abcb017123013f2baa215fd665b4092495de00a1` after +4,831 local tests, 129 explicit skips, typecheck, golden-path smoke and all +latest-head CI checks passed, with 5/5 review and no unresolved findings. Its +automatic staging controller deployment is starting; enrollment retry waits for +that serving revision. The original held deployment and attached volume were +independently confirmed unchanged after the first failure. + +The rebuilt pair now has byte-identical Cloud bundles, SDKs and dependency lock: +`cloud-41300941768-1` (Core `1d348bbef0`) and `cloud-41301234904-1` (Core +`d941ed3879`). Both select the same canonical Runner source. The production +`ephemeralPackageOverlap` function accepts this pair. There is no database diff +between the two Core commits. Exact manifest receipt: +`/private/tmp/paperclip-zdt-canonical-pair-proof.json`. + +Each exact image passed its own packaged migration, complete predecessor-volume +deletion and two empty-home replacements against real PostgreSQL and S3: 118 +reads/24 saves and 116 reads/24 saves respectively, no failures, all owned +resources cleaned. Receipts: `/private/tmp/paperclip-zdt-container-woB7ci/evidence.json` +and `/private/tmp/paperclip-zdt-container-tMUSnD/evidence.json`. Both images were +added to the staging allowlist with deployment skipped, under the normal quiet +guards, before the reviewed controller rollout. The repeated hosted experiment +will invoke the existing fleet workflow driver with exact release references and +a single pinned fixture; it retains campaign/release/serving verification while +avoiding repeated GitHub scheduler waits. No hosted success is inferred from +these local receipts. Active-agent credentials remain blocked as previously noted. + + +### 2026-11-01 11:34 UTC maintenance entrypoint qualification + +The corrected controller is serving `abcb017123013f2baa215fd665b4092495de00a1`. +The first prepare reached the durable `preparing` phase and stopped the old +deployment, but Railway's remove call timed out. A read-only provider check +confirmed removal and retention of the attached volume, then the same attempt +resumed successfully to `migrating`. Its healthy maintenance process reported +`blocked` during configuration; no detach was attempted. + +Railway's start-command override replaces ENTRYPOINT, whereas the original local +Docker rehearsal replaced only CMD. The maintenance process therefore ran as root +and refused the original UID-1000-owned signing key. An explicit local entrypoint +override reproduced the same blocked status and captured root process identity +and key ownership, without reading key contents into evidence. Failed receipt: +`/private/tmp/paperclip-zdt-container-5nGSUs/evidence.json`. + +Cloud #1101 preserves Core's init and entrypoint wrapper in the maintenance +command. With actual entrypoint replacement, the corrected packaged rehearsal +passed original-key preservation, volume deletion and later empty-home recovery: +116 reads, 24 saves, no failures, cleanup confirmed in +`/private/tmp/paperclip-zdt-container-POFr0C/evidence.json`. Twelve lifecycle tests, +root typecheck, the full 4,831-pass suite and ordinary golden-path smoke passed. +The PR is awaiting CI/review; hosted preparation will resume with the same +attempt after deployment. The original live volume remains attached and retained. +No hosted overlap or active-agent acceptance has passed. + +### 2026-11-01 11:48 UTC reviewed entrypoint fix + +Cloud #1101 merged as `76f2dedf076ed139af3a7136e988c6748589d50b` with +all latest-head checks passing and no unresolved review findings. The exact image +currently held in staging also passed the corrected Railway-entrypoint rehearsal: +118 reads, 24 saves, zero failures, original signing authority retained, predecessor +volume deleted and all owned test resources cleaned. Private receipt: +`/private/tmp/paperclip-zdt-container-vyIrsD/evidence.json`. + +Automatic staging controller workflow `41302710585` passed preflight and is +waiting for the ordinary controller coordination gate. The concurrent app campaign +has completed 16 of 17 targets with one still running; the pinned qualification +fixture is excluded. No cancellation or coordination bypass is permitted. The +serving controller remains `abcb017123013f2baa215fd665b4092495de00a1`. +Enrollment must resume with its original attempt identity after the new controller +is serving. The active-model credential remains unavailable; no paid-agent result +or hosted overlap acceptance is inferred from the local container proofs. + + +### 2026-11-01 12:34 UTC live enrollment and runtime admission findings + +Cloud #1101 reached staging at 12:05 UTC. The owning enrollment attempt completed +migration and authenticated eligibility: 34 agent objects, 12 skill versions, +zero blockers. The original signing key was stored before the volume detached. +The original volume remains retained. Final app boot failed because Railway left +the maintenance start command unchanged when the provider sent null. + +The same attempt exposed delayed detach reads, automatic maintenance deployment +after detach, and successful deployment stops whose API responses timed out. +Cloud #1102 handles those provider transitions with bounded ownership-checked +reconciliation, an empty-string start-command reset, and app reconfiguration on +same-attempt boot retry. Root typecheck, 4,835 tests (129 gated skips), ordinary +routing/sleep smoke and 352 focused provider tests passed. Review/deployment is +pending. Tenant routing remains in maintenance; no ten-roll acceptance is claimed. + +The user supplied a working credential location; the OpenAI authentication gate +is resolved without exposing the key. Two live attempts exposed premature runtime +admission: fresh run rows still say legacy while workspace preparation runs, +before the native runtime selector commits its decision. Workspace preparation +must validate storage authority; runtime selection must validate the resolved +native warm lifecycle. Two focused regressions failed before this source fix; +all 156 workspace/runtime/ephemeral-contract tests passed after it. Full typecheck +and build passed; the full test suite and rebuilt live cell are running. + +A third live attempt mistakenly used the old compiled server and reproduced the +old guard. That retained failure is not evidence against the source fix. The +launcher now records the compiled server hash and rejects that stale guard. +All three completed attempts revoked their owned Daytona credentials and deleted +their disposable object buckets. No paid-agent acceptance has passed yet. + + +### 2026-11-01 12:43 UTC recovery fix merged and live cleanup boundary + +Cloud #1102 merged as `461c4aea28a1ffc31b1175db42158338dd9de1ae` after +4,853 passing tests (129 gated skips), typecheck, routing/sleep smoke, latest-head +required CI and 5/5 review with no unresolved threads. The optional security +scanner was inconclusive because its sandbox could not start; a rerun request +returned 404. Automatic staging deployment is workflow `41307729362`. + +The rebuilt live cell completed two native Codex/Daytona runs. Its first web +replacement then failed while deleting an immutable skill-cache directory in the +stopped predecessor home. The attempt remains failed, with partial estimated LLM +cost $0.1072965 and incomplete usage coverage. The shared harness cleanup now +makes stopped owned directories removable without following symlinks; seven +focused regressions and the E2E typecheck pass. No receipt is accepted without +whole-home deletion. Its leaked remote fixture is being reconciled through exact +company/environment labels and recorded lease IDs; the first delete was refused +while the sandbox was stopping. Original attempt evidence is retained. + +Required broad checks exposed separate failures: `pnpm test:run` stopped at a +15-second chat integration timeout (338 passed); that exact test passed in +isolation without changing its limit. Product E2E unit checks reported 5,898 +passes and 13 failures in untouched onboarding expectations, operating receipts, +Pi catalog assertions, workflow security assertions and embedded PostgreSQL +startup. These are not counted as passes; baseline attribution is pending. +Workspace typecheck/build and all 156 affected Core tests remain passing.