From 247552fe0719dcc92f7b42c50b658be6f624986d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 1 Nov 2026 05:13:30 -0600 Subject: [PATCH] test(server): avoid deep traversal in durable file assertions (#18533) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Cloud web servers restore agent files from shared storage after replacement. > - The durability test checks restoration, corruption, cancellation and retry. > - Its generic equality assertions traverse every byte in many large buffers. > - The post-merge check for #18531 exceeded the 15-second test timeout. > - This change uses native buffer equality and keeps every byte check. ## Linked Issues or Issue Description Refs #18531. The post-merge Cloud-source verification run timed out in the new cold-folder test. Other shards passed. The unchanged test passed locally. This change removes avoidable assertion cost; it does not increase test or production timeouts. ## What Changed - Use `Buffer.equals` for the cold-folder content assertions. - Keep the concurrency, cancellation, cleanup, old-projection and retry assertions. - Record packaged-image proof, the hosted deployment state and the original CI failure in the existing plan. ## Verification - Eight durability tests passed against a real local PostgreSQL fixture. Object storage uses controlled asynchronous streams. - The same test file took 10.03 seconds before and 5.90 seconds after this change. The changed cold-folder case took 561 ms after the change. - The first local attempt could not start embedded PostgreSQL and ran no tests. The existing external test database enabled the comparison. - Original failed CI: https://github.com/paperclipai/paperclip/actions/runs/41296253723 . That failure remains in the qualification record. - Server typecheck and build passed. All latest-head PR checks passed, including the full test, typecheck, build and browser gates. One cancelled runner job was retried after its log showed an infrastructure shutdown; the retry passed. This PR changes no production code or database schema. ## Risks - A failed content check now reports a boolean mismatch instead of a large byte diff. Equality still covers buffer length and every byte. - Local timing is evidence for the fixture, not a general CI performance guarantee. - Hosted active-agent and rollout acceptance remain incomplete as recorded in the plan. ## Model Used OpenAI Codex, based on GPT-6. The exact runtime snapshot and context-window size are not exposed. Used code editing, shell execution and test analysis. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../2026-10-31-zero-downtime-web-deploys.md | 86 +++++++++++++++++++ .../__tests__/agent-file-durability.test.ts | 8 +- 2 files changed, 90 insertions(+), 4 deletions(-) diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md index ab753651e23..afbbe4d8282 100644 --- a/doc/plans/2026-10-31-zero-downtime-web-deploys.md +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -442,3 +442,89 @@ the new launcher. Existing live failure/cleanup receipts remain unchanged. Still required: active execution and continuation, cross-process follow-up, migrated packaged overlap, ten staging replacements including rollback, browser and sleep/wake/idle verification there, and one internal production canary. + +### 2026-11-01 10:12 UTC hosted qualification preparation + +Core maintenance #18529 merged as `d941ed3879`; bounded cold restore #18531 +merged as `1d348bbef0`. Both passed all current-head CI and Greptile 5/5 with +no unresolved review threads. This checkout now starts from the latter merge. +Published image and Cloud composition qualification are still pending. + +The earlier staging target was superseded: its historical local execution +workspaces require their own migration review. It was unpinned at 09:59 UTC +without a volume or deployment mutation by this chat. The superseded browser +fixture auto-assigned an agent despite the composer showing no assignee; +that attempted run failed. The task was then unassigned, paused and hidden. +Do not count this as successful agent execution or assume it incurred no cost. +Private setup and restoration receipts remain under `/private/tmp/paperclip-zdt-`. + +A dedicated staging fixture was provisioned through the normal signup API. +Its concrete host and stack identity remain in the private qualification receipt. +It is pinned through the audited user-managed API, and the existing operator +was added through the audited membership API. Normal organization-launcher +entry reached its dashboard. A separate synthetic fixture owner authenticated +through the normal Cloud entry-code/session flow. The tenant serves Core +`16787e4167` in Cloud distribution `cloud-41292158258-1`. + +The fixture contains a paused, wake-disabled agent with 34 synthetic files +(524,936 bytes), all read back through the API, and unassigned backlog task +with one saved comment. The data will exercise volume enrollment and +subsequent replacement. Receipt: `/private/tmp/paperclip-zdt-hosted-fixture.json`. +No hosted enrollment or overlap has yet run; the original volume is attached. +This controlled fixture does not qualify arbitrary historic local workspaces. +The saved E2E model credential remains an external blocker for paid-agent proof. + +The published Linux/amd64 maintenance image for `d941ed3879`, digest +`sha256:5370ef6c938a639d5994bb20fb85e0a07b9210b264389b2076cc5890f66ea686`, +passed the real-volume migration rehearsal. A legacy container wrote two files +and generated its decision-signing key on its volume. After confirmed shutdown, +the packaged maintenance entry migrated the files, verified shared objects and +exported that exact original key; unauthenticated status was denied and ordinary +app routes were absent. Both predecessor containers and the entire volume were +then deleted. A new empty-home container retained the original session and read +the migrated files. Two further replacements served 111 reads and 23 saves with +zero failures; unready replacement refusal and clean exits passed. Owned test +resources were removed. Receipt: `/private/tmp/paperclip-zdt-container-7r1Nwv/evidence.json`. +This small-file Core image rehearsal complements the multi-page migration tests; +it does not qualify Cloud routing, large cold folders or active agents. + +### 2026-11-01 10:29 UTC packaged Cloud proof + +The exact Cloud composition `cloud-41296664811-1`, Core `d941ed3879`, +digest `sha256:bce104e05623bd365740013c8fbb87680da7e152f9c94bd099884a9a5f6367cc`, +passed the same real-volume maintenance, key preservation, full volume removal +and empty-home replacement rehearsal: 121 reads, 25 saves, zero failures. +Receipt: `/private/tmp/paperclip-zdt-container-JBK8C7/evidence.json`. +This verifies the private Cloud packaging contains the working maintenance entry; +it does not substitute for Railway routing or active-agent acceptance. + +The dedicated hosted fixture passed a ten-second authenticated baseline with +19 reads, three saves, one WebSocket connection and all 34 file hashes intact. +Normal browser entry and an issue-title edit persisted. Receipt: +`/private/tmp/paperclip-zdt-hosted-smoke-s1Q33a/evidence.json`. +The first named-stack deployment was refused before creating a campaign because +an existing fleet campaign still reserved the fixture. That campaign completed +at 10:24:40 UTC and skipped the now-pinned fixture. Its operation lease was not +held; the campaign reservation was the actual conflict. The refused workflow +was explicitly retried, and the named-stack request +is deploying the qualified maintenance composition. No hosted enrollment has +started and no controller allowlist has changed yet. + +The named-stack request completed at 10:31 UTC with serving commit verification +for `d941ed3879`. A new continuous deployment campaign then started. The controller +restart guard correctly refuses the allowlist change until that campaign finishes. +The original volume remains attached and the fixture is pinned. + +Core `1d348bbef0` published its image, but its post-merge Cloud-source check +timed out after 15 seconds in the new cold-folder regression. All other shards +passed. The unchanged test passed locally against the owned PostgreSQL fixture; +the eight-test file took 10.03 seconds. Native `Buffer.equals` retains complete +byte equality while avoiding generic assertion traversal over 40 large buffers. +After that assertion-only change, all eight tests passed in 5.90 seconds, with +the cold-folder case taking 561 ms. The 15-second test timeout and 30-second +production deadline remain unchanged. The original CI failure is retained at +`/private/tmp/paperclip-zdt-final-source-verification-failed.log`; local before +and after logs are `/private/tmp/paperclip-zdt-buffer-test-{before,after}-pg.log`. +A first local attempt failed before tests because embedded PostgreSQL could not +start; the existing external test database enabled the comparison. The failed +post-merge shard was retried once while the assertion fix proceeds through review. diff --git a/server/src/__tests__/agent-file-durability.test.ts b/server/src/__tests__/agent-file-durability.test.ts index d198d614e26..6348f2c60a7 100644 --- a/server/src/__tests__/agent-file-durability.test.ts +++ b/server/src/__tests__/agent-file-durability.test.ts @@ -115,12 +115,12 @@ describe("durable agent file authority", () => { expect(await migrateAgentFiles(db, { companyId, apply: true, writersStopped: true })).toMatchObject({ pageReady: true }); await replaceFilesystem(); objects.readDelayMs = 25; - expect(await store().read(companyId, agentId, "nested/file-39.bin", actor())).toEqual(contents); + expect((await store().read(companyId, agentId, "nested/file-39.bin", actor()))?.equals(contents)).toBe(true); expect(objects.peakReads).toBeGreaterThan(1); expect(objects.peakReads).toBeLessThanOrEqual(8); expect(objects.activeReads).toBe(0); for (let index = 0; index < 40; index++) { - expect(await fs.readFile(path.join(root, `nested/file-${index}.bin`))).toEqual(contents); + expect((await fs.readFile(path.join(root, `nested/file-${index}.bin`))).equals(contents)).toBe(true); } // A corrupt replacement must preserve the old projection and await every @@ -130,9 +130,9 @@ describe("durable agent file authority", () => { await expect(store().read(companyId, agentId, "nested/file-39.bin", actor())).rejects.toThrow(/committed (size|digest)/); expect(objects.activeReads).toBe(0); expect((await fs.readdir(path.dirname(root))).filter(name => name.startsWith(".agent-files-"))).toEqual([]); - expect(await fs.readFile(path.join(root, "nested/file-39.bin"))).toEqual(contents); + expect((await fs.readFile(path.join(root, "nested/file-39.bin"))).equals(contents)).toBe(true); objects.corruptReads = false; - expect(await store().read(companyId, agentId, "nested/file-39.bin", actor())).toEqual(contents); + expect((await store().read(companyId, agentId, "nested/file-39.bin", actor()))?.equals(contents)).toBe(true); }); it("does not promote a rejected first save through subsequent local adoption", async () => {