From 006d5691a6245d5a710e25ba3df6d609754e163a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Sun, 1 Nov 2026 01:58:54 -0600 Subject: [PATCH] feat(server): support volume-free web server handoff (#18518) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/agent-files.md | 68 ++- .../2026-10-31-zero-downtime-web-deploys.md | 259 +++++++++ maps/agent-access-map/instance-operations.md | 3 + maps/feature-map/instance-operations.md | 6 + packages/db/src/client.ts | 24 + packages/db/src/index.ts | 2 + .../db/src/migrations/0422_eminent_darwin.sql | 13 + ...{0417_snapshot.json => 0422_snapshot.json} | 511 +++++++++++++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/agent_file_roots.ts | 19 + packages/db/src/schema/index.ts | 1 + packages/db/src/work-signals.ts | 47 +- packages/shared/src/types/live.ts | 2 +- server/scripts/bundle-runtime.mjs | 7 +- server/scripts/bundle-runtime.test.mjs | 9 + .../agent-directory-working-copies.test.ts | 19 +- .../__tests__/agent-file-durability.test.ts | 178 ++++++ .../agent-instructions-service.test.ts | 5 +- .../delivery-work-coordinator.test.ts | 18 + .../deployment-notifications.test.ts | 102 ++++ .../ephemeral-server-contract.test.ts | 42 ++ .../__tests__/ephemeral-server-data.test.ts | 42 ++ server/src/__tests__/idle-admission.test.ts | 9 +- server/src/__tests__/liveness.test.ts | 20 + .../native-cancellation-delivery.test.ts | 78 +++ .../native-runtime-response-delivery.test.ts | 126 +++++ server/src/__tests__/openapi-routes.test.ts | 3 +- .../src/__tests__/retained-ui-assets.test.ts | 68 +++ .../server-startup-feedback-export.test.ts | 2 +- server/src/app.ts | 10 + server/src/index.ts | 64 ++- server/src/maintenance/agent-files.ts | 41 ++ server/src/middleware/idle-admission.ts | 37 +- server/src/realtime/live-event-delivery.ts | 7 +- server/src/realtime/live-events-ws.ts | 5 +- server/src/realtime/runner-prp-ws.ts | 3 +- server/src/retained-ui-assets.ts | 77 +++ server/src/routes/agents.ts | 6 +- server/src/routes/openapi.ts | 21 + server/src/routes/ready.ts | 18 + server/src/services/agent-configuration.ts | 2 + server/src/services/agent-file-durability.ts | 184 +++++++ server/src/services/agent-file-maintenance.ts | 98 ++++ server/src/services/agent-file-store.ts | 21 +- server/src/services/agent-hiring.ts | 5 + .../services/agent-instruction-revisions.ts | 5 + .../agent-instruction-working-copies.ts | 2 +- server/src/services/agent-instructions.ts | 78 ++- server/src/services/built-in-agents.ts | 11 +- .../src/services/delivery-work-coordinator.ts | 14 +- .../services/delivery-work-notifications.ts | 4 + .../src/services/deployment-notifications.ts | 100 ++++ .../src/services/ephemeral-server-contract.ts | 76 +++ .../src/services/heartbeat/recovery.test.ts | 23 + server/src/services/heartbeat/recovery.ts | 8 +- .../heartbeat/runtime-selection.test.ts | 21 +- .../services/heartbeat/runtime-selection.ts | 5 +- .../heartbeat/workspace-preparation.ts | 7 +- .../native-runtime/native-active-runs.ts | 10 + .../native-cancellation-delivery.ts | 88 +++ .../native-runtime-response-delivery.ts | 141 +++++ .../native-session-executor.test.ts | 20 +- .../native-runtime/native-session-executor.ts | 35 +- .../native-runtime/runtime-context.ts | 2 +- .../services/plugin-agent-runtime-state.ts | 10 +- server/src/services/remote-workspaces.ts | 24 + server/src/services/task-admission.ts | 4 + server/src/shutdown.test.ts | 14 + server/src/shutdown.ts | 23 +- server/src/startup-recovery-state.ts | 2 +- tests/runner-e2e/README.md | 25 + .../runner-e2e/rolling-deployment-browser.ts | 64 +++ .../runner-e2e/rolling-deployment-process.ts | 94 ++++ .../rolling-deployment-proxy.test.ts | 59 ++ tests/runner-e2e/rolling-deployment-proxy.ts | 80 +++ tests/runner-e2e/rolling-deployment-smoke.ts | 301 +++++++++++ .../rolling-deployment-storage-proxy.ts | 58 ++ 77 files changed, 3596 insertions(+), 101 deletions(-) create mode 100644 doc/plans/2026-10-31-zero-downtime-web-deploys.md create mode 100644 packages/db/src/migrations/0422_eminent_darwin.sql rename packages/db/src/migrations/meta/{0417_snapshot.json => 0422_snapshot.json} (99%) create mode 100644 packages/db/src/schema/agent_file_roots.ts create mode 100644 server/src/__tests__/agent-file-durability.test.ts create mode 100644 server/src/__tests__/deployment-notifications.test.ts create mode 100644 server/src/__tests__/ephemeral-server-contract.test.ts create mode 100644 server/src/__tests__/ephemeral-server-data.test.ts create mode 100644 server/src/__tests__/native-cancellation-delivery.test.ts create mode 100644 server/src/__tests__/native-runtime-response-delivery.test.ts create mode 100644 server/src/__tests__/retained-ui-assets.test.ts create mode 100644 server/src/maintenance/agent-files.ts create mode 100644 server/src/retained-ui-assets.ts create mode 100644 server/src/routes/ready.ts create mode 100644 server/src/services/agent-file-durability.ts create mode 100644 server/src/services/agent-file-maintenance.ts create mode 100644 server/src/services/deployment-notifications.ts create mode 100644 server/src/services/ephemeral-server-contract.ts create mode 100644 server/src/services/native-runtime/native-cancellation-delivery.ts create mode 100644 server/src/services/native-runtime/native-runtime-response-delivery.ts create mode 100644 tests/runner-e2e/rolling-deployment-browser.ts create mode 100644 tests/runner-e2e/rolling-deployment-process.ts create mode 100644 tests/runner-e2e/rolling-deployment-proxy.test.ts create mode 100644 tests/runner-e2e/rolling-deployment-proxy.ts create mode 100644 tests/runner-e2e/rolling-deployment-smoke.ts create mode 100644 tests/runner-e2e/rolling-deployment-storage-proxy.ts diff --git a/doc/agent-files.md b/doc/agent-files.md index 5dc4d180708..2f6c83c46e1 100644 --- a/doc/agent-files.md +++ b/doc/agent-files.md @@ -13,7 +13,29 @@ or a responsible user alone does not grant peer access. ## Layout -The canonical host directory keeps its existing physical location: +`PAPERCLIP_DURABLE_AGENT_FILES=true` opts managed files into shared storage. This +requires the database migration that creates `agent_file_roots` and the existing +S3 application-storage configuration. Postgres holds the current manifest; S3 +holds immutable file bytes. Saves publish their bytes before committing the +manifest, and the local `instructions/` directory becomes a disposable projection. +An existing durable head remains authoritative when the opt-in is disabled. +Disabling S3 or rolling back to a binary unaware of durable heads is unsupported. + +For existing installations, import from the server that still owns the original +files while writers are stopped, and verify every managed agent before removing +its filesystem. A replacement that has no durable head and cannot access the +original managed directory returns `AGENT_FILES_MIGRATION_REQUIRED`; it must not +publish an empty replacement. The `file-sync/durable-adoption.json` receipt retains +the pre-write import manifest across a failed first transaction. It is only a +local import retry aid; after the head commits, a replacement requires neither +that receipt nor any other file from the old server. + +The storage opt-in is not a zero-downtime deployment qualification. Readiness, +HTTP draining, active execution, event delivery, Cloud rollout, and rollback have +separate acceptance checks. Superseded and uncertain-upload objects currently +remain retained until the explicit bounded object cleanup described below. + +The host directory keeps its existing physical location: ``` /companies//agents// @@ -84,9 +106,11 @@ and, for an oversized file, its path. The previous saved folder is used on the next run. The temporary run copy is discarded, including on a limit failure; there is no retained recovery archive or partial-save option. Transient sync failures get up to three attempts at the stop boundary before cleanup and an -explicit failure receipt. Individual file writes are atomic, but an I/O failure +explicit failure receipt. With local storage, individual file writes are atomic, but an I/O failure partway through a sync can leave some files updated; a failed receipt does not -claim whole-folder success. +claim whole-folder success. With shared storage, the Postgres manifest commits +the whole tree atomically; an upload or transaction failure leaves the previous +head authoritative and invalidates the uncommitted local projection. Sync failures are diagnostics for the affected run, not errors on the current files in the Instructions Editor. Historical failures remain in the run log; @@ -300,3 +324,41 @@ remove it and save a new small file. All tasks must succeed, with warnings visible in run details while full and cleared after cleanup. Run results, including unavailable credentials, must be reported separately from unit or matcher results; a passing matcher does not prove a live run. +### Explicit migration and object cleanup + +Run `server/src/maintenance/agent-files.ts` with the instance's database, +storage, secrets and original `PAPERCLIP_HOME`. The default migration phase is +read-only. Before applying, stop ingress and execution writers on that instance +and set `PAPERCLIP_DURABLE_AGENT_FILES=true`: + +```sh +pnpm exec tsx server/src/maintenance/agent-files.ts --phase=migration +pnpm exec tsx server/src/maintenance/agent-files.ts --phase=migration --apply --writers-stopped +``` + +The production runtime build includes the same command without a TypeScript +loader: `node server/dist-runtime/agent-files.mjs`. Its read-only +`--phase=eligibility` validates the ephemeral configuration and checks all durable +data prerequisites. A failing eligibility or applied migration page exits nonzero. +Eligibility is a point-in-time check; keep writers stopped through volume removal. + +Follow each returned `next` cursor with `--after=`. All pages must report +`pageReady: true` before removing the volume. A missing legacy directory, external +instructions or active execution prevents migration; resolve that condition on +the original host. Repeating apply verifies/restores the committed projection. +This migrates agent files only; other storage families have their own eligibility +checks. Keep the original volume until the deployment migration is qualified. + +Object cleanup is an explicit bounded command, with dry-run as the default: + +```sh +pnpm exec tsx server/src/maintenance/agent-files.ts --phase=objects --company= --agent= +``` + +Add `--apply` to delete eligible objects, and follow `next` with `--after` until +it is null. Objects younger than 30 days, current-head references, and every +object belonging to an agent without a committed head are retained. The latter +protects interrupted one-time imports. Deleted-agent objects become eligible +after the same grace. Deletion checks the current head under the agent lock and +removes all versions. There is no automatic idle scan; operators own scheduling +this maintenance during the bounded rollout. diff --git a/doc/plans/2026-10-31-zero-downtime-web-deploys.md b/doc/plans/2026-10-31-zero-downtime-web-deploys.md new file mode 100644 index 00000000000..88336ec4224 --- /dev/null +++ b/doc/plans/2026-10-31-zero-downtime-web-deploys.md @@ -0,0 +1,259 @@ +# Zero-downtime web deployments + +Date: 2026-10-31. Status: implementation in progress; local HTTP/storage overlap, +failure recovery and graceful drain passed. Cloud shape implementation underway. + +## Outcome and endpoint + +Keep the main Paperclip web/API usable through an app-only Railway deployment. +A person saves or answers while an agent works, sees that execution finish once, +and continues with its saved result after both webserver replacements. Finish at +reviewed/merged changes and a verified forward deploy and compatible rollback on +one internal production canary. Fleet-wide production rollout and online schema +migrations are deferred. One-time maintenance for volume migration is allowed. + +## Acceptance + +| Case | Required evidence | +| --- | --- | +| Fresh and migrated data | Separate empty server homes share only Postgres, object storage and secrets; acknowledged bytes remain readable | +| A/B overlap | Continuous requests have no deploy-caused failures or lost acknowledged writes; no hidden client retries | +| Cross-process commands | Stop, answer and follow-up arriving at B reach execution owned by A through durable authority | +| Browser | Existing tab reconnects within five seconds and loads a previously unopened page after replacement | +| Agent | Same remote execution; one completion and accounting result; independently verified files | +| Later continuation | Another empty-home replacement after completion can continue the retained session | +| Failure and rollback | Unready B leaves A serving; compatible A/B/A rollback uses current shared data | +| Lifecycle | Repair and sleep/wake retain volume-free shape; idle database suspension remains possible | + +Crash recovery is qualified separately from planned zero-downtime deployment. + +## Delivery units and ownership + +The lead in this chat owns integration and target deployment; no other deployment +owner has been assigned. Preserve the existing five storage workstreams and their +evidence. PR #18403 is merged; implementation starts from Core `34f8d25089` and +Cloud `936e23c8` in isolated `codex/zero-downtime-web-deploys` worktrees. + +1. Extend existing controller replacement / Product E2E owners with a two-server + proxy, disjoint writable roots, exact artifact identities and retained evidence. +2. Close remaining authoritative agent-file and provider-backup storage gaps using + existing storage/authorization owners, Postgres manifests and immutable objects. +3. Add readiness and bounded HTTP/controller handoff; durable cross-process work + hints, ownership and live update reconciliation; retained compatible UI assets. +4. Add persisted Cloud `single-ephemeral` shape, explicit eligibility, migration, + rollout, rollback, repair and sleep/wake behavior through the existing provider + and campaign owners. Defaults: readiness `/api/ready`, startup 600 seconds, + overlap 5 seconds, draining 90 seconds, retained volume/assets 30 days. +5. Qualify locally, then ten consecutive successful staging deployments including + rollback, then a production canary forward deploy and rollback. + +Existing active work must not block replacement HTTP readiness. Shared queues and +row ownership remain durable truth; notifications only accelerate reconciliation. +No permanent idle polling or idle database listeners. Ineligible stacks keep their +current path. Migrated stacks cannot accept disk-dependent configurations. Rollback +never restores stale local volume state after shared state advances. + +## Feedback and current state + +First experiment: create and save through A, start empty-home B against the same +disposable Postgres/S3, read through B, switch traffic, then SIGTERM A under writes. +Qualify harness identifiers, evidence and cleanup before introducing live agents. +Next use the real Runner with deterministic provider, then Codex/Claude Daytona. +Measure one-time setup separately from repeated attempts. Keep failed evidence. + +| Item | State | Next action | +| --- | --- | --- | +| Prior prerequisites | #18403 now merged; current source fetched | Characterize remaining local authority paths | +| Core implementation | `/private/tmp/paperclip-zero-downtime-20261031`, durable files, readiness/drain and migration/cleanup implemented in draft PR #18518 | Cross-process commands, browser/assets, active Runner qualification | +| Cloud implementation | `/private/tmp/paperclip-cloud-zero-downtime-20261031`, persisted ephemeral shape and image/variable gates implemented in draft PR #1096 | Complete lifecycle guards, test and deploy qualification | +| Staging / production | Not selected or deployed | Record exact target, owner, artifact and campaign before action | + +Keep PRs, tested revisions, artifact identities, evidence paths, deployment targets +and unresolved gates here. A green CI/build/deploy status alone is not acceptance. + +### Local characterization, 2026-10-31 + +Current source `34f8d25089` with the new source-mode harness: A ready in 12.7s, +B in 10.9s, handoff including shutdown/deletion in 2.5s. 96 reads and 20 +acknowledged comment writes passed across overlap; every acknowledged comment +survived once. A saved managed agent file was readable through B only while A's +directory remained; after deleting A's home the same B request returned 404. +The stored absolute path allowed B to read A's directory on the same host. +Overall result: **failed**, with agent-file durable authority a demonstrated +prerequisite. No browser, live agent, managed-cloud auth or packaged build proof. + +Private evidence: `/private/var/folders/74/zhncxl_j05xb5_r6_vfbk73h0000gn/T/paperclip-runner-e2e-4Q7BmG/rolling-deployment-evidence.json`. +Setup failures are retained: host embedded Postgres could not initialize; +the first full-server boot lacked the fresh SDK build. Isolated Docker Postgres +and a built SDK resolved those setup failures. No product assertions were reached +in those attempts. The proxy's in-flight write handoff regression passed. + +Owned temporary dependencies: Docker containers `paperclip-zdt-postgres-20261031` +and `paperclip-zdt-s3-20261031`, loopback only. Connection files under +`/private/tmp/paperclip-zdt-{database,storage}.json` are private and must never be +committed or uploaded. Each attempt cleans its own database and bucket. Retire +these containers and credential files when local qualification no longer needs them. + +### Durable agent-file implementation + +The agent-file service owns a current Postgres manifest and immutable S3 objects. +Managed roots are disposable projections; existing authorization, hash conflicts, +runtime directory merges and activity logging keep their owners. A durable head +remains authoritative if `PAPERCLIP_DURABLE_AGENT_FILES` is subsequently disabled. +Legacy local data must be adopted on its owning server before filesystem removal. +Separate bounded file-transfer connections protect ordinary HTTP and controller +renewals during S3 stalls. No idle polling or listeners were added. + +Source-mode evidence (base `34f8d25089` plus uncommitted changes): + +- `paperclip-runner-e2e-A05NfC`: 76 reads/16 acknowledged writes, deletion of A's + full home, saved agent file recovered by B; all checks passed. +- `paperclip-runner-e2e-NcNG3F`: bidirectional A/B edits and stale-write 409 also + passed. No browser or active-agent claim. +- `paperclip-runner-e2e-zQMAji`: both transfer connections deliberately held; + unrelated task read completed in 20ms. A failed-write read unexpectedly returned + 200, so this expanded run failed and is retained for diagnosis. Fresh C did + recover the primary saved file after deleting both previous homes. + +Each directory above is under +`/private/var/folders/74/zhncxl_j05xb5_r6_vfbk73h0000gn/T/` and contains +`rolling-deployment-evidence.json`. All owned attempt databases/buckets were +cleaned. Exact lockfile dependencies were installed after initial characterization; +repeat affected qualification before relying on a final receipt. + +The failed first write exposed rollback followed by re-import of dirty local +bytes. A private pre-mutation import receipt fixes that retry; committed heads +remain solely in Postgres. The harness also now explicitly seeds process-agent +instructions through the canonical API (agent creation ignores that field). +The corrected run `paperclip-runner-e2e-Xdn96H` passed all eight checks. + +Latest source-mode receipt `paperclip-runner-e2e-eGJIqp` passed all nine checks: +114 reads, 23 acknowledged writes, 2,508ms handoff, 31ms unrelated task read while +both storage connections were held. An admitted file save finished after SIGTERM; +C restored both saved files after A and B homes were removed. Cleanup confirmed. +The receipt records the source patch and lockfile digests. No browser, managed +auth, distinct packaged releases or live-agent qualification is claimed. + +An earlier passing drain receipt `paperclip-runner-e2e-iW2OCt` took 32 seconds: +Node's listener-close callback also waited for upgraded sockets. HTTP drain now +uses the existing request/promise tracker, while browser and agent sockets retain +their own shutdown owners. Readiness becomes unavailable immediately on SIGTERM. + +Verification: full workspace typecheck passed; HTTP drain/idle/readiness 39 tests, +shutdown/WebSocket 28 tests, instructions service/routes 39 tests, and durable +authority/migration/retention six integration cases passed. The repository suite +is running with disposable external PostgreSQL for server tests. Broad storage +regressions initially had five race-fixture failures after the file pool moved; +the affected race cases were corrected to intercept that pool and passed. + +Agent object cleanup is explicit bounded maintenance, retaining objects for at +least 30 days and preserving all objects while an agent has no durable head. +It checks current authority under the writer lock and removes all object versions. +This introduces no idle polling. Migration requires stopped writers and refuses +external roots, active runs, or missing legacy files. Every migration page must +pass; no partial page is an instance-wide eligibility report. + +### Browser and overlap commands, 2026-11-01 + +Receipt `paperclip-runner-e2e-oYdmER` passed all ten checks using independent +compiled server/UI directories and workspace dependencies: 91 reads, 19 writes, +2,870ms handoff, 1,013ms browser reconnect and 23ms task read during the S3 outage. +The original tab opened Audit after A's entire home/build was deleted, loading +its old lazy chunk from S3 because that chunk was absent from B's directory. +Zero page errors; cleanup confirmed. The preceding `WWdWtO` attempt failed on +an outdated harness label (Activity versus Audit); it remains failed evidence. +This is local-trusted auth, not a released-container or active-agent proof. + +Core now validates ephemeral configuration and existing durable data before +readiness, rejects future local-only execution/instructions, initializes new +agent manifests transactionally, and publishes/serves retained immutable UI +assets. PostgreSQL work hints listen only during handoff or active execution. +Durable Stop and runtime-response delivery use existing run authority; request +lifecycle, company membership and privileged approval access are rechecked at +consumption. A's command workers remain alive through controller handoff; +committing the released lease notifies B. Duplicate local Stops join cleanup. +Six PG command-delivery cases passed with a simulated provider; 33 targeted +Stop/recovery cases and 27 coordinator cases passed. Real A/B provider execution, +high-level question delivery and same-turn steering remain unqualified. + +Cloud's full suite passed 4,778 cases with 129 gated skips before the latest fresh +canary selection change; its three affected provider cases also passed. Runtime +and direct-listener credentials use the same Neon application role in separate +vault refs. Persisted ephemeral shape survives repair/wake/rollback; explicit +fresh canary IDs leave fleet defaults and existing volumes unchanged. No stack +has been enrolled, detached, deployed or charged for a new hosted test. + +The full Core build and typecheck passed before the newest command changes. +The broad suite is still running; it has reported two failures (native bootstrap +Stop's direct embedded-DB fixture, and a workspace-reference conflict assertion). +Do not claim a green full suite. The server has been rebuilt with the current +command changes; packaged maintenance entry is being verified separately. + +Remaining gates: large cold directories, real Runner A/B overlap (including +Stop/answer/follow-up), complete bounded shutdown, packaged-release overlap, +existing-volume maintenance orchestration, exact staging/production targets, +review/merge, ten staging deploys and production forward/rollback. Draft PRs are +[Core #18518](https://github.com/paperclipai/paperclip/pull/18518) at `cf31fa7620` and +[Cloud #1096](https://github.com/paperclipai/paperclip-cloud/pull/1096) at `387ebb37`. +Cloud CI passed; Core CI and review corrections are underway. No merge or deployment yet. The same owned worktrees and acceptance remain. + +Migration must preserve the original `decision-signing.key` as the stable +`PAPERCLIP_DECISION_SIGNING_SECRET` before removing the volume. Ephemeral startup +now refuses a missing/short value. Fresh Cloud provisioning must supply a stable +key too; the storage master key alone does not preserve decision signatures. + +Review round 1 corrected missing decision-key admission, stale runtime answers, +and raw-versus-effective lifecycle policy. Local server typecheck passed using +the installed TypeScript binary (the pnpm shim could not reach its signature +registry in the sandbox). Runtime/config regressions passed 38 tests; startup +shutdown and OpenAPI regressions passed 152 tests. CI's response mock counts and +missing WebSocket mock were fixed; `/api/ready` now has explicit GET/HEAD OpenAPI +contracts. These receipts precede the next commit and do not qualify a live run. +Cloud review corrections cover partial-create retries, legacy shape preservation, +pre-allocation image qualification, actual extra-service detection and protected +storage settings. Fresh ephemeral stacks now initialize a decision key in the +vault before deployment; already-started stacks require their existing key. + + +### 2026-11-01 overlap workflow qualification + +Core `0b8fcca44f42d9d43894e728e42fa55545cc3665` and Cloud +`bd772707a0f7398990d0d84a2b75bd242445ffaf` both received fresh 5/5 reviews. +All checks on those pushed heads pass. The Cloud local suite passed 4,795 tests +with 129 explicit skips. The superseded Core full local run was stopped after +three hours after its pushed head passed CI; its historical failures are not +relabeled. Fresh recovery-file verification passed all 75 tests. + +The next Core commit adds the explicit Product E2E `rolling-web-deployment` +cell. It reuses the remote workspace, active execution and later retained-session +oracles, with a separate ready-before-handoff receipt. An independent home and +compiled build are used for A/B/C; Postgres, S3 and generated secrets are shared. +Read/save probes perform no HTTP retries. The old five-minute outage suites retain +their contract. The observer captures A before handoff so B cannot adopt before +the baseline is recorded. The normal recovery code owns all leases and replay. + +Credential-free evidence: 143 affected harness tests, 120 catalog/fixture/report +tests, E2E typecheck, and map structure checks passed. Discovery selects one +explicit Codex Daytona cell, four expected admissions, one attempt, $10 company +budget and the existing 30-minute limit. This does not establish live execution. + +The actual supervisor passed two replacements with no model calls, 436 ordinary +reads and 45 saved/read-back comments, zero request failures, old homes removed, +and owned database/bucket/process cleanup confirmed. B readiness was 18,694 ms; +C readiness was 109,182 ms. Receipt (private local): +`/private/var/folders/74/zhncxl_j05xb5_r6_vfbk73h0000gn/T/paperclip-runner-e2e-meJSIF/replacement-evidence.json`. +It is source/compiled-with-workspace-dependencies, not a released container. +The prior attempt `srF1rz` timed out on C and remains failed; its first replacement +had 107 reads and 11 saves without failures. The initial `Ro7sxQ` setup failed +because a partial TypeScript build had not copied the compiled Runner vendor; +the normal build vendoring step corrected it. All three attempts cleaned up. + +Staging read-only discovery and authentication work with AWS `paperclip-dev` +(account `078455283791`); production `paperclip-prod` is also authenticated. +The existing shared `canary-staging` stack is `stack-pool-8fabb27531db77af`. +Do not assume it is exclusively owned: select a dedicated zero-downtime test +stack or acquire the existing stack operation lock before any hosted mutation. +No hosted mutation or paid model call has occurred for this task yet. + +Remaining: live overlap proof; released artifacts; migration/enrollment and +compatible rollback; ten staging deployments; internal production canary. diff --git a/maps/agent-access-map/instance-operations.md b/maps/agent-access-map/instance-operations.md index f2d4250a4cb..db0c87834f0 100644 --- a/maps/agent-access-map/instance-operations.md +++ b/maps/agent-access-map/instance-operations.md @@ -168,6 +168,9 @@ API: `present` — representative operation IDs: status remains an in-memory read. [Route coverage](../../server/src/__tests__/instance-settings-routes.test.ts) and [PostgreSQL coverage](../../server/src/services/heartbeat/upgrade-readiness.test.ts) establish this local boundary; hosted active-run continuity requires separate proof. +- `GET /api/ready` and `HEAD /api/ready` — public deployment readiness only; + 200 after recovery, 503 while starting, recovering, or draining. No SQL, + session resolution, company identity, or control authority. - `GET /api/liveness` and `HEAD /api/liveness` — public process startup status and build commit only; no database readiness or workspace identity. diff --git a/maps/feature-map/instance-operations.md b/maps/feature-map/instance-operations.md index dd43901323e..e42e8a1c48c 100644 --- a/maps/feature-map/instance-operations.md +++ b/maps/feature-map/instance-operations.md @@ -51,6 +51,12 @@ It remains reachable during an idle drain. Use `/api/health` for explicit databa readiness, signed bootstrap, and authorized diagnostics; liveness does not establish database health or claim a workspace. +Deployment routing uses public `GET /api/ready` (or `HEAD`). It returns 200 only +after startup recovery finishes and 503 while starting, recovering, or draining. +The probe reads in-memory state without SQL or session resolution. Local +[readiness tests](../../server/src/__tests__/liveness.test.ts) establish that +boundary; hosted overlap and active-run continuity require separate qualification. + Automated: [static UI idle completion](../../server/src/__tests__/static-ui-idle.test.ts) covers real aborted file streams, disconnects before file lookup, nested routers, and unfinished parent/fallback writes. Mounted Express applications retain their diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 66ff29b3aa6..5897666fff9 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -320,6 +320,27 @@ export async function closeCheckpointDb(db: Db, timeoutSeconds: number): Promise await checkpointDbs.get(db)?.$client.end({ timeout: timeoutSeconds }); } +// Managed file publication holds an agent lock while transferring bytes. A +// stalled object store must not consume request or controller-renewal capacity. +const fileStorageDbFactories = new WeakMap Db>(); +const fileStorageDbs = new WeakMap(); +const closedFileStorageOwners = new WeakSet(); +export function fileStorageDb(db: Db): Db { + const existing = fileStorageDbs.get(db); + if (existing) return existing; + const factory = fileStorageDbFactories.get(db); + if (!factory) return db; // Preserve the caller's transaction/savepoint. + if (closedFileStorageOwners.has(db)) throw new Error("file_storage_pool_closed"); + const created = factory(); + fileStorageDbs.set(db, created); + fileStorageDbs.set(created, created); + return created; +} +export async function closeFileStorageDb(db: Db, timeoutSeconds: number): Promise { + closedFileStorageOwners.add(db); + await fileStorageDbs.get(db)?.$client.end({ timeout: timeoutSeconds }); +} + export function createDb(url: string, options?: DatabaseClientOptions) { const resolved = resolveDatabaseClientOptions(options ?? databaseClientOptionsFromEnv()); return createDbClient(url, resolved); @@ -345,6 +366,9 @@ function createDbClient(url: string, resolved: DatabaseClientOptions, owner?: ob checkpointDbFactories.set(db, () => createDbClient(url, { ...resolved, maxConnections: CHECKPOINT_POOL_MAX_CONNECTIONS, applicationName: "paperclip-runner-checkpoint", }, db)); + fileStorageDbFactories.set(db, () => createDbClient(url, { + ...resolved, maxConnections: 2, applicationName: "paperclip-file-storage", + }, db)); return db; } diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index 82ba301ebdd..53abe16e078 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -5,6 +5,8 @@ export { closeLivenessDb, checkpointDb, closeCheckpointDb, + fileStorageDb, + closeFileStorageDb, closeRegisteredClients, getPostgresDataDirectory, ensurePostgresDatabase, diff --git a/packages/db/src/migrations/0422_eminent_darwin.sql b/packages/db/src/migrations/0422_eminent_darwin.sql new file mode 100644 index 00000000000..816d35e48cb --- /dev/null +++ b/packages/db/src/migrations/0422_eminent_darwin.sql @@ -0,0 +1,13 @@ +CREATE TABLE IF NOT EXISTS "agent_file_roots" ( + "company_id" uuid NOT NULL, + "agent_id" uuid NOT NULL, + "generation" uuid NOT NULL, + "entries" jsonb NOT NULL, + "updated_at" timestamp with time zone DEFAULT now() NOT NULL, + CONSTRAINT "agent_file_roots_company_id_agent_id_pk" PRIMARY KEY("company_id","agent_id") +); +--> statement-breakpoint +DO $$ BEGIN + ALTER TABLE "agent_file_roots" ADD CONSTRAINT "agent_file_roots_company_id_agent_id_agents_company_id_id_fk" FOREIGN KEY ("company_id","agent_id") REFERENCES "public"."agents"("company_id","id") ON DELETE cascade ON UPDATE no action; +EXCEPTION WHEN duplicate_object THEN NULL; +END $$; diff --git a/packages/db/src/migrations/meta/0417_snapshot.json b/packages/db/src/migrations/meta/0422_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0417_snapshot.json rename to packages/db/src/migrations/meta/0422_snapshot.json index a81b201cc91..66b83d1df9a 100644 --- a/packages/db/src/migrations/meta/0417_snapshot.json +++ b/packages/db/src/migrations/meta/0422_snapshot.json @@ -1,6 +1,6 @@ { - "id": "c2c69a4a-8281-419c-9eeb-a7bcbc2beaae", - "prevId": "bbaeecd8-ff3f-4e70-a069-291bf3dc97bb", + "id": "fe0b877c-d63e-46d9-8798-c7729fd13eb7", + "prevId": "deb3f1b9-6eb7-447f-adf2-dee69f943c59", "version": "7", "dialect": "postgresql", "tables": { @@ -2233,6 +2233,74 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.agent_file_roots": { + "name": "agent_file_roots", + "schema": "", + "columns": { + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "agent_id": { + "name": "agent_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "generation": { + "name": "generation", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "entries": { + "name": "entries", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": {}, + "foreignKeys": { + "agent_file_roots_company_id_agent_id_agents_company_id_id_fk": { + "name": "agent_file_roots_company_id_agent_id_agents_company_id_id_fk", + "tableFrom": "agent_file_roots", + "tableTo": "agents", + "columnsFrom": [ + "company_id", + "agent_id" + ], + "columnsTo": [ + "company_id", + "id" + ], + "onDelete": "cascade", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": { + "agent_file_roots_company_id_agent_id_pk": { + "name": "agent_file_roots_company_id_agent_id_pk", + "columns": [ + "company_id", + "agent_id" + ] + } + }, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.agent_identity_keys": { "name": "agent_identity_keys", "schema": "", @@ -29705,6 +29773,241 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.execution_workspace_snapshots": { + "name": "execution_workspace_snapshots", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true, + "default": "gen_random_uuid()" + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "execution_workspace_id": { + "name": "execution_workspace_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "source_operation_id": { + "name": "source_operation_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "source_run_id": { + "name": "source_run_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "parent_snapshot_id": { + "name": "parent_snapshot_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "generation": { + "name": "generation", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "epoch": { + "name": "epoch", + "type": "integer", + "primaryKey": false, + "notNull": true + }, + "status": { + "name": "status", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "format": { + "name": "format", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'paperclip.remote-workspace-archive.v1'" + }, + "object_key": { + "name": "object_key", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "sha256": { + "name": "sha256", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "byte_size": { + "name": "byte_size", + "type": "bigint", + "primaryKey": false, + "notNull": false + }, + "key": { + "name": "key", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "manifest_digest": { + "name": "manifest_digest", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": false + }, + "delete_after": { + "name": "delete_after", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": false + }, + "last_error": { + "name": "last_error", + "type": "text", + "primaryKey": false, + "notNull": false + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "execution_workspace_snapshots_operation_uniq": { + "name": "execution_workspace_snapshots_operation_uniq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "execution_workspace_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "source_operation_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_workspace_snapshots_generation_uniq": { + "name": "execution_workspace_snapshots_generation_uniq", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "execution_workspace_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "generation", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_workspace_snapshots_workspace_idx": { + "name": "execution_workspace_snapshots_workspace_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "execution_workspace_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, + "execution_workspace_snapshots_cleanup_idx": { + "name": "execution_workspace_snapshots_cleanup_idx", + "columns": [ + { + "expression": "status", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "delete_after", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": {}, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.execution_workspaces": { "name": "execution_workspaces", "schema": "", @@ -37105,6 +37408,15 @@ "client_request_id" ] }, + "issue_comments_agent_client_request_uq": { + "name": "issue_comments_agent_client_request_uq", + "nullsNotDistinct": false, + "columns": [ + "issue_id", + "author_agent_id", + "client_request_id" + ] + }, "issue_comments_company_id_uq": { "name": "issue_comments_company_id_uq", "nullsNotDistinct": false, @@ -51574,6 +51886,195 @@ "checkConstraints": {}, "isRLSEnabled": false }, + "public.sandbox_creation_operations": { + "name": "sandbox_creation_operations", + "schema": "", + "columns": { + "id": { + "name": "id", + "type": "uuid", + "primaryKey": true, + "notNull": true + }, + "company_id": { + "name": "company_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "environment_id": { + "name": "environment_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "original_environment_id": { + "name": "original_environment_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "run_id": { + "name": "run_id", + "type": "uuid", + "primaryKey": false, + "notNull": true + }, + "request_key": { + "name": "request_key", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'run'" + }, + "issue_id": { + "name": "issue_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "execution_workspace_id": { + "name": "execution_workspace_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "provider": { + "name": "provider", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "plugin_id": { + "name": "plugin_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "driver_key": { + "name": "driver_key", + "type": "text", + "primaryKey": false, + "notNull": true + }, + "state": { + "name": "state", + "type": "text", + "primaryKey": false, + "notNull": true, + "default": "'pending'" + }, + "config": { + "name": "config", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "metadata": { + "name": "metadata", + "type": "jsonb", + "primaryKey": false, + "notNull": true + }, + "lease_id": { + "name": "lease_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "next_attempt_at": { + "name": "next_attempt_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true + }, + "created_at": { + "name": "created_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + }, + "updated_at": { + "name": "updated_at", + "type": "timestamp with time zone", + "primaryKey": false, + "notNull": true, + "default": "now()" + } + }, + "indexes": { + "sandbox_creation_operations_pending_deadline_idx": { + "name": "sandbox_creation_operations_pending_deadline_idx", + "columns": [ + { + "expression": "next_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"sandbox_creation_operations\".\"state\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "sandbox_creation_operations_owner_idx": { + "name": "sandbox_creation_operations_owner_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "original_environment_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "run_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "request_key", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": true, + "concurrently": false, + "method": "btree", + "with": {} + } + }, + "foreignKeys": { + "sandbox_creation_operations_environment_id_environments_id_fk": { + "name": "sandbox_creation_operations_environment_id_environments_id_fk", + "tableFrom": "sandbox_creation_operations", + "tableTo": "environments", + "columnsFrom": [ + "environment_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" + } + }, + "compositePrimaryKeys": {}, + "uniqueConstraints": {}, + "policies": {}, + "checkConstraints": {}, + "isRLSEnabled": false + }, "public.secret_access_events": { "name": "secret_access_events", "schema": "", @@ -52581,6 +53082,12 @@ "primaryKey": false, "notNull": false }, + "next_eval_reason": { + "name": "next_eval_reason", + "type": "text", + "primaryKey": false, + "notNull": false + }, "next_eval_at": { "name": "next_eval_at", "type": "timestamp with time zone", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 06863b4a75d..b3bc9221371 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2941,6 +2941,13 @@ "when": 1793469782391, "tag": "0421_bright_skaar", "breakpoints": true + }, + { + "idx": 422, + "version": "7", + "when": 1793503729899, + "tag": "0422_eminent_darwin", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/agent_file_roots.ts b/packages/db/src/schema/agent_file_roots.ts new file mode 100644 index 00000000000..f48f6d8e49e --- /dev/null +++ b/packages/db/src/schema/agent_file_roots.ts @@ -0,0 +1,19 @@ +import { foreignKey, jsonb, pgTable, primaryKey, timestamp, uuid } from "drizzle-orm/pg-core"; +import { agents } from "./agents.js"; + +export type DurableAgentFileEntry = { path: string } & ( + | { kind: "dir" } + | { kind: "file"; mode: number; size: number; sha256: string; objectKey: string | null } +); + +/** Current managed files. The filesystem is a projection once this row exists. */ +export const agentFileRoots = pgTable("agent_file_roots", { + companyId: uuid("company_id").notNull(), + agentId: uuid("agent_id").notNull(), + generation: uuid("generation").notNull(), + entries: jsonb("entries").$type().notNull(), + updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), +}, t => ({ + identity: primaryKey({ columns: [t.companyId, t.agentId] }), + owner: foreignKey({ columns: [t.companyId, t.agentId], foreignColumns: [agents.companyId, agents.id] }).onDelete("cascade"), +})); diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 6e2a2b21327..1af7fbcfdf2 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -233,6 +233,7 @@ export { announcementDismissals, announcementPublications } from "./announcement export { chatTaskHandoffs, chatCompletionDeliveries } from "./chat_completion_deliveries.js"; export { agentInstructionRevisions, agentInstructionHeads, agentInstructionWorkingCopies } from "./agent_instruction_revisions.js"; +export { agentFileRoots, type DurableAgentFileEntry } from "./agent_file_roots.js"; export { browserUseSettings, browserUseSessions, browserUseRuns, browserUseBrowsers } from "./browser_use.js"; diff --git a/packages/db/src/work-signals.ts b/packages/db/src/work-signals.ts index cf914398e5d..46e2071dda4 100644 --- a/packages/db/src/work-signals.ts +++ b/packages/db/src/work-signals.ts @@ -1,11 +1,12 @@ import { sql } from "drizzle-orm"; +import { randomUUID } from "node:crypto"; -// Process-local hints about existing durable queues. No SQL triggers, tables, -// background timers, or driver query replay. Only opted-in transactions fetch -// an XID, so unrelated transactions keep their existing SQL behavior. +// Hints about existing durable queues; process-local by default, with optional +// commit-bound PostgreSQL notifications during controller replacement. No SQL +// triggers, tables, timers, or query replay. Only registered work fetches an XID. type Listener = (event: "intent" | "settled", hints?: ReadonlySet) => void; type Transaction = { topics: Set; hints: Map>; xid?: Promise; uncertain: boolean }; -type Scope = { key: object; listeners: Map>; transactions: Set; root: Connection }; +type Scope = { key: object; listeners: Map>; transactions: Set; root: Connection; notificationOrigin: string; notify: boolean }; type Context = { scope: Scope; transaction?: Transaction }; type Connection = { transaction: (...args: any[]) => Promise; execute: (...args: any[]) => Promise }; const contexts = new WeakMap(); @@ -19,7 +20,7 @@ function publish(scope: Scope, topic: string, event: "intent" | "settled" = "set /** Installed once by createDb; dedicated child pools share their owner's scope. */ export function installDatabaseWorkSignals(db: T, owner?: object): T { - const scope = (owner && contexts.get(owner)?.scope) || { key: {}, listeners: new Map(), transactions: new Set(), root: db }; + const scope = (owner && contexts.get(owner)?.scope) || { key: {}, listeners: new Map(), transactions: new Set(), root: db, notificationOrigin: randomUUID(), notify: false }; install(db, { scope }); return db; } @@ -73,6 +74,42 @@ export async function signalDatabaseWork(connection: object, topic: string, hint // Keep a failed acquisition distinguishable from an uncertain queue write. try { await state.xid; } catch (error) { state.xid = undefined; throw error; } + if (context.scope.notify) { + // PostgreSQL delivers only after the outer commit, including savepoints. + // Queue rows remain authoritative; notification payloads contain no data. + await (connection as Connection).execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(context.scope, topic)})`); + } +} + +export const DATABASE_WORK_CHANNEL = "paperclip_work_v1"; +const validTopic = (topic: string) => /^[a-zA-Z0-9:_-]{1,200}$/.test(topic); +function notification(scope: Scope, topic: string) { + if (!validTopic(topic)) throw new Error("Invalid database work topic"); + return JSON.stringify({ version: 1, origin: scope.notificationOrigin, topic }); +} +export function enableDatabaseWorkNotifications(owner: object): void { + const scope = contexts.get(owner)?.scope; + if (!scope) throw new Error("Database work notifications require createDb"); + scope.notify = true; +} +/** A post-commit invalidation hint, for already-persisted live UI state. */ +export async function publishDatabaseWorkHint(owner: object, topic: string): Promise { + const scope = contexts.get(owner)?.scope; + if (scope?.notify) await scope.root.execute(sql`select pg_notify(${DATABASE_WORK_CHANNEL}, ${notification(scope, topic)})`); +} +export function receiveDatabaseWorkNotification(owner: object, payload: string): void { + const scope = contexts.get(owner)?.scope; + if (!scope || payload.length > 1024) return; + let value: { version?: unknown; origin?: unknown; topic?: unknown }; + try { value = JSON.parse(payload); } catch { return; } + if (!value || value.version !== 1 || typeof value.origin !== "string" || value.origin === scope.notificationOrigin + || typeof value.topic !== "string" || !validTopic(value.topic)) return; + publish(scope, value.topic); +} +/** Reconnect can lose hints. Reconstruct existing durable queues once; no poll. */ +export function reconcileDatabaseWorkNotifications(owner: object): void { + const scope = contexts.get(owner)?.scope; + if (scope) for (const topic of scope.listeners.keys()) publish(scope, topic); } export function databaseWorkPending(owner: object, topic: string): boolean { diff --git a/packages/shared/src/types/live.ts b/packages/shared/src/types/live.ts index 0536e4ee79a..b90e55e4de8 100644 --- a/packages/shared/src/types/live.ts +++ b/packages/shared/src/types/live.ts @@ -16,4 +16,4 @@ export type LiveDeliveryClientMessage = { type: "live.hello"; version: 1; capabi export type LiveDeliveryServerMessage = | { type: "live.capabilities"; version: 1; capabilities: string[] } | { type: "live.subscribed"; version: 1; revision: number } - | { type: "live.resync"; version: 1; revision: number; reason: "delivery_overflow" }; + | { type: "live.resync"; version: 1; revision: number; reason: "delivery_overflow" | "controller_change" }; diff --git a/server/scripts/bundle-runtime.mjs b/server/scripts/bundle-runtime.mjs index 7e36010e2d1..45754641187 100644 --- a/server/scripts/bundle-runtime.mjs +++ b/server/scripts/bundle-runtime.mjs @@ -71,10 +71,10 @@ function externalizeThirdParty(isBundled, runtimeDir) { }; } -export async function bundleRuntime({ entry, outdir, isBundled = (specifier) => specifier.startsWith("@paperclipai/") }) { +export async function bundleRuntime({ entry, outdir, maintenanceEntry, isBundled = (specifier) => specifier.startsWith("@paperclipai/") }) { await rm(outdir, { recursive: true, force: true }); const result = await build({ - entryPoints: { index: "paperclip-runtime-entry", server: entry }, + entryPoints: { index: "paperclip-runtime-entry", server: entry, ...(maintenanceEntry ? { "agent-files": maintenanceEntry } : {}) }, outdir, outExtension: { ".js": ".mjs" }, chunkNames: "chunk-[hash]", @@ -102,7 +102,8 @@ export async function bundleRuntime({ entry, outdir, isBundled = (specifier) => async function main() { const started = performance.now(); - const summary = await bundleRuntime({ entry: join(serverDir, "dist", "index.js"), outdir: join(serverDir, "dist-runtime") }); + const summary = await bundleRuntime({ entry: join(serverDir, "dist", "index.js"), outdir: join(serverDir, "dist-runtime"), + maintenanceEntry: join(serverDir, "dist", "maintenance", "agent-files.js") }); console.log(`[bundle-runtime] ${summary.inputs} inputs -> ${summary.chunks} chunks in ${Math.round(performance.now() - started)} ms: ${summary.outfile}`); } diff --git a/server/scripts/bundle-runtime.test.mjs b/server/scripts/bundle-runtime.test.mjs index 6e9375f61c4..78f4072ac84 100644 --- a/server/scripts/bundle-runtime.test.mjs +++ b/server/scripts/bundle-runtime.test.mjs @@ -71,6 +71,15 @@ describe("bundleRuntime", () => { rmSync(root, { recursive: true, force: true }); }); + it("packages a maintenance entry without starting the application or needing a TypeScript loader", async () => { + const maintenanceEntry = path.join(root, "server", "dist", "maintenance.js"); + write(maintenanceEntry, 'import { asset } from "@paperclipai/fixture"; console.log(JSON.stringify({ asset, phase: process.argv.at(-1) }));'); + const outdir = path.join(root, "server", "dist-runtime"); + await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir, maintenanceEntry }); + const { stdout } = await execFileAsync(process.execPath, [path.join(outdir, "agent-files.mjs"), "--phase=eligibility"], { timeout: 30_000 }); + expect(JSON.parse(stdout)).toEqual({ asset: "asset beside its source", phase: "--phase=eligibility" }); + }); + it("bundles workspace TypeScript and runs with plain node", async () => { const outdir = path.join(root, "server", "dist-runtime"); const summary = await bundleRuntime({ entry: path.join(root, "server", "dist", "index.js"), outdir }); diff --git a/server/src/__tests__/agent-directory-working-copies.test.ts b/server/src/__tests__/agent-directory-working-copies.test.ts index 25bc17d1b79..183769a38d6 100644 --- a/server/src/__tests__/agent-directory-working-copies.test.ts +++ b/server/src/__tests__/agent-directory-working-copies.test.ts @@ -16,7 +16,7 @@ import { createHash, randomUUID } from "node:crypto"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { eq } from "drizzle-orm"; import { agentFileStore, fileHash, inspectAgentFile, snapshotAgentFiles, MAX_AGENT_FILE_BYTES, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES } from "../services/agent-file-store.js"; -import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb } from "@paperclipai/db"; +import { agents, companies, authUsers, companyMemberships, principalPermissionGrants, heartbeatRuns, environmentLeases, environments, agentInstructionWorkingCopies, agentInstructionRevisions, agentInstructionHeads, createDb, fileStorageDb } from "@paperclipai/db"; import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { agentInstructionRevisionService } from "../services/agent-instruction-revisions.js"; import { agentInstructionWorkingCopyService, instructionWorkingCopyGuidance } from "../services/agent-instruction-working-copies.js"; @@ -399,12 +399,11 @@ describe("persistent agent directories", () => { expect(await copies.hasChanges({ companyId, runId: first.runId })).toBe(false); await db.update(heartbeatRuns).set({ status: "succeeded", runtimeMode: "native" }).where(eq(heartbeatRuns.id, first.runId)); const secondId = await nextRun(); - const transaction = db.transaction.bind(db); - let transactions = 0; + const fileDb = fileStorageDb(db); + const transaction = fileDb.transaction.bind(fileDb); let injected = false; - const spy = vi.spyOn(db, "transaction").mockImplementation((async (callback, options) => { - transactions++; - if (transactions === 2 && !injected) { + const spy = vi.spyOn(fileDb, "transaction").mockImplementation((async (callback, options) => { + if (!injected) { injected = true; const before = (await copies.get(companyId, first.runId))!; vi.useFakeTimers({ toFake: ["Date"] }); @@ -432,11 +431,11 @@ describe("persistent agent directories", () => { const secondId = await nextRun(); const warning = vi.spyOn(logger, "warn"); const privateValue = "private-materialization-/tmp/untrusted-token-do-not-log"; - const transaction = db.transaction.bind(db); - let calls = 0; + const fileDb = fileStorageDb(db); + const transaction = fileDb.transaction.bind(fileDb); let injected = false; - const spy = vi.spyOn(db, "transaction").mockImplementation((async (callback, options) => { - if (++calls === 2) { + const spy = vi.spyOn(fileDb, "transaction").mockImplementation((async (callback, options) => { + if (!injected) { injected = true; const prior = (await copies.get(companyId, first.runId))!; const values = change === "materialization" ? { receipt: { ...prior.receipt, materializationIdentity: privateValue } } diff --git a/server/src/__tests__/agent-file-durability.test.ts b/server/src/__tests__/agent-file-durability.test.ts new file mode 100644 index 00000000000..dfd674dda9a --- /dev/null +++ b/server/src/__tests__/agent-file-durability.test.ts @@ -0,0 +1,178 @@ +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { randomUUID } from "node:crypto"; +import { Readable } from "node:stream"; +import { beforeAll, beforeEach, afterEach, afterAll, describe, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { agents, agentFileRoots, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, createDb, closeFileStorageDb } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { agentFileStore, adoptAgentFiles, fileHash } from "../services/agent-file-store.js"; +import { invalidateAgentFileProjection, persistDurableAgentFiles } from "../services/agent-file-durability.js"; +import { materializeInstructionBytes } from "../services/agent-instruction-files.js"; +import { resolveManagedInstructionsRoot } from "../services/agent-instructions.js"; +import { AGENT_FILE_OBJECT_GRACE_MS, collectAgentFileObjects, migrateAgentFiles } from "../services/agent-file-maintenance.js"; +import { getStorageService } from "../storage/index.js"; + +const objects = vi.hoisted(() => ({ bytes: new Map(), rejectWrites: false, corruptReads: false })); +vi.mock("../storage/index.js", () => ({ getStorageService: () => ({ + provider: "s3", + async putFile(input: { companyId: string; namespace: string; body: Readable; byteSize: number; sha256: string }) { + if (objects.rejectWrites) throw new Error("fixture object-store outage"); + const chunks: Buffer[] = []; + for await (const chunk of input.body) chunks.push(Buffer.from(chunk)); + const body = Buffer.concat(chunks); + expect(body.length).toBe(input.byteSize); + expect(fileHash(body)).toBe(input.sha256); + const objectKey = `${input.companyId}/${input.namespace}/${randomUUID()}`; + objects.bytes.set(objectKey, body); + return { objectKey }; + }, + async getObject(companyId: string, objectKey: string) { + expect(objectKey.startsWith(`${companyId}/`)).toBe(true); + const body = objects.bytes.get(objectKey); + if (!body) throw new Error("fixture object missing"); + return { stream: Readable.from([objects.corruptReads ? Buffer.from("wrong bytes") : body]) }; + }, + async headObject(_companyId: string, key: string) { return { exists: objects.bytes.has(key), lastModified: new Date(0) }; }, + async deleteObject(companyId: string, key: string, options: { allVersions: boolean }) { + expect(key.startsWith(`${companyId}/`)).toBe(true); + expect(options.allVersions).toBe(true); + objects.bytes.delete(key); + }, +}) })); + +describe("durable agent file authority", () => { + let database: Awaited>; + let db: ReturnType; + let home: string, root: string, companyId: string, agentId: string, userId: string; + const actor = () => ({ type: "board" as const, userId, source: "session" as const }); + const store = () => agentFileStore(db); + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("durable-agent-files-"); + db = createDb(database.connectionString); + }, 90_000); + afterAll(async () => { + if (db) { await closeFileStorageDb(db, 1); await db.$client.end({ timeout: 1 }); } + await database?.cleanup(); + }); + beforeEach(async () => { + home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "durable-agent-files-"))); + vi.stubEnv("PAPERCLIP_HOME", home); vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true"); + objects.bytes.clear(); objects.rejectWrites = false; objects.corruptReads = false; + companyId = randomUUID(); agentId = randomUUID(); userId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "File authority", issuePrefix: randomUUID().slice(0, 8) }); + await db.insert(authUsers).values({ id: userId, name: "Editor", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(agents).values({ id: agentId, companyId, name: "File owner" }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "operator" }); + await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, + permissionKey: "agents:configure", scope: { agentIds: [agentId] } }); + root = resolveManagedInstructionsRoot({ id: agentId, companyId, name: "File owner", adapterConfig: {} }); + }); + afterEach(async () => { + vi.unstubAllEnvs(); + await fs.rm(home, { recursive: true, force: true }); + }); + async function replaceFilesystem() { + await fs.rm(home, { recursive: true, force: true }); + home = await fs.realpath(await fs.mkdtemp(path.join(os.tmpdir(), "durable-agent-replacement-"))); + vi.stubEnv("PAPERCLIP_HOME", home); + root = resolveManagedInstructionsRoot({ id: agentId, companyId, name: "File owner", adapterConfig: {} }); + } + + it("imports legacy bytes, modes and empty entries, then restores without any old filesystem", async () => { + const binary = Buffer.alloc(2 * 1024 * 1024, 0x81); + await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Original")); + await materializeInstructionBytes(root, "nested/data.bin", binary); + await materializeInstructionBytes(root, "empty.txt", Buffer.alloc(0)); + await materializeInstructionBytes(root, "script.sh", Buffer.from("#!/bin/sh\n")); + await fs.chmod(path.join(root, "script.sh"), 0o755); + await fs.mkdir(path.join(root, "empty-folder")); + await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root } }).where(eq(agents.id, agentId)); + expect(await store().read(companyId, agentId, "AGENTS.md", actor())).toEqual(Buffer.from("Original")); + await replaceFilesystem(); + vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "false"); + expect(await store().read(companyId, agentId, "nested/data.bin", actor())).toEqual(binary); + expect(await store().read(companyId, agentId, "empty.txt", actor())).toEqual(Buffer.alloc(0)); + expect((await fs.stat(path.join(root, "script.sh"))).mode & 0o777).toBe(0o755); + expect((await fs.stat(path.join(root, "empty-folder"))).isDirectory()).toBe(true); + }); + + it("does not promote a rejected first save through subsequent local adoption", async () => { + objects.rejectWrites = true; + await expect(store().write({ companyId, agentId, path: "failed.txt", bytes: Buffer.from("uncommitted"), baseHash: null }, actor())) + .rejects.toThrow("fixture object-store outage"); + expect(await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId))).toEqual([]); + objects.rejectWrites = false; + expect(await store().read(companyId, agentId, "failed.txt", actor())).toBeNull(); + await replaceFilesystem(); + expect(await store().read(companyId, agentId, "failed.txt", actor())).toBeNull(); + }); + + it("dry-runs migration without adopting files and requires stopped writers for publication", async () => { + await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Legacy")); + const input = { companyId, apply: false, writersStopped: false }; + expect(await migrateAgentFiles(db, input)).toMatchObject({ results: [{ agentId, state: "needs_import" }], pageReady: false }); + expect(await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId))).toEqual([]); + await expect(migrateAgentFiles(db, { ...input, apply: true })).rejects.toThrow("stopped writers"); + expect(await migrateAgentFiles(db, { ...input, apply: true, writersStopped: true })).toMatchObject({ pageReady: true }); + await replaceFilesystem(); + expect(await store().read(companyId, agentId, "AGENTS.md", actor())).toEqual(Buffer.from("Legacy")); + }); + + it("refuses migration while the agent has queued or running execution", async () => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning(); + for (const status of ["running", "queued"]) { + await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, run!.id)); + expect(await migrateAgentFiles(db, { companyId, apply: true, writersStopped: true })) + .toMatchObject({ results: [{ agentId, state: "active_execution" }], pageReady: false }); + expect(await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId))).toEqual([]); + } + }); + + it("retains current and unadopted objects, then collects expired unreferenced and deleted-owner objects", async () => { + const prefix = `${companyId}/agent-files/${agentId}/`; + const abandoned = `${prefix}abandoned`; + objects.bytes.set(abandoned, Buffer.from("uncertain upload")); + const provider = { id: "s3" as const, listObjects: async () => ({ keys: [...objects.bytes.keys()].filter(key => key.startsWith(prefix)) }) }; + const input = { companyId, agentId, apply: true, now: AGENT_FILE_OBJECT_GRACE_MS + 1 }; + const collect = (options = input) => collectAgentFileObjects(db, provider, getStorageService(), options); + expect(await collect()).toMatchObject({ retained: 1, deleted: 0 }); + await store().write({ companyId, agentId, path: "AGENTS.md", bytes: Buffer.from("Current"), baseHash: null }, actor()); + expect(await collect({ ...input, now: 1 })).toMatchObject({ retained: 2, deleted: 0 }); + expect(await collect({ ...input, apply: false })).toMatchObject({ retained: 1, eligible: 1, deleted: 0 }); + expect(objects.bytes.has(abandoned)).toBe(true); + expect(await collect()).toMatchObject({ retained: 1, deleted: 1 }); + await replaceFilesystem(); + expect(await store().read(companyId, agentId, "AGENTS.md", actor())).toEqual(Buffer.from("Current")); + await db.delete(agents).where(eq(agents.id, agentId)); + expect(await collect()).toMatchObject({ retained: 0, deleted: 1 }); + expect(objects.bytes.size).toBe(0); + }); + + it("refuses to treat a missing legacy filesystem as an empty durable folder", async () => { + await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Not migrated yet")); + await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "managed", instructionsRootPath: root } }).where(eq(agents.id, agentId)); + await replaceFilesystem(); + await expect(store().read(companyId, agentId, "AGENTS.md", actor())) + .rejects.toMatchObject({ status: 503, details: { code: "AGENT_FILES_MIGRATION_REQUIRED" } }); + expect(await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId))).toEqual([]); + }); + + it("restores the committed head after SQL rollback and rejects corrupted object bytes", async () => { + await store().write({ companyId, agentId, path: "AGENTS.md", bytes: Buffer.from("Committed"), baseHash: null }, actor()); + await expect(db.transaction(async tx => { + const [owner] = await tx.select().from(agents).where(eq(agents.id, agentId)).for("update"); + await adoptAgentFiles(tx, owner!); + await invalidateAgentFileProjection(root); + await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Rolled back")); + await persistDurableAgentFiles(tx, owner!, root); + throw new Error("fixture transaction rollback"); + })).rejects.toThrow("fixture transaction rollback"); + expect(await store().read(companyId, agentId, "AGENTS.md", actor())).toEqual(Buffer.from("Committed")); + await replaceFilesystem(); objects.corruptReads = true; + await expect(store().read(companyId, agentId, "AGENTS.md", actor())).rejects.toThrow(/committed (size|digest)/); + objects.corruptReads = false; + expect(await store().read(companyId, agentId, "AGENTS.md", actor())).toEqual(Buffer.from("Committed")); + }); +}); diff --git a/server/src/__tests__/agent-instructions-service.test.ts b/server/src/__tests__/agent-instructions-service.test.ts index 7641dcc6778..b165da9e20b 100644 --- a/server/src/__tests__/agent-instructions-service.test.ts +++ b/server/src/__tests__/agent-instructions-service.test.ts @@ -91,13 +91,16 @@ describe("agent instructions service", () => { await fs.mkdir(path.join(externalRoot, "docs"), { recursive: true }); await fs.writeFile(path.join(externalRoot, "docs", "TOOLS.md"), "## Tools\n", "utf8"); - const svc = agentInstructionsService({ select: () => ({ from: () => ({ where: async () => [] }) }) } as never); const agent = makeAgent({ instructionsBundleMode: "external", instructionsRootPath: externalRoot, instructionsEntryFile: "AGENTS.md", instructionsFilePath: path.join(externalRoot, "AGENTS.md"), }); + const select = () => ({ from: () => ({ where: () => Object.assign(Promise.resolve([]), { for: async () => [agent] }) }) }); + const svc = agentInstructionsService({ select, + transaction: async (run: (tx: { select: typeof select }) => Promise) => run({ select }), + } as never); const result = await svc.updateBundle(agent, { mode: "managed" }); diff --git a/server/src/__tests__/delivery-work-coordinator.test.ts b/server/src/__tests__/delivery-work-coordinator.test.ts index 2ec8054336f..c432456b799 100644 --- a/server/src/__tests__/delivery-work-coordinator.test.ts +++ b/server/src/__tests__/delivery-work-coordinator.test.ts @@ -334,6 +334,24 @@ describe("delivery work coordinator", () => { expect(t.run).toHaveBeenCalledTimes(1); expect(idleWorkSnapshot().active).toBe(0); }); + it("keeps admitted owner commands live while dispatch queues stop, then closes every worker", async () => { + const s = setup(), dispatch = task(), commands = task(); + await s.coordinator.register(DELIVERY_QUEUES.queuedRunDispatch, dispatch).ready; + await s.coordinator.register(DELIVERY_QUEUES.nativeCancellation, commands).ready; + await s.coordinator.stop(new Set([DELIVERY_QUEUES.nativeCancellation])); + await s.enqueue(DELIVERY_QUEUES.queuedRunDispatch); + await s.enqueue(DELIVERY_QUEUES.nativeCancellation); + await vi.advanceTimersByTimeAsync(1); + expect(dispatch.run).toHaveBeenCalledTimes(1); + expect(commands.run).toHaveBeenCalledTimes(2); + expect(() => s.coordinator.register(DELIVERY_QUEUES.feedback, task())).toThrow("stopped"); + await s.coordinator.stop(); + await s.enqueue(DELIVERY_QUEUES.nativeCancellation); + await vi.advanceTimersByTimeAsync(10_000); + expect(commands.run).toHaveBeenCalledTimes(2); + expect(vi.getTimerCount()).toBe(0); + expect(idleWorkSnapshot().active).toBe(0); + }); it("awaits in-flight work on shutdown and releases the subscription for restart", async () => { const s = setup(), t = task(); let release!: () => void; diff --git a/server/src/__tests__/deployment-notifications.test.ts b/server/src/__tests__/deployment-notifications.test.ts new file mode 100644 index 00000000000..08d51689a4a --- /dev/null +++ b/server/src/__tests__/deployment-notifications.test.ts @@ -0,0 +1,102 @@ +import { randomUUID } from "node:crypto"; +import { setTimeout as delay } from "node:timers/promises"; +import { sql } from "drizzle-orm"; +import { afterAll, beforeAll, expect, it } from "vitest"; +import { createDb } from "@paperclipai/db"; +import { enableDatabaseWorkNotifications, publishDatabaseWorkHint, signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db/work-signals"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { deploymentNotifications } from "../services/deployment-notifications.js"; +import { setNativeExecutionActive } from "../services/native-runtime/native-active-runs.js"; + +let database: Awaited>; +let a: ReturnType, b: ReturnType; +beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("deployment-notifications-"); + a = createDb(database.connectionString); b = createDb(database.connectionString); + enableDatabaseWorkNotifications(b); + await a.execute(sql`create table deployment_notification_fixture (id text primary key)`); +}, 90_000); +afterAll(async () => { await a?.$client.end({ timeout: 1 }); await b?.$client.end({ timeout: 1 }); await database?.cleanup(); }); +async function eventually(check: () => Promise) { + const deadline = Date.now() + 5000; + while (!await check()) { if (Date.now() > deadline) throw new Error("Notification condition did not settle"); await delay(10); } +} +async function open(handoffMs = 120_000) { + const errors: unknown[] = []; + const coordinator = createDeliveryWorkCoordinator({ owner: a, canRun: () => true, onError: error => errors.push(error) }); + const bridge = deploymentNotifications(a, coordinator, { directDatabaseUrl: database.connectionString, handoffMs }); + await bridge.ready(); + return { bridge, errors, async stop() { await bridge.stop(); await coordinator.stop(); } }; +} +async function listenerCount() { + const [row] = await b.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity + where datname = current_database() and application_name = 'paperclip-deployment-notifications'`); + return row!.count; +} + +it("delivers committed hints across independent pools, without exposing an outer rollback or nested savepoint", async () => { + const fixture = await open(); + const observed: number[] = []; + const topic = `test-${randomUUID()}`; + const unsubscribe = subscribeDatabaseWork(a, topic, () => { + void a.execute(sql`select id from deployment_notification_fixture`).then(rows => observed.push(rows.length)); + }); + try { + await b.transaction(async outer => { + await outer.transaction(async nested => { + await signalDatabaseWork(nested, topic); + await nested.execute(sql`insert into deployment_notification_fixture values ('committed')`); + }); + expect(observed).toEqual([]); + }); + await eventually(async () => observed.length === 1); + expect(observed).toEqual([1]); + await expect(b.transaction(async tx => { + await signalDatabaseWork(tx, topic); + await tx.execute(sql`insert into deployment_notification_fixture values ('rollback')`); + throw new Error("rollback"); + })).rejects.toThrow("rollback"); + let barrier = false; + const stopBarrier = subscribeDatabaseWork(a, "test-barrier", () => { barrier = true; }); + await publishDatabaseWorkHint(b, "test-barrier"); + await eventually(async () => barrier); stopBarrier(); + expect(observed).toEqual([1]); + expect(fixture.errors).toEqual([]); + } finally { unsubscribe(); await fixture.stop(); } +}); + +it("reconstructs on listener reconnect and scopes live refresh hints to the company", async () => { + const fixture = await open(); + let refreshed = 0, other = 0, reconstructed = 0; + const companyId = randomUUID(); + const stopCompany = fixture.bridge.subscribeCompany(companyId, () => { refreshed++; }); + const stopOther = fixture.bridge.subscribeCompany(randomUUID(), () => { other++; }); + const stopQueue = subscribeDatabaseWork(a, "test-reconstruction", () => { reconstructed++; }); + try { + await publishDatabaseWorkHint(b, `live:${companyId}`); + await eventually(async () => refreshed === 1); + expect(other).toBe(0); + await b.execute(sql`select pg_terminate_backend(pid) from pg_stat_activity where datname = current_database() + and application_name = 'paperclip-deployment-notifications'`); + await eventually(async () => reconstructed > 0); + // Reconnect cannot know which hints were lost, so it refreshes both company + // subscriptions and existing durable queue consumers once. + expect(other).toBeGreaterThan(0); + } finally { stopCompany(); stopOther(); stopQueue(); await fixture.stop(); } +}); + +it("disconnects at the handoff deadline, reopens for active execution and becomes quiet again", async () => { + const fixture = await open(25), runId = randomUUID(); + const stopCompany = fixture.bridge.subscribeCompany(randomUUID(), () => {}); + try { + expect(await listenerCount()).toBe(1); + fixture.bridge.markReady(); + await eventually(async () => await listenerCount() === 0); + setNativeExecutionActive(runId, true); + await eventually(async () => await listenerCount() === 1); + setNativeExecutionActive(runId, false); + await eventually(async () => await listenerCount() === 0); + expect(fixture.errors).toEqual([]); + } finally { setNativeExecutionActive(runId, false); stopCompany(); await fixture.stop(); } +}); diff --git a/server/src/__tests__/ephemeral-server-contract.test.ts b/server/src/__tests__/ephemeral-server-contract.test.ts new file mode 100644 index 00000000000..72c319fac84 --- /dev/null +++ b/server/src/__tests__/ephemeral-server-contract.test.ts @@ -0,0 +1,42 @@ +import { afterEach, expect, it, vi } from "vitest"; +import { assertEphemeralServerConfig } from "../services/ephemeral-server-contract.js"; +import { assertEphemeralWorkspaceExecution, assertEphemeralNativeLifecycle } from "../services/remote-workspaces.js"; +import { assertEphemeralAgentInstructions } from "../services/agent-instructions.js"; + +const config = { databaseUrl: "postgresql://runtime:fixture@ep-example-pooler.neon.test/tenant", storageProvider: "s3" as const }; +const env = { PAPERCLIP_EPHEMERAL_SERVER: "true", PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true", + PAPERCLIP_DECISION_SIGNING_SECRET: "synthetic-decision-key-for-deployment-tests", + PAPERCLIP_SECRETS_MASTER_KEY: Buffer.alloc(32, 7).toString("base64"), // 32-byte fixture, never an operator key + PAPERCLIP_DATABASE_LISTEN_URL: "postgresql://runtime:fixture@ep-example.neon.test/tenant" }; +afterEach(() => vi.unstubAllEnvs()); +it("requires shared storage and stable secrets without changing volume-backed startup", () => { + expect(assertEphemeralServerConfig(config, env)).toBe(env.PAPERCLIP_DATABASE_LISTEN_URL); + expect(assertEphemeralServerConfig({ storageProvider: "local_disk", databaseUrl: undefined }, {})).toBeNull(); + for (const key of ["PAPERCLIP_DURABLE_AGENT_FILES", "PAPERCLIP_SHARED_UI_ASSETS", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_DECISION_SIGNING_SECRET", "PAPERCLIP_DATABASE_LISTEN_URL"]) { + expect(() => assertEphemeralServerConfig(config, { ...env, [key]: "" })).toThrow("Volume-free server configuration"); + } + expect(() => assertEphemeralServerConfig({ ...config, storageProvider: "local_disk" }, env)).toThrow("shared S3"); +}); +it("never connects the listener with migration privileges, another tenant, or through a transaction pooler", () => { + for (const url of [env.PAPERCLIP_DATABASE_LISTEN_URL.replace("runtime:", "migration:"), + env.PAPERCLIP_DATABASE_LISTEN_URL.replace("/tenant", "/neighbor"), config.databaseUrl, + env.PAPERCLIP_DATABASE_LISTEN_URL.replace("ep-example.", "ep-neighbor.")]) { + expect(() => assertEphemeralServerConfig(config, { ...env, PAPERCLIP_DATABASE_LISTEN_URL: url })).toThrow("runtime role"); + } +}); +it("rejects disk-dependent execution and instructions while retaining remote warm execution", () => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + const remote = { externalConversation: false, native: true, adapterType: "paperclip_runner", pluginTaskWorkspace: false, remoteAuthority: true }; + expect(() => assertEphemeralWorkspaceExecution(remote)).not.toThrow(); + expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false, pluginTaskWorkspace: true })).not.toThrow(); + for (const patch of [{ native: false }, { remoteAuthority: false }, { adapterType: "process" }]) { + expect(() => assertEphemeralWorkspaceExecution({ ...remote, ...patch })).toThrow("warm Paperclip Runner"); + } + expect(() => assertEphemeralNativeLifecycle("warm")).not.toThrow(); + expect(() => assertEphemeralNativeLifecycle("per_turn")).toThrow("warm Paperclip Runner"); + const agent = { id: "agent", companyId: "company", name: "Agent", adapterConfig: {} }; + expect(() => assertEphemeralAgentInstructions(agent)).not.toThrow(); + expect(() => assertEphemeralAgentInstructions({ ...agent, adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/old-volume" } })).toThrow("managed agent instructions"); + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "false"); + expect(() => assertEphemeralWorkspaceExecution({ ...remote, remoteAuthority: false })).not.toThrow(); +}); diff --git a/server/src/__tests__/ephemeral-server-data.test.ts b/server/src/__tests__/ephemeral-server-data.test.ts new file mode 100644 index 00000000000..6c42c6de952 --- /dev/null +++ b/server/src/__tests__/ephemeral-server-data.test.ts @@ -0,0 +1,42 @@ +import { randomUUID } from "node:crypto"; +import { beforeAll, afterAll, afterEach, expect, it, vi } from "vitest"; +import { eq } from "drizzle-orm"; +import { agentFileRoots, agents, companies, executionWorkspaces, heartbeatRuns, providerTraceRecords, createDb } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { assertEphemeralServerData } from "../services/ephemeral-server-contract.js"; +import { initializeDurableAgentFiles } from "../services/agent-file-durability.js"; +vi.mock("../storage/index.js", () => ({ getStorageService: () => ({ provider: "s3" }) })); +let database: Awaited>; +let db: ReturnType; +beforeAll(async () => { database = await startEmbeddedPostgresTestDatabase("ephemeral-data-"); db = createDb(database.connectionString); }, 90_000); +afterAll(async () => { await db?.$client.end({ timeout: 1 }); await database?.cleanup(); }); +afterEach(() => vi.unstubAllEnvs()); +it("refuses volume-dependent rows before readiness and accepts their durable replacement", async () => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true"); + await expect(assertEphemeralServerData(db)).resolves.toBeUndefined(); + const companyId = randomUUID(), agentId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Deployment gate", issuePrefix: "GATE" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Legacy files" }); + await expect(assertEphemeralServerData(db)).rejects.toThrow("agent_files_require_migration"); + await db.transaction(tx => initializeDurableAgentFiles(tx, { id: agentId, companyId })); + await expect(assertEphemeralServerData(db)).resolves.toBeUndefined(); + await db.update(agents).set({ adapterConfig: { instructionsBundleMode: "external", instructionsRootPath: "/lost-disk" } }).where(eq(agents.id, agentId)); + await expect(assertEphemeralServerData(db)).rejects.toThrow("external_agent_instructions"); + await db.update(agents).set({ adapterConfig: {} }).where(eq(agents.id, agentId)); + const [workspace] = await db.insert(executionWorkspaces).values({ companyId, mode: "isolated", strategyType: "directory", name: "Legacy workspace", cwd: "/lost-disk/work" }).returning(); + await expect(assertEphemeralServerData(db)).rejects.toThrow("workspaces_require_migration"); + await db.delete(executionWorkspaces).where(eq(executionWorkspaces.id, workspace!.id)); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, invocationSource: "on_demand", status: "succeeded", logStore: "local_file", logRef: "old-log" }).returning(); + await expect(assertEphemeralServerData(db)).rejects.toThrow("run_logs_require_migration"); + await db.update(heartbeatRuns).set({ logStore: "s3_segments", logRef: "committed-object-ref" }).where(eq(heartbeatRuns.id, run!.id)); + const [trace] = await db.insert(providerTraceRecords).values({ companyId, runId: run!.id, provider: "fixture", requestedBy: "fixture", traceRef: "old-trace", expiresAt: new Date(Date.now() + 60_000) }).returning(); + await expect(assertEphemeralServerData(db)).rejects.toThrow("provider_traces_require_migration"); + await db.update(providerTraceRecords).set({ deletedAt: new Date(), status: "deleted" }).where(eq(providerTraceRecords.id, trace!.id)); + await expect(assertEphemeralServerData(db)).resolves.toBeUndefined(); + await expect(db.transaction(async tx => { + const [created] = await tx.insert(agents).values({ companyId, name: "Rolled back hire" }).returning(); + await initializeDurableAgentFiles(tx, created!); + throw new Error("abort hire"); + })).rejects.toThrow("abort hire"); + expect(await db.select().from(agentFileRoots)).toHaveLength(1); +}); diff --git a/server/src/__tests__/idle-admission.test.ts b/server/src/__tests__/idle-admission.test.ts index 4c8e86775f3..6ab60153332 100644 --- a/server/src/__tests__/idle-admission.test.ts +++ b/server/src/__tests__/idle-admission.test.ts @@ -3,7 +3,7 @@ import request from "supertest"; import { request as httpRequest } from "node:http"; import { createServer as createViteServer } from "vite"; import { afterEach, describe, expect, it } from "vitest"; -import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js"; +import { drainHttpRequestWork, idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js"; import { idleWorkSnapshot, startTaskDrain, stopTaskDrain, readTaskDrain, beginIdleTrackedWork } from "../services/task-admission.js"; function deferred() { @@ -43,10 +43,15 @@ describe("idle admission", () => { trackIdleRequestHandlers(server); await request(server).post("/api/work").expect(202); expect(idleWorkSnapshot().active).toBe(1); + let drained = false; + const drain = drainHttpRequestWork(server, new AbortController().signal).then(() => { drained = true; }); + await new Promise(resolve => setImmediate(resolve)); + expect(drained).toBe(false); startTaskDrain({ purpose: "idle", ttlMs: 60_000 }); await request(server).post("/api/work").expect(503); done.resolve(); - await new Promise(resolve => setImmediate(resolve)); + await drain; + expect(drained).toBe(true); expect(idleWorkSnapshot().active).toBe(0); }); diff --git a/server/src/__tests__/liveness.test.ts b/server/src/__tests__/liveness.test.ts index d8ce096df56..8e7e635f3f3 100644 --- a/server/src/__tests__/liveness.test.ts +++ b/server/src/__tests__/liveness.test.ts @@ -4,6 +4,7 @@ import request from "supertest"; import { afterEach, describe, expect, it, vi } from "vitest"; import type { Db } from "@paperclipai/db"; import { livenessRoutes } from "../routes/liveness.js"; +import { readinessRoutes } from "../routes/ready.js"; import { healthRoutes } from "../routes/health.js"; import { privateHostnameGuard } from "../middleware/private-hostname-guard.js"; import { cloudRuntimeIdentityMiddleware } from "../middleware/cloud-runtime-identity.js"; @@ -28,6 +29,7 @@ function fixture(standby = false) { app.use(privateHostnameGuard({ enabled: true, allowedHostnames: ["allowed.example"], bindHost: "0.0.0.0" })); app.use(cloudRuntimeIdentityMiddleware(db)); app.use("/api/liveness", livenessRoutes()); + app.use("/api/ready", readinessRoutes()); app.use(cloudWarmStandbyMiddleware(() => standby, health)); app.use(auth); app.use("/api/health", health); @@ -38,6 +40,24 @@ function fixture(standby = false) { } describe("database-free process liveness", () => { + it.each([false, true])("gates deployment readiness on bootstrap and drain without waking SQL (standby: %s)", async standby => { + const f = fixture(standby); + const before = idleWorkSnapshot(); + startTaskDrain({ purpose: "idle", ttlMs: 60_000 }); + for (const phase of ["starting", "recovering", "ready", "draining"] as const) { + setStartupRecoveryPhase(phase); + const status = phase === "ready" ? 200 : 503; + const response = await f.http.get("/api/ready").set("authorization", "Bearer invalid-fixture").expect(status); + expect(response.body).toEqual({ status: phase }); + expect(response.headers["cache-control"]).toBe("no-store"); + await f.http.head("/api/ready").expect(status); + } + await f.http.post("/api/ready").expect(503); + await f.http.get("/api/ready/details").expect(503); + expect(f.auth).not.toHaveBeenCalled(); + expect(f.db.execute).not.toHaveBeenCalled(); + expect(idleWorkSnapshot()).toEqual(before); + }); it.each([false, true])("does not resolve auth or query SQL with an unavailable database (standby: %s)", async standby => { const f = fixture(standby); for (let i = 0; i < 3; i++) { diff --git a/server/src/__tests__/native-cancellation-delivery.test.ts b/server/src/__tests__/native-cancellation-delivery.test.ts new file mode 100644 index 00000000000..b5f8261d1bb --- /dev/null +++ b/server/src/__tests__/native-cancellation-delivery.test.ts @@ -0,0 +1,78 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, afterEach, expect, it, vi } from "vitest"; +import { agents, companies, issues, heartbeatRuns, nativeRunFinalizations, createDb } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { deploymentNotifications } from "../services/deployment-notifications.js"; +import { nativeCancellationHasRemoteOwner, notifyNativeCancellation, registerNativeCancellationDelivery, waitForNativeCancellation } from "../services/native-runtime/native-cancellation-delivery.js"; +import { currentNativeControllerIdentity } from "../services/native-runtime/native-restart-recovery.js"; +import { setNativeExecutionActive } from "../services/native-runtime/native-active-runs.js"; +const provider = vi.hoisted(() => ({ stop: vi.fn() })); +vi.mock("../services/native-runtime/native-session-executor.js", () => ({ cancelNativeSession: (...args: unknown[]) => provider.stop(...args) })); +let database: Awaited>; +let a: ReturnType, b: ReturnType; +let companyId: string, runId: string, intentId: string; +let bridgeA: ReturnType, bridgeB: ReturnType; +let coordinatorA: ReturnType, coordinatorB: ReturnType; +let worker: ReturnType; +const errors: unknown[] = []; +beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("cancellation-delivery-"); + a = createDb(database.connectionString); b = createDb(database.connectionString); + companyId = randomUUID(); runId = randomUUID(); intentId = `native-cancellation:${randomUUID()}`; + const agentId = randomUUID(), issueId = randomUUID(), identity = await currentNativeControllerIdentity(); + await a.insert(companies).values({ id: companyId, name: "Cancellation fixture", issuePrefix: "STOP" }); + await a.insert(agents).values({ id: agentId, companyId, name: "Remote fixture" }); + await a.insert(issues).values({ id: issueId, companyId, title: "Stop fixture", status: "in_progress" }); + await a.insert(heartbeatRuns).values({ id: runId, companyId, agentId, runtimeMode: "native", status: "running", nativeIssueId: issueId }); + await a.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "executing", controllerBootId: identity.bootId, + leaseOwner: "fixture-owner", leaseExpiresAt: new Date(Date.now() + 60_000) }); + coordinatorA = createDeliveryWorkCoordinator({ owner: a, canRun: () => true, onError: error => errors.push(error) }); + coordinatorB = createDeliveryWorkCoordinator({ owner: b, canRun: () => true, onError: error => errors.push(error) }); + bridgeA = deploymentNotifications(a, coordinatorA, { directDatabaseUrl: database.connectionString }); + bridgeB = deploymentNotifications(b, coordinatorB, { directDatabaseUrl: database.connectionString }); + await Promise.all([bridgeA.ready(), bridgeB.ready()]); + setNativeExecutionActive(runId, true); + provider.stop.mockImplementation(async (id: string, _reason: string, options: { db: typeof a }) => { + expect(id).toBe(runId); expect(options.db).toBe(a); + // Simulated provider edge; the production consumer, PG transport, owner + // selection, intent and accepting-server ACK wait are exercised for real. + await a.transaction(async tx => { + await notifyNativeCancellation(tx); + await tx.update(heartbeatRuns).set({ resultJson: { nativeCancellation: { intentId, dispatchState: "acknowledged", dispatched: true, + acknowledgementAuditId: "fixture-audit" } } }).where(eq(heartbeatRuns.id, runId)); + }); + }); + worker = registerNativeCancellationDelivery(a, coordinatorA); await worker.ready; +}, 90_000); +afterEach(() => vi.unstubAllEnvs()); +afterAll(async () => { + setNativeExecutionActive(runId, false); await worker?.stop(); + await Promise.all([bridgeA?.stop(), bridgeB?.stop()]); + await Promise.all([coordinatorA?.stop(), coordinatorB?.stop()]); + await Promise.all([a?.$client.end({ timeout: 1 }), b?.$client.end({ timeout: 1 })]); await database?.cleanup(); +}); +it("routes a committed Stop from B to its live owner A and returns only A's acknowledgement", async () => { + await b.transaction(async tx => { + await notifyNativeCancellation(tx); + await tx.update(heartbeatRuns).set({ resultJson: { nativeCancellation: { schema: "paperclip.native-cancellation.v1", intentId, + scope: "run", dispatchState: "pending" } } }).where(eq(heartbeatRuns.id, runId)); + }); + await expect(waitForNativeCancellation(b, { companyId, runId, intentId })).resolves.toEqual({ dispatched: true, decisionId: null, auditId: "fixture-audit" }); + expect(provider.stop).toHaveBeenCalledTimes(1); expect(errors).toEqual([]); +}); +it("fails closed on a different intent and never accepts a no-op acknowledgement", async () => { + await expect(waitForNativeCancellation(b, { companyId, runId, intentId: "wrong-intent" })).rejects.toThrow("intent_conflict"); + await b.update(heartbeatRuns).set({ resultJson: { nativeCancellation: { intentId, dispatchState: "acknowledged", dispatched: false, + acknowledgementAuditId: "not-dispatched" } } }).where(eq(heartbeatRuns.id, runId)); + await expect(waitForNativeCancellation(b, { companyId, runId, intentId }, 50)).rejects.toThrow("pending_recovery"); +}); +it("distinguishes the current controller, a live replacement owner, and an expired owner", async () => { + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(false); + await b.update(nativeRunFinalizations).set({ controllerBootId: "another-controller" }).where(eq(nativeRunFinalizations.runId, runId)); + expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(true); + await b.update(nativeRunFinalizations).set({ leaseExpiresAt: new Date(0) }).where(eq(nativeRunFinalizations.runId, runId)); + expect(await nativeCancellationHasRemoteOwner(b, runId)).toBe(false); +}); diff --git a/server/src/__tests__/native-runtime-response-delivery.test.ts b/server/src/__tests__/native-runtime-response-delivery.test.ts new file mode 100644 index 00000000000..5930736f8dc --- /dev/null +++ b/server/src/__tests__/native-runtime-response-delivery.test.ts @@ -0,0 +1,126 @@ +import { createHash, randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, expect, it, vi } from "vitest"; +import { agents, companies, companyMemberships, instanceUserRoles, issues, heartbeatRuns, heartbeatRunEvents, + nativeRunFinalizations, createDb } from "@paperclipai/db"; +import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { deploymentNotifications } from "../services/deployment-notifications.js"; +import { enqueueNativeRuntimeResponse, registerNativeRuntimeResponseDelivery } from "../services/native-runtime/native-runtime-response-delivery.js"; +import { currentNativeControllerIdentity } from "../services/native-runtime/native-restart-recovery.js"; +import { setNativeExecutionActive } from "../services/native-runtime/native-active-runs.js"; +import type { PendingNativeRuntimeRequest } from "../services/native-runtime/runtime-request-resolution-authority.js"; +const provider = vi.hoisted(() => ({ resolve: vi.fn(), accepted: vi.fn() })); +vi.mock("../services/native-runtime/native-session-executor.js", () => ({ + resolveNativeRuntimeRequest: (...args: unknown[]) => provider.resolve(...args), + NativeRuntimeRequestResolutionError: class extends Error { constructor(readonly code: string) { super(code); } }, +})); +let database: Awaited>; +let a: ReturnType, b: ReturnType; +let companyId: string, runId: string, agentId: string, userId: string; +let bridgeA: ReturnType, bridgeB: ReturnType; +let coordinatorA: ReturnType, coordinatorB: ReturnType; +let worker: ReturnType; +const errors: unknown[] = []; +let seq = 0; +beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("runtime-response-delivery-"); + a = createDb(database.connectionString); b = createDb(database.connectionString); + companyId = randomUUID(); runId = randomUUID(); agentId = randomUUID(); userId = randomUUID(); + const issueId = randomUUID(), identity = await currentNativeControllerIdentity(); + await a.insert(companies).values({ id: companyId, name: "Response fixture", issuePrefix: "RESP" }); + await a.insert(agents).values({ id: agentId, companyId, name: "Remote fixture" }); + await a.insert(issues).values({ id: issueId, companyId, title: "Response fixture", status: "in_progress" }); + await a.insert(heartbeatRuns).values({ id: runId, companyId, agentId, runtimeMode: "native", status: "running", nativeIssueId: issueId }); + await a.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "executing", controllerBootId: identity.bootId, + leaseOwner: "fixture-owner", leaseExpiresAt: new Date(Date.now() + 60_000) }); + await a.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "operator" }); + await a.insert(instanceUserRoles).values({ userId, role: "instance_admin" }); + coordinatorA = createDeliveryWorkCoordinator({ owner: a, canRun: () => true, onError: error => errors.push(error) }); + coordinatorB = createDeliveryWorkCoordinator({ owner: b, canRun: () => true, onError: error => errors.push(error) }); + bridgeA = deploymentNotifications(a, coordinatorA, { directDatabaseUrl: database.connectionString }); + bridgeB = deploymentNotifications(b, coordinatorB, { directDatabaseUrl: database.connectionString }); + await Promise.all([bridgeA.ready(), bridgeB.ready()]); + provider.resolve.mockImplementation(async (input: { authorizeBeforeDispatch: () => Promise; commandId: string }) => { + await input.authorizeBeforeDispatch(); + provider.accepted(input.commandId); + return { commandId: input.commandId }; + }); + worker = registerNativeRuntimeResponseDelivery(a, coordinatorA); await worker.ready; +}, 90_000); +afterAll(async () => { + setNativeExecutionActive(runId, false); await worker?.stop(); + await Promise.all([bridgeA?.stop(), bridgeB?.stop()]); + await Promise.all([coordinatorA?.stop(), coordinatorB?.stop()]); + await Promise.all([a?.$client.end({ timeout: 1 }), b?.$client.end({ timeout: 1 })]); await database?.cleanup(); +}); +async function pending(): Promise { + const request = { companyId, runId, requestId: randomUUID(), requestKind: "command_approval" as const, + turnId: "turn-fixture", resolverPolicy: "instance_admin" as const }; + await a.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: ++seq, eventType: "runtime_request.created", + payload: { prpEvent: { schema: "paperclip.prp.event.v1", eventType: "runtime_request.created", sourceKind: "runner", runId, + payload: { request: { ...request, status: "pending" } } } } }); + return request; +} +const enqueue = (request: PendingNativeRuntimeRequest) => enqueueNativeRuntimeResponse(b, { request, + actor: { type: "user", userId, isInstanceAdmin: true }, localImplicit: false, resolution: { action: "decline" } }); +async function state(requestId: string) { + const [run] = await b.select({ result: heartbeatRuns.resultJson }).from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)); + const entries = run.result?.nativeRuntimeResponses as Record; + return entries?.[createHash("sha256").update(requestId).digest("hex")]?.state; +} +it("persists B's response before an owner is attached, then dispatches once through A", async () => { + const request = await pending(); + const saved = await enqueue(request); + expect(await enqueue(request)).toEqual(saved); + expect(await state(request.requestId)).toBe("pending"); + expect(provider.resolve).not.toHaveBeenCalled(); + setNativeExecutionActive(runId, true); + await expect.poll(() => state(request.requestId)).toBe("queued"); + expect(provider.resolve).toHaveBeenCalledTimes(1); + expect(provider.resolve.mock.calls[0]?.[0].commandId).toBe(saved.commandId); + await enqueue(request); + expect(provider.resolve).toHaveBeenCalledTimes(1); + await expect(enqueueNativeRuntimeResponse(b, { request, actor: { type: "user", userId, isInstanceAdmin: true }, + localImplicit: false, resolution: { action: "cancel" } })).rejects.toThrow("different response"); + expect(errors).toEqual([]); +}); +it("rechecks company access even for an administrator before dispatch", async () => { + setNativeExecutionActive(runId, false); + const request = await pending(); await enqueue(request); + await a.update(companyMemberships).set({ status: "inactive" }).where(eq(companyMemberships.principalId, userId)); + setNativeExecutionActive(runId, true); + await expect.poll(() => state(request.requestId)).toBe("denied"); + expect(provider.accepted).not.toHaveBeenCalled(); + expect(errors).toEqual([]); +}); +it("does not dispatch an answer after its request has become terminal", async () => { + setNativeExecutionActive(runId, false); + const request = await pending(); await enqueue(request); + await a.insert(heartbeatRunEvents).values({ companyId, runId, agentId, seq: ++seq, eventType: "runtime_request.resolved", + payload: { prpEvent: { payload: { requestId: request.requestId } } } }); + setNativeExecutionActive(runId, true); + await expect.poll(() => state(request.requestId)).toBe("obsolete"); + expect(provider.accepted).not.toHaveBeenCalled(); + expect(errors).toEqual([]); +}); + +it("retires a stale provider turn and still delivers the next pending answer", async () => { + setNativeExecutionActive(runId, false); + await a.update(companyMemberships).set({ status: "active" }).where(eq(companyMemberships.principalId, userId)); + const stale = await pending(), current = await pending(); + await enqueue(stale); + await enqueue(current); + const { NativeRuntimeRequestResolutionError } = await import("../services/native-runtime/native-session-executor.js"); + provider.resolve.mockImplementation(async (input: { requestId: string; authorizeBeforeDispatch: () => Promise; commandId: string }) => { + if (input.requestId === stale.requestId) throw new NativeRuntimeRequestResolutionError("runtime_request_stale_turn", "fixture stale turn"); + await input.authorizeBeforeDispatch(); + provider.accepted(input.commandId); + return { commandId: input.commandId }; + }); + setNativeExecutionActive(runId, true); + await expect.poll(() => state(stale.requestId)).toBe("obsolete"); + await expect.poll(() => state(current.requestId)).toBe("queued"); + expect(provider.accepted).toHaveBeenCalledTimes(1); + expect(errors).toEqual([]); +}); diff --git a/server/src/__tests__/openapi-routes.test.ts b/server/src/__tests__/openapi-routes.test.ts index 45381053729..de2b3c0042d 100644 --- a/server/src/__tests__/openapi-routes.test.ts +++ b/server/src/__tests__/openapi-routes.test.ts @@ -65,6 +65,7 @@ const apiPrefixes: Record = { "folders.ts": "/api", "goals.ts": "/api", "health.ts": "/api/health", + "ready.ts": "/api/ready", "liveness.ts": "/api/liveness", "inbox-agent-policy.ts": "/api", "inbox-dismissals.ts": "/api", @@ -202,7 +203,7 @@ function resolveMountedPath(file: string, prefix: string, routePath: string) { ) { return routePath; } - if ((file === "companies.ts" || file === "health.ts" || file === "liveness.ts") && routePath === "/") { + if ((file === "companies.ts" || file === "health.ts" || file === "liveness.ts" || file === "ready.ts") && routePath === "/") { return prefix; } if (file === "companies.ts" || file === "health.ts") { diff --git a/server/src/__tests__/retained-ui-assets.test.ts b/server/src/__tests__/retained-ui-assets.test.ts new file mode 100644 index 00000000000..221359f27a0 --- /dev/null +++ b/server/src/__tests__/retained-ui-assets.test.ts @@ -0,0 +1,68 @@ +import express from "express"; +import request from "supertest"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { Readable } from "node:stream"; +import { afterEach, expect, it } from "vitest"; +import { retainedUiAssets } from "../retained-ui-assets.js"; +import type { StorageProvider } from "../storage/types.js"; +import { staticUiFiles } from "../middleware/static-ui.js"; +import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const roots: string[] = []; +afterEach(async () => { await Promise.all(roots.splice(0).map(root => fs.rm(root, { recursive: true, force: true }))); }); +async function directory() { const root = await fs.mkdtemp(path.join(os.tmpdir(), "retained-ui-")); roots.push(root); return root; } +function storage() { + const objects = new Map(); + const provider: StorageProvider = { + id: "s3", putObject: async () => { throw new Error("Expected conditional publication"); }, + async putObjectConditional(input) { + if (objects.has(input.objectKey)) throw new Error("Precondition failed"); + const chunks: Buffer[] = []; + for await (const chunk of input.body) chunks.push(Buffer.from(chunk)); + objects.set(input.objectKey, Buffer.concat(chunks)); return { etag: '"fixture-etag"' }; + }, + async getObject(input) { + const bytes = objects.get(input.objectKey); + if (!bytes) throw Object.assign(new Error("Object missing"), { status: 404 }); + return { stream: Readable.from([bytes]), contentLength: bytes.length, etag: '"fixture-etag"' }; + }, + async headObject(input) { return { exists: objects.has(input.objectKey) }; }, + async deleteObject(input) { objects.delete(input.objectKey); }, + }; + return { provider, objects }; +} + +it("serves an old tab's lazy chunk from shared storage after removing its entire old build", async () => { + const a = await directory(), b = await directory(), { provider } = storage(); + await fs.writeFile(path.join(a, "lazy-oldhash.js"), "export const revision = 'A';"); + const retained = retainedUiAssets(provider); + await retained.publish(a); + await retained.publish(a); // Concurrent/new same-image boots do not rewrite bytes. + await fs.rm(a, { recursive: true }); + await fs.writeFile(path.join(b, "lazy-newhash.js"), "export const revision = 'B';"); + await retained.publish(b); + const app = express(), before = idleWorkSnapshot().active; + app.use(idleAdmissionMiddleware); + app.use("/assets", staticUiFiles(b), retained.serve); + trackIdleRequestHandlers(app); + await request(app).get("/assets/lazy-oldhash.js").expect(200).expect("Content-Type", /javascript/) + .expect("Cache-Control", /immutable/).expect("export const revision = 'A';"); + await request(app).get("/assets/lazy-newhash.js").expect(200).expect("export const revision = 'B';"); + await request(app).head("/assets/lazy-oldhash.js").expect(200); + await request(app).get("/assets/missing.js").expect(404); + await request(app).get("/assets/%2e%2e%2fsecrets").expect(404); + expect(idleWorkSnapshot().active).toBe(before); +}); + +it("refuses a build which reuses a published name for different bytes", async () => { + const root = await directory(), { provider, objects } = storage(); + const retained = retainedUiAssets(provider); + await fs.writeFile(path.join(root, "chunk-hash.js"), "original"); + await retained.publish(root); + await fs.writeFile(path.join(root, "chunk-hash.js"), "collision"); + await expect(retained.publish(root)).rejects.toThrow("collides"); + expect(objects.get("ui-assets/v1/chunk-hash.js")?.toString()).toBe("original"); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index dabee1c2f03..afbcbbab33a 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -391,7 +391,7 @@ vi.mock("../middleware/logger.js", () => ({ })); vi.mock("../realtime/live-events-ws.js", () => ({ - setupLiveEventsWebSocketServer: vi.fn(), + setupLiveEventsWebSocketServer: vi.fn(() => ({ clients: new Set(), close: vi.fn() })), })); vi.mock("../services/attention-retention.js", async importOriginal => ({ diff --git a/server/src/app.ts b/server/src/app.ts index 8a985cbc2de..06adc15edb8 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -13,6 +13,9 @@ import { createLifecycleDriver } from "./services/agent-lifecycle-driver.js"; import { startAgentLifecycle } from "./services/agent-lifecycle.js"; import { idleAdmissionMiddleware, trackIdleRequestHandlers } from "./middleware/idle-admission.js"; import { staticUiFiles } from "./middleware/static-ui.js"; +import { retainedUiAssets } from "./retained-ui-assets.js"; +import { createStorageProviderFromConfig } from "./storage/provider-registry.js"; +import { loadStorageConfig } from "./storage/config.js"; import { isIdleTaskDrainActive, trackIdleWork } from "./services/task-admission.js"; import { customerSuccessRoutes } from "./routes/customer-success.js"; import { cloudWarmStandbyMiddleware } from "./middleware/cloud-warm-standby.js"; @@ -80,6 +83,7 @@ import { import { registerImportSpoolCleanupWork } from "./services/import-spool-cleanup-work.js"; import { companyTransferRunService } from "./services/company-transfer-runs.js"; import { healthRoutes } from "./routes/health.js"; +import { readinessRoutes } from "./routes/ready.js"; import { livenessRoutes } from "./routes/liveness.js"; import { cloudRuntimeIdentityMiddleware } from "./middleware/cloud-runtime-identity.js"; import { cloudControlMiddleware } from "./middleware/cloud-control.js"; @@ -536,6 +540,7 @@ export async function createApp( // Routine probes stop before session/actor resolution, even after claim. // Keep hostname validation and the signed-bootstrap endpoint guard above. app.use("/api/liveness", livenessRoutes()); + app.use("/api/ready", readinessRoutes()); // A signed claim above commits identity before any normal request can seed // company data. Unclaimed probes bypass session resolution as well as SQL. app.use(cloudWarmStandbyMiddleware(isWarmStandby, health, staticUi)); @@ -1058,6 +1063,11 @@ export async function createApp( immutable: true, }), ); + if (process.env.PAPERCLIP_SHARED_UI_ASSETS === "true") { + const retained = retainedUiAssets(createStorageProviderFromConfig(loadStorageConfig())); + await retained.publish(path.join(uiDist, "assets")); + staticUi.use("/assets", retained.serve); + } // Serve root/index through the same runtime HTML transform as SPA routes. staticUi.get(["/", "/index.html"], (_req, res) => { res.type("html").set("Cache-Control", "no-cache").send(readBrandedStaticIndexHtml(uiDist)); diff --git a/server/src/index.ts b/server/src/index.ts index 51af366e482..dfea5d79693 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,5 @@ +import { registerNativeCancellationDelivery } from "./services/native-runtime/native-cancellation-delivery.js"; +import { registerNativeRuntimeResponseDelivery } from "./services/native-runtime/native-runtime-response-delivery.js"; import { attentionRetentionService, registerAttentionRetentionWork } from "./services/attention-retention.js"; import { registerRoutineExecutionStatusWork } from "./services/routine-execution-status-work.js"; import { routineExecutionStatusService } from "./services/routine-run-status.js"; @@ -54,6 +56,8 @@ import { registerBudgetEnforcementWork, registerCostAccountingWork } from "./ser import { caseHandoffDeliveryService } from "./services/case-handoff-delivery.js"; import { registerScheduledWork } from "./services/scheduled-work.js"; import type { createDeliveryWorkCoordinator } from "./services/delivery-work-coordinator.js"; +import { assertEphemeralServerConfig, assertEphemeralServerData } from "./services/ephemeral-server-contract.js"; +import { deploymentNotifications } from "./services/deployment-notifications.js"; import { registerTerminalWorkspaceWork, TERMINAL_WORKSPACE_RETRY_MS } from "./services/terminal-workspace-work.js"; import { isIdleTaskDrainActive, readTaskDrain, beginIdleTrackedWork, trackIdleWork, registerTaskDrainExpiryWork } from "./services/task-admission.js"; import { markIdleStartupComplete, registerIdleLocalWorkProbe } from "./services/idle-local-work.js"; @@ -107,9 +111,11 @@ import { instanceUserRoles, closeLivenessDb, closeCheckpointDb, + closeFileStorageDb, } from "@paperclipai/db"; import detectPort from "detect-port"; import { createApp } from "./app.js"; +import { drainHttpRequestWork } from "./middleware/idle-admission.js"; import { initializeCloudRuntimeConfig } from "./services/cloud-runtime-config.js"; import { loadCloudBootstrapConfig, loadConfig, type Config } from "./config.js"; import { logger } from "./middleware/logger.js"; @@ -299,6 +305,7 @@ async function startServerWithDatabaseTeardown( await sentryReady; const startupConfig = isCloudManagedInstance() ? loadCloudBootstrapConfig() : loadConfig(); if ("deploymentMode" in startupConfig) { + assertEphemeralServerConfig(startupConfig); ensureDecisionSigningSecret(); initializeRuntimeConfigConsumers(startupConfig); } @@ -722,7 +729,7 @@ async function startServerWithDatabaseTeardown( const closeDatabaseClients = async () => { setSentryRunLinkResolver(null); const clients = pluginMigrationDb === db ? [db] : [db, pluginMigrationDb]; - await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5), closeCheckpointDb(db, 5)]); + await Promise.all([...clients.map((client) => endDatabaseClient(client, 5)), closeLivenessDb(db, 5), closeCheckpointDb(db, 5), closeFileStorageDb(db, 5)]); }; startupDatabase.close = closeDatabaseClients; setSentryRunLinkResolver(createSentryRunLinkResolver(db)); @@ -731,6 +738,7 @@ async function startServerWithDatabaseTeardown( await initializeCloudRuntimeConfig(db as any); let config = "deploymentMode" in startupConfig ? startupConfig : loadConfig(); + const deploymentListenUrl = assertEphemeralServerConfig(config); // Cloud storage depends on the canonical DB snapshot. Validate it before // signing keys or other configuration consumers can mutate local state. if (isCloudManagedInstance()) ensureDecisionSigningSecret(); @@ -1066,13 +1074,25 @@ async function startServerWithDatabaseTeardown( process.env.PAPERCLIP_RUNTIME_API_CANDIDATES_JSON = JSON.stringify(runtimeApiCandidates); process.env.PAPERCLIP_API_URL = configuredApiUrl; + const deliveryWork = app.locals.deliveryWork as ReturnType; + const deploymentHints = deploymentListenUrl !== null + ? deploymentNotifications(db as any, deliveryWork, { + directDatabaseUrl: deploymentListenUrl, isWarmStandby, + }) : null; + const cancellationDelivery = deploymentListenUrl !== null ? registerNativeCancellationDelivery(db as any, deliveryWork) : null; + const runtimeResponseDelivery = deploymentListenUrl !== null ? registerNativeRuntimeResponseDelivery(db as any, deliveryWork) : null; + await deploymentHints?.ready(); + await cancellationDelivery?.ready; + await runtimeResponseDelivery?.ready; + let startupListenerBound = false; try { setupEnvironmentCustomImageTerminalWebSocketServer(server, db as any, { pluginWorkerManager, }); - setupLiveEventsWebSocketServer(server, db as any, { + const liveEventsWebSocketServer = setupLiveEventsWebSocketServer(server, db as any, { deploymentMode: config.deploymentMode, + subscribeInvalidation: deploymentHints?.subscribeCompany, resolveSessionFromHeaders, // Cloud-proxied browsers carry trusted x-paperclip-cloud-* headers instead // of a local Better Auth session; without this lane every live-events @@ -1305,7 +1325,6 @@ async function startServerWithDatabaseTeardown( heartbeat: environmentLeaseCleanupHeartbeat, resolveNativeQuestion: (interaction) => deliverNativeQuestionResponse(db as any, interaction), }); - const deliveryWork = app.locals.deliveryWork as ReturnType; externalObjects.registerRefreshWork(deliveryWork, () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby()); if (config.nativeHarnessBackupExpirationEnabled) { @@ -1912,8 +1931,16 @@ async function startServerWithDatabaseTeardown( throw err; } + if (deploymentListenUrl !== null) { + await assertEphemeralServerData(db as any); + const qualification = await qualifyCloudConfigurationForCurrentImage(db as any); + if (qualification && !qualification.configurationPackagesEligible) { + throw new Error(`Volume-free server packages are not eligible: ${qualification.blockers.map(blocker => blocker.code).join(", ")}`); + } + } markIdleStartupComplete({ scheduledBackups: config.databaseBackupEnabled }); setStartupRecoveryPhase("ready"); + deploymentHints?.markReady(); logger.info(`Server startup recovery complete on ${config.host}:${listenPort}`); void systemdNotify(["--ready", `--status=Listening on ${config.host}:${listenPort}`]).then((notified) => { if (notified) logger.info("Notified systemd that Paperclip is ready"); @@ -1964,10 +1991,17 @@ async function startServerWithDatabaseTeardown( ); } - const shutdown = async ( + const performShutdown = async ( signal: "SIGINT" | "SIGTERM", exitProcess: boolean, ) => { + setStartupRecoveryPhase("draining"); + for (const client of liveEventsWebSocketServer.clients) client.close(1012, "Server restarting"); + liveEventsWebSocketServer.close(); + // Finish admitted HTTP requests while their services, workers and database + // are still available. Railway must switch routing before signaling A. + await closeHttpListenerForShutdown({ server, signal, timeoutMs: 30_000, log: logger, + drainRequests: abort => drainHttpRequestWork(app, abort) }); await systemdNotify(["--stopping", `--status=Stopping after ${signal}`]); heartbeatSchedulerStopped = true; skillVersionRetentionWorker?.unsubscribe(); @@ -1978,7 +2012,12 @@ async function startServerWithDatabaseTeardown( unsubscribeNativeResponsePresentation?.(); unsubscribeNativeWorkspaceFinalization?.(); unsubscribeRetainedNativeSessionCleanup?.(); - await deliveryWork.stop(); + // B can accept Stop/answers while A still owns an execution. Preserve the + // owner command workers and their transport until the handoff below ends. + await deliveryWork.stop(deploymentHints ? new Set([ + DELIVERY_QUEUES.nativeCancellation, DELIVERY_QUEUES.nativeRuntimeResponses, DELIVERY_QUEUES.question, + DELIVERY_QUEUES.deploymentNotifications, DELIVERY_QUEUES.liveInvalidations, + ]) : undefined); await runEvidenceMigrationWork?.stop(); await providerTraceRetentionWork.stop(); await skillVersionRetentionWorker?.dispose(); @@ -2029,6 +2068,11 @@ async function startServerWithDatabaseTeardown( }); } + await cancellationDelivery?.stop(); + await runtimeResponseDelivery?.stop(); + await deploymentHints?.stop(); + await deliveryWork.stop(); + // Whatever the drain did not finalize (timed-out runs, the hot-restart // skip path) still has a local-only tail when the in-flight run-log // mirror is enabled; upload those tails now so an orderly restart @@ -2049,10 +2093,8 @@ async function startServerWithDatabaseTeardown( // setup-token login session must stop and release its sandbox lease before // the database and the provider stop, so an orderly shutdown never leaves a // sandbox lease or confidential login state alive past the process exit. - // The HTTP listener closes first, while every service is still up, so a - // request in flight is drained against a working server and none reaches - // a route once the pool is gone; the programmatic close below then finds - // the listener already closed and skips. + // HTTP admission and request work were drained above before stopping workers. + // Keep the finalizer's defensive listener close for partial teardown paths. await finalizeServerShutdown({ signal, shutdownAppServices: appShutdown, @@ -2083,6 +2125,10 @@ async function startServerWithDatabaseTeardown( if (exitProcess) process.exit(0); }; + let shutdownPromise: Promise | undefined; + const shutdown = (signal: "SIGINT" | "SIGTERM", exitProcess: boolean) => + shutdownPromise ??= performShutdown(signal, exitProcess); + process.once("SIGINT", () => { void shutdown("SIGINT", true); }); diff --git a/server/src/maintenance/agent-files.ts b/server/src/maintenance/agent-files.ts new file mode 100644 index 00000000000..1ac6b4993c4 --- /dev/null +++ b/server/src/maintenance/agent-files.ts @@ -0,0 +1,41 @@ +import { closeRegisteredClients, createDb } from "@paperclipai/db"; +import { loadStorageConfig } from "../storage/config.js"; +import { createStorageProviderFromConfig } from "../storage/provider-registry.js"; +import { createStorageService } from "../storage/service.js"; +import { collectAgentFileObjects, migrateAgentFiles } from "../services/agent-file-maintenance.js"; +import { cloudRuntimeConfigService } from "../services/cloud-runtime-config.js"; +import { setCloudRuntimeConfigSnapshot } from "../services/cloud-runtime-config-snapshot.js"; +import { isCloudManagedInstance } from "../services/cloud-instance.js"; +import { loadConfig } from "../config.js"; +import { assertEphemeralServerConfig, readEphemeralServerDataEligibility } from "../services/ephemeral-server-contract.js"; + +const value = (flag: string) => process.argv.find(arg => arg.startsWith(`${flag}=`))?.slice(flag.length + 1); +const url = process.env.DATABASE_URL; +if (!url) throw new Error("DATABASE_URL must explicitly select the database"); +const db = createDb(url, { maxConnections: 2 }); +try { + if (isCloudManagedInstance()) { + const config = await cloudRuntimeConfigService(db).inspectStored(); + if (!config) throw new Error("Initialize Cloud runtime configuration before agent file maintenance"); + setCloudRuntimeConfigSnapshot(config); + } + const phase = value("--phase") ?? "migration"; + const apply = process.argv.includes("--apply"); + const provider = createStorageProviderFromConfig(loadStorageConfig()); + if (provider.id !== "s3") throw new Error("Agent file maintenance requires shared object storage"); + if (phase === "eligibility") { + assertEphemeralServerConfig(loadConfig(), { ...process.env, PAPERCLIP_EPHEMERAL_SERVER: "true" }); + const result = await readEphemeralServerDataEligibility(db); + console.log(JSON.stringify({ phase, ...result })); + if (!result.eligible) process.exitCode = 1; + } else { + const result = phase === "migration" ? await migrateAgentFiles(db, { + apply, writersStopped: process.argv.includes("--writers-stopped"), companyId: value("--company"), + after: value("--after"), limit: Number(value("--limit") ?? 25), + }) : phase === "objects" ? await collectAgentFileObjects(db, provider, createStorageService(provider), { + apply, companyId: value("--company") ?? "", agentId: value("--agent") ?? "", continuationToken: value("--after"), + }) : (() => { throw new Error("--phase must be migration, eligibility or objects"); })(); + console.log(JSON.stringify({ phase, apply, ...result })); + if (phase === "migration" && apply && "pageReady" in result && !result.pageReady) process.exitCode = 1; + } +} finally { await closeRegisteredClients(url, { scope: "database" }); } diff --git a/server/src/middleware/idle-admission.ts b/server/src/middleware/idle-admission.ts index aed6be2a1f5..cbcf5cb7275 100644 --- a/server/src/middleware/idle-admission.ts +++ b/server/src/middleware/idle-admission.ts @@ -2,14 +2,41 @@ import { markIdleIngressTracked } from "../services/idle-local-work.js"; import type { Application, Request, RequestHandler } from "express"; import { beginIdleTrackedWork, isIdleTaskDrainActive } from "../services/task-admission.js"; -type RequestWork = { application: Application; pending: number; ended: boolean; closed: boolean; asyncRoute: boolean; routeSequence: number; active: boolean; finish: () => void }; +type RequestWork = { application: Application; pending: number; ended: boolean; closed: boolean; responseFinished: boolean; asyncRoute: boolean; routeSequence: number; active: boolean; finish: () => void }; const requests = new WeakMap(); const trackedApplications = new WeakSet(); +const httpWork = new WeakMap; waiters: Set<() => void> }>(); +function applicationHttpWork(app: Application) { + let state = httpWork.get(app); + if (!state) { state = { requests: new Set(), waiters: new Set() }; httpWork.set(app, state); } + return state; +} function settle(work: RequestWork) { if (work.active && (work.ended || (work.closed && work.asyncRoute)) && work.pending === 0) { work.active = false; work.finish(); } + if (!work.active && (work.responseFinished || work.closed)) { + const state = applicationHttpWork(work.application); + state.requests.delete(work); + if (state.requests.size === 0) for (const resolve of [...state.waiters]) resolve(); + } +} + +/** Wait for admitted HTTP handlers and response streams, including mutations + * still committing after client disconnect. Upgraded sockets have their own + * owner lifecycle and must not hold HTTP teardown open. */ +export async function drainHttpRequestWork(app: Application, signal: AbortSignal): Promise { + signal.throwIfAborted(); + const state = applicationHttpWork(app); + if (state.requests.size === 0) return; + await new Promise((resolve, reject) => { + const cleanup = () => { state.waiters.delete(done); signal.removeEventListener("abort", aborted); }; + const done = () => { cleanup(); resolve(); }; + const aborted = () => { cleanup(); reject(signal.reason); }; + state.waiters.add(done); + signal.addEventListener("abort", aborted, { once: true }); + }); } export function completeIdleReadOnlyResponse(req: Request, application: Application): void { @@ -35,7 +62,7 @@ function isControlRequest(req: Request) { export const idleAdmissionMiddleware: RequestHandler = (req, res, next) => { // This exact route is synchronous and has no auth, SQL, or accepted work. // Probing it must not prevent an otherwise quiescent instance from draining. - if (req.path.replace(/\/$/, "") === "/api/liveness" && ["GET", "HEAD"].includes(req.method)) { + if (["/api/liveness", "/api/ready"].includes(req.path.replace(/\/$/, "")) && ["GET", "HEAD"].includes(req.method)) { next(); return; } @@ -48,9 +75,11 @@ export const idleAdmissionMiddleware: RequestHandler = (req, res, next) => { next(); return; } - const work: RequestWork = { application: req.app, pending: 0, ended: false, closed: false, asyncRoute: false, routeSequence: 0, active: true, finish: beginIdleTrackedWork() }; + const work: RequestWork = { application: req.app, pending: 0, ended: false, closed: false, responseFinished: false, asyncRoute: false, routeSequence: 0, active: true, finish: beginIdleTrackedWork() }; requests.set(req, work); + applicationHttpWork(req.app).requests.add(work); res.once("close", () => { work.closed = true; settle(work); }); + res.once("finish", () => { work.responseFinished = true; settle(work); }); // close/aborted are NOT completion: an async handler can still be committing // a mutation after its client leaves. Wait for the application's end and all // returned handler promises. Callback-only handlers abandoned without end @@ -82,7 +111,7 @@ export function trackIdleRequestHandlers(app: Application): void { const invoke = (req: Request, args: unknown[]) => { const work = requests.get(req); if (!work) return original(...args); - if (!work.active) { work.active = true; work.finish = beginIdleTrackedWork(); } + if (!work.active) { work.active = true; work.finish = beginIdleTrackedWork(); applicationHttpWork(work.application).requests.add(work); } work.pending++; const routeSequence = route ? ++work.routeSequence : undefined; if (route) work.asyncRoute = false; diff --git a/server/src/realtime/live-event-delivery.ts b/server/src/realtime/live-event-delivery.ts index 2df3a70dfa1..3d7f4371779 100644 --- a/server/src/realtime/live-event-delivery.ts +++ b/server/src/realtime/live-event-delivery.ts @@ -29,6 +29,7 @@ export function createLiveEventDelivery(input: { let draining = false; let disposed = false; let needsResync = false; + let resyncReason = "delivery_overflow"; let notifiedRevision = -1; let congestionTimer: ReturnType | undefined; let pendingReadTimer: ReturnType | undefined; @@ -57,14 +58,15 @@ export function createLiveEventDelivery(input: { } function resync() { if (!needsResync || !isOpen() || notifiedRevision === revision) return; - if (control({ type: "live.resync", version: 1, revision, reason: "delivery_overflow" })) notifiedRevision = revision; + if (control({ type: "live.resync", version: 1, revision, reason: resyncReason })) notifiedRevision = revision; } - function overflow() { + function overflow(reason = "delivery_overflow") { if (!negotiated) { socket.close(1013, "event backlog"); return; } epoch += 1; for (const item of queued) { retainedBytes -= item.bytes; retainedItems -= 1; } queued = []; needsResync = true; + resyncReason = reason; if (!congestionTimer) { congestionTimer = setTimeout(() => { congestionTimer = undefined; @@ -162,6 +164,7 @@ export function createLiveEventDelivery(input: { queued.push({ event, bytes, epoch }); retainedBytes += bytes; retainedItems += 1; void drain(); }, + requestResync: () => overflow("controller_change"), dispose, metrics: () => ({ retainedBytes, retainedItems, negotiated, revision, needsResync }), }; diff --git a/server/src/realtime/live-events-ws.ts b/server/src/realtime/live-events-ws.ts index 06bffa0ae68..260ac29adaf 100644 --- a/server/src/realtime/live-events-ws.ts +++ b/server/src/realtime/live-events-ws.ts @@ -31,6 +31,7 @@ interface WsSocket { interface WsServer { clients: Set; + close(): void; on(event: "connection", listener: (socket: WsSocket, req: IncomingMessage) => void): void; on(event: "close", listener: () => void): void; handleUpgrade( @@ -248,6 +249,7 @@ export function setupLiveEventsWebSocketServer( db: Db, opts: { deploymentMode: DeploymentMode; + subscribeInvalidation?: (companyId: string, resync: () => void) => () => void; resolveSessionFromHeaders?: (headers: Headers) => Promise; /** * Resolves a Cloud-proxied browser's identity from the trusted @@ -375,11 +377,12 @@ export function setupLiveEventsWebSocketServer( onError: err => logger.warn({ err, companyId: context.companyId }, "live event authorization failed"), }); const unsubscribe = subscribeCompanyLiveEvents(context.companyId, event => delivery.enqueue(event)); + const unsubscribeInvalidation = opts.subscribeInvalidation?.(context.companyId, delivery.requestResync); socket.on("message", (data, isBinary) => { if (isBinary) { socket.close(1003, "text messages required"); return; } delivery.receive(data.toString()); }); - cleanupByClient.set(socket, () => { unsubscribe(); delivery.dispose(); }); + cleanupByClient.set(socket, () => { unsubscribe(); unsubscribeInvalidation?.(); delivery.dispose(); }); aliveByClient.set(socket, true); socket.on("pong", () => { diff --git a/server/src/realtime/runner-prp-ws.ts b/server/src/realtime/runner-prp-ws.ts index 5ac629e89a1..ad5f9b4a9e6 100644 --- a/server/src/realtime/runner-prp-ws.ts +++ b/server/src/realtime/runner-prp-ws.ts @@ -242,6 +242,7 @@ export class RunnerPrpRuntimeRequestResolutionError extends Error { * the same request fails closed instead of answering the provider twice. */ export function queueRunnerPrpRuntimeRequestResolution(input: { + readonly commandId?: string; readonly companyId: string; readonly runId: string; readonly pendingRequest: PendingNativeRuntimeRequest; @@ -299,7 +300,7 @@ export function queueRunnerPrpRuntimeRequestResolution(input: { resolution: input.resolution, resolutionActor: input.actor, }, - undefined, + input.commandId, true, ); registration.runtimeRequestResolutions.set(pending.requestId, { diff --git a/server/src/retained-ui-assets.ts b/server/src/retained-ui-assets.ts new file mode 100644 index 00000000000..89c618a62ad --- /dev/null +++ b/server/src/retained-ui-assets.ts @@ -0,0 +1,77 @@ +import { createHash } from "node:crypto"; +import { createReadStream } from "node:fs"; +import fs from "node:fs/promises"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import type { Readable } from "node:stream"; +import type { RequestHandler } from "express"; +import type { StorageProvider } from "./storage/types.js"; + +const prefix = "ui-assets/v1/"; +function assetPath(value: string): string | null { + const relative = value.replace(/^\//, ""); + return relative.length <= 512 && relative.split("/").every(part => /^[a-zA-Z0-9_.-]+$/.test(part) && part !== "." && part !== "..") + && !relative.endsWith(".map") ? relative : null; +} +async function digest(stream: Readable) { + const hash = createHash("sha256"); + for await (const chunk of stream) hash.update(chunk); + return hash.digest("hex"); +} + +/** Public, immutable build assets only. Publishing before readiness lets an + * older tab fetch its chunks after the last old container has disappeared. + * Objects have no automatic expiry: retain them at least 30 days after the + * last serving deployment, including rollback, before operator reclamation. */ +export function retainedUiAssets(provider: StorageProvider): { publish(directory: string): Promise; serve: RequestHandler } { + if (provider.id !== "s3" || !provider.putObjectConditional) throw new Error("Retained UI assets require conditional shared object storage"); + async function publish(directory: string) { + async function visit(relativeDirectory: string) { + for (const entry of await fs.readdir(path.join(directory, relativeDirectory), { withFileTypes: true })) { + const relative = path.posix.join(relativeDirectory, entry.name); + if (entry.isDirectory()) { await visit(relative); continue; } + if (relative.endsWith(".map")) continue; + if (!entry.isFile() || !assetPath(relative)) throw new Error("Unsupported static build asset"); + const filename = path.join(directory, relative); + const signal = AbortSignal.timeout(30_000); + const contentLength = (await fs.stat(filename)).size; + const expected = await digest(createReadStream(filename, { signal })); + const objectKey = `${prefix}${relative}`; + const body = createReadStream(filename, { signal }); + try { + await provider.putObjectConditional!({ objectKey, body, contentLength, contentType: "application/octet-stream", ifNoneMatch: "*", signal }); + } catch (error) { + // An existing key or uncertain upload may be accepted only if its + // complete bytes match. Never overwrite a colliding build asset. + const existing = await provider.getObject({ objectKey, signal }).catch(() => { throw error; }); + if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes"); + } finally { body.destroy(); } + } + } + await visit(""); + } + const serve: RequestHandler = async (req, res, next) => { + if (req.method !== "GET" && req.method !== "HEAD") { next(); return; } + let relative: string | null; + try { relative = assetPath(decodeURIComponent(req.path)); } + catch { relative = null; } + if (!relative) { res.status(404).end(); return; } + const controller = new AbortController(); + const signal = AbortSignal.any([controller.signal, AbortSignal.timeout(30_000)]); + const cancel = () => controller.abort(); + res.once("close", cancel); + try { + const object = await provider.getObject({ objectKey: `${prefix}${relative}`, signal }); + res.type(relative).set("Cache-Control", "public, max-age=31536000, immutable"); + if (object.etag) res.set("ETag", object.etag); + if (object.contentLength !== undefined) res.set("Content-Length", String(object.contentLength)); + if (req.method === "HEAD") { object.stream.destroy(); res.end(); return; } + await pipeline(object.stream, res, { signal }); + } catch (error) { + if (controller.signal.aborted) return; + if ((error as { status?: number }).status === 404) { res.status(404).end(); return; } + next(error); + } finally { res.off("close", cancel); } + }; + return { publish, serve }; +} diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index a4c6d75010a..4afd6910b08 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -1,4 +1,5 @@ import { readTaskPanelActivations } from "../services/plugin-task-panels.js"; +import { enqueueNativeRuntimeResponse } from "../services/native-runtime/native-runtime-response-delivery.js"; import { latestFailedRunsByAgent } from "@paperclipai/shared"; import { agentConfigurationSpec } from "@paperclipai/shared"; import { agentConfigurationRevision, restoreAgentConfigurationRedactions } from "../lib/agent-configuration-revision.js"; @@ -7319,7 +7320,10 @@ export function agentRoutes( throw conflict("This runtime request is stale or is no longer pending."); } let queued: { commandId: string }; - try { + if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true") { + queued = await enqueueNativeRuntimeResponse(db, { request: currentPendingRequest, + actor: resolutionActor, localImplicit: req.actor.source === "local_implicit", resolution }); + } else try { queued = await resolveNativeRuntimeRequest({ runId, requestId, diff --git a/server/src/routes/openapi.ts b/server/src/routes/openapi.ts index 927e29bf3dc..6b7e02f026e 100644 --- a/server/src/routes/openapi.ts +++ b/server/src/routes/openapi.ts @@ -1374,6 +1374,8 @@ const PUBLIC_OPERATIONS = new Set([ "GET /api/health", "GET /api/liveness", "HEAD /api/liveness", + "GET /api/ready", + "HEAD /api/ready", "GET /api/openapi.json", "GET /api/board-claim/{token}", "POST /api/cli-auth/challenges", @@ -1823,6 +1825,25 @@ registry.registerPath({ responses: { 200: { description: "Process startup recovery is complete" }, 503: { description: "Startup recovery is not complete" } }, }); +registry.registerPath({ + method: "get", + path: "/api/ready", + tags: ["health"], + summary: "Deployment readiness without database access", + description: "Public in-memory admission status after startup dependencies and recovery have been verified. Returns 503 immediately during shutdown. Responses are not cached; an unavailable server supplies Retry-After: 1.", + responses: { + 200: r.ok(z.object({ status: z.literal("ready") }).strict()), + 503: { description: "Starting, recovering, or draining", content: { + "application/json": { schema: z.object({ status: z.enum(["starting", "recovering", "draining"]) }).strict() }, + } }, + }, +}); +registry.registerPath({ + method: "head", path: "/api/ready", tags: ["health"], + summary: "Deployment readiness without a response body", + responses: { 200: { description: "Ready to receive traffic" }, 503: { description: "Starting, recovering, or draining" } }, +}); + // Shared by the healthy and database-unreachable responses: full details // (including serverInfo) ride only on board/agent-actor responses. const healthServerInfoSchema = z diff --git a/server/src/routes/ready.ts b/server/src/routes/ready.ts new file mode 100644 index 00000000000..6e647b827f7 --- /dev/null +++ b/server/src/routes/ready.ts @@ -0,0 +1,18 @@ +import { Router, type RequestHandler } from "express"; +import { getStartupRecoveryState } from "../startup-recovery-state.js"; + +/** Deployment admission, distinct from process liveness and diagnostic health. + * Bootstrap has already verified its dependencies before publishing ready. This + * endpoint neither authenticates nor opens SQL connections on an idle server. */ +export function readinessRoutes() { + const router = Router(); + const reply: RequestHandler = (_req, res) => { + const { phase } = getStartupRecoveryState(); + res.set("Cache-Control", "no-store"); + if (phase !== "ready") res.set("Retry-After", "1"); + res.status(phase === "ready" ? 200 : 503).json({ status: phase }); + }; + router.head("/", reply); + router.get("/", reply); + return router; +} diff --git a/server/src/services/agent-configuration.ts b/server/src/services/agent-configuration.ts index 4f2e30c737b..c46abea410c 100644 --- a/server/src/services/agent-configuration.ts +++ b/server/src/services/agent-configuration.ts @@ -1,3 +1,4 @@ +import { assertEphemeralAgentInstructions } from "./agent-instructions.js"; import { notifyStrandedAgentStatus } from "./live-run-recovery-notifications.js"; import { notifyDeferredExecutionEligibility } from "./deferred-execution-notifications.js"; import { notifyRunDispatch } from "./run-dispatch-notifications.js"; @@ -97,6 +98,7 @@ export async function prepareAgentConfiguration(db: Db, existing: AgentConfigura if (normalizedPatch.adapterConfig !== undefined) { await notifySkillVersionPinChange(db, existing.companyId, existing.adapterConfig ?? {}, normalizedPatch.adapterConfig); } + assertEphemeralAgentInstructions({ ...existing, ...normalizedPatch }); return normalizedPatch; } diff --git a/server/src/services/agent-file-durability.ts b/server/src/services/agent-file-durability.ts new file mode 100644 index 00000000000..54d13c1d7fa --- /dev/null +++ b/server/src/services/agent-file-durability.ts @@ -0,0 +1,184 @@ +import { createHash, randomUUID } from "node:crypto"; +import fs from "node:fs/promises"; +import { constants } from "node:fs"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import { Transform } from "node:stream"; +import { and, eq } from "drizzle-orm"; +import { agentFileRoots, type Db, type DurableAgentFileEntry } from "@paperclipai/db"; +import { HttpError } from "../errors.js"; +import { getStorageService } from "../storage/index.js"; +import { assertInstructionPathSafe, instructionPath } from "./agent-instruction-files.js"; +import { cachedAgentFileManifest } from "./agent-file-checkpoints.js"; +import { AgentFileLimitError, MAX_AGENT_DIRECTORY_BYTES, MAX_AGENT_DIRECTORY_ENTRIES, MAX_AGENT_FILE_BYTES } from "./agent-file-store.js"; + +type Tx = Parameters[0]>[0]; +type Owner = { companyId: string; id: string; createdAt: Date }; +type Head = typeof agentFileRoots.$inferSelect; +const where = (owner: Owner) => and(eq(agentFileRoots.companyId, owner.companyId), eq(agentFileRoots.agentId, owner.id)); +const marker = (root: string) => path.join(path.dirname(root), "file-sync", "durable-generation"); +const adoptionReceipt = (root: string) => path.join(path.dirname(root), "file-sync", "durable-adoption.json"); +export const durableAgentFilesEnabled = () => process.env.PAPERCLIP_DURABLE_AGENT_FILES === "true"; + +/** A newly created agent has no legacy files to import. Commit its empty + * authority with the agent row, before any API can acknowledge file mutations. */ +export async function initializeDurableAgentFiles(tx: Db, owner: { companyId: string; id: string }) { + if (!durableAgentFilesEnabled()) return; + storage(); + await tx.insert(agentFileRoots).values({ companyId: owner.companyId, agentId: owner.id, + generation: randomUUID(), entries: [] }).onConflictDoNothing(); +} + +function storage() { + const service = getStorageService(); + if (service.provider !== "s3") throw new HttpError(503, "Durable agent files require shared object storage"); + return service; +} + +/** Call before changing a projection: a failed write must not leave a valid cache marker. */ +export async function invalidateAgentFileProjection(root: string) { + await fs.rm(marker(root), { force: true }); +} +async function mark(root: string, generation: string) { + await atomicPrivateWrite(marker(root), generation); +} +async function atomicPrivateWrite(filename: string, contents: string) { + await fs.mkdir(path.dirname(filename), { recursive: true, mode: 0o700 }); + const temporary = `${filename}.${randomUUID()}`; + try { + await fs.writeFile(temporary, contents, { mode: 0o600 }); + await fs.rename(temporary, filename); + } finally { await fs.rm(temporary, { force: true }); } +} + +/** Under the agent row lock. A durable head remains authoritative even after + * the opt-in is removed; silently falling back would lose acknowledged writes. */ +export async function restoreDurableAgentFiles(tx: Tx, owner: Owner, root: string): Promise { + let [head] = await tx.select().from(agentFileRoots).where(where(owner)); + if (!head) { + const filename = await assertInstructionPathSafe(path.dirname(adoptionReceipt(root)), path.basename(adoptionReceipt(root))); + const contents = await fs.readFile(filename, "utf8").catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (contents !== null) { + const receipt = JSON.parse(contents) as { version: number; agentCreatedAt: string; head: Head }; + if (receipt.version !== 1 || receipt.head.companyId !== owner.companyId || receipt.head.agentId !== owner.id + || receipt.agentCreatedAt !== owner.createdAt.toISOString() || !Array.isArray(receipt.head.entries)) { + throw new Error("Agent file adoption receipt does not match its owner"); + } + // A first save can roll back the head after changing local files. Replay + // only the pre-mutation manifest whose immutable bytes were uploaded, + // never adopt the now-dirty directory as fresh legacy input. + [head] = await tx.insert(agentFileRoots).values({ companyId: owner.companyId, agentId: owner.id, + generation: receipt.head.generation, entries: receipt.head.entries }).returning(); + await invalidateAgentFileProjection(root); + } + } + if (!head) return false; + const service = storage(); + const generation = await fs.readFile(marker(root), "utf8").catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; + throw error; + }); + if (generation === head.generation && await fs.stat(root).then(s => s.isDirectory()).catch(() => false)) return true; + const temporary = path.join(path.dirname(root), `.agent-files-${randomUUID()}`); + await fs.mkdir(temporary, { recursive: true, mode: 0o700 }); + const signal = AbortSignal.timeout(30_000); + try { + // Complete and verify the replacement before discarding an older projection. + for (const entry of head.entries) { + signal.throwIfAborted(); + const relative = instructionPath(entry.path); + if (entry.kind === "dir") { + await assertInstructionPathSafe(temporary, `${relative}/.path-check`); + await fs.mkdir(path.join(temporary, relative), { recursive: true, mode: 0o700 }); continue; + } + const filename = await assertInstructionPathSafe(temporary, relative); + await fs.mkdir(path.dirname(filename), { recursive: true, mode: 0o700 }); + const handle = await fs.open(filename, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL, entry.mode & 0o777); + try { + const hash = createHash("sha256"); + let size = 0; + if (entry.objectKey !== null) { + const object = await service.getObject(owner.companyId, entry.objectKey, { signal }); + await pipeline(object.stream, new Transform({ transform(chunk: Buffer, _encoding, callback) { + size += chunk.length; + if (size > entry.size) return callback(new Error("Agent file exceeds its committed size")); + hash.update(chunk); callback(null, chunk); + } }), handle.createWriteStream(), { signal }); + } + if (size !== entry.size || hash.digest("hex") !== entry.sha256) throw new Error("Agent file does not match its committed digest"); + } finally { await handle.close(); } + } + await invalidateAgentFileProjection(root); + await fs.rm(root, { recursive: true, force: true }); + await fs.rename(temporary, root); + await mark(root, head.generation); + return true; + } finally { await fs.rm(temporary, { recursive: true, force: true }); } +} + +/** Publish immutable bytes before the database head. A failed/uncertain database + * commit never removes uploaded bytes that may have become authoritative. */ +export async function persistDurableAgentFiles(tx: Tx, owner: Owner, root: string, importingLegacy = false) { + const [previous] = await tx.select().from(agentFileRoots).where(where(owner)); + if (!previous && !durableAgentFilesEnabled()) return; + const service = storage(); + const manifest = await cachedAgentFileManifest(root); + if (!importingLegacy) { + if (manifest.entries.length > MAX_AGENT_DIRECTORY_ENTRIES) throw new AgentFileLimitError("Agent directory exceeds the 100,000-entry limit (files and folders)"); + let total = 0; + for (const [relative, entry] of manifest.entries) if (entry.kind === "file") { + if (entry.size! > MAX_AGENT_FILE_BYTES) throw new AgentFileLimitError(`Agent file "${relative}" exceeds the 256 MiB per-file limit`); + total += entry.size!; + } + if (total > MAX_AGENT_DIRECTORY_BYTES) throw new AgentFileLimitError("Agent directory exceeds the 2 GiB total storage limit"); + } + const before = new Map(previous?.entries.map(entry => [entry.path, entry])); + const entries: DurableAgentFileEntry[] = []; + const signal = AbortSignal.timeout(30_000); + for (const [relative, entry] of manifest.entries) { + signal.throwIfAborted(); + if (entry.kind === "dir") { entries.push({ path: relative, kind: "dir" }); continue; } + if (entry.kind !== "file") throw new Error("Agent files cannot contain symbolic links"); + if (entry.size === undefined || entry.mode === undefined || !entry.hash) throw new Error("Agent file metadata is incomplete"); + const old = before.get(relative); + let objectKey = old?.kind === "file" && old.sha256 === entry.hash && old.size === entry.size ? old.objectKey : null; + if (!objectKey && entry.size !== 0) { + const filename = await assertInstructionPathSafe(root, relative); + const handle = await fs.open(filename, constants.O_RDONLY | constants.O_NOFOLLOW); + const source = handle.createReadStream(); + const hash = createHash("sha256"); + let size = 0; + const body = new Transform({ transform(chunk: Buffer, _encoding, callback) { + size += chunk.length; + if (size > entry.size!) return callback(new Error("Agent file changed during upload")); + hash.update(chunk); callback(null, chunk); + }, flush(callback) { + callback(size === entry.size && hash.digest("hex") === entry.hash ? null : new Error("Agent file changed during upload")); + } }); + try { + const [result] = await Promise.all([service.putFile({ companyId: owner.companyId, namespace: `agent-files/${owner.id}`, + originalFilename: null, contentType: "application/octet-stream", body, + byteSize: entry.size, sha256: entry.hash, signal }), pipeline(source, body, { signal })]); + objectKey = result.objectKey; + } finally { source.destroy(); body.destroy(); await handle.close(); } + } + entries.push({ path: relative, kind: "file", mode: entry.mode, size: entry.size, sha256: entry.hash, objectKey }); + } + if (previous && JSON.stringify(previous.entries) === JSON.stringify(entries)) { + await mark(root, previous.generation); + return; + } + const generation = randomUUID(); + if (!previous) { + // This is a retry receipt for the one-time local import, not serving + // authority. Once committed, Postgres and S3 suffice without this file. + await atomicPrivateWrite(adoptionReceipt(root), JSON.stringify({ version: 1, agentCreatedAt: owner.createdAt.toISOString(), + head: { companyId: owner.companyId, agentId: owner.id, generation, entries } })); + } + await tx.insert(agentFileRoots).values({ companyId: owner.companyId, agentId: owner.id, generation, entries }) + .onConflictDoUpdate({ target: [agentFileRoots.companyId, agentFileRoots.agentId], set: { generation, entries, updatedAt: new Date() } }); + await mark(root, generation); +} diff --git a/server/src/services/agent-file-maintenance.ts b/server/src/services/agent-file-maintenance.ts new file mode 100644 index 00000000000..bafd76e7847 --- /dev/null +++ b/server/src/services/agent-file-maintenance.ts @@ -0,0 +1,98 @@ +import fs from "node:fs/promises"; +import path from "node:path"; +import { and, asc, eq, gt, inArray } from "drizzle-orm"; +import { agentFileRoots, agents, fileStorageDb, heartbeatRuns, type Db } from "@paperclipai/db"; +import type { StorageProvider, StorageService } from "../storage/types.js"; +import { adoptAgentFiles, inspectAgentDirectory } from "./agent-file-store.js"; +import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js"; +import { durableAgentFilesEnabled } from "./agent-file-durability.js"; + +export const AGENT_FILE_OBJECT_GRACE_MS = 30 * 24 * 60 * 60_000; + +/** One-time operator migration on the host that owns the volume. Cloud must + * stop ingress and execution writers before apply; row locks also serialize + * ordinary file mutations. A dry run never adopts or rewrites local files. */ +export async function migrateAgentFiles(db: Db, input: { + apply: boolean; writersStopped: boolean; companyId?: string; after?: string; limit?: number; +}) { + if (input.apply && (!input.writersStopped || !durableAgentFilesEnabled())) { + throw new Error("Agent file migration requires stopped writers and PAPERCLIP_DURABLE_AGENT_FILES=true"); + } + const limit = input.limit ?? 25; + if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) throw new Error("Migration limit must be between 1 and 100"); + const rows = await db.select({ id: agents.id }).from(agents).where(and( + input.companyId ? eq(agents.companyId, input.companyId) : undefined, + input.after ? gt(agents.id, input.after) : undefined, + )).orderBy(asc(agents.id)).limit(limit + 1); + const results: Array<{ agentId: string; state: string }> = []; + for (const candidate of rows.slice(0, limit)) { + const state = await fileStorageDb(db).transaction(async tx => { + const [agent] = await tx.select().from(agents).where(eq(agents.id, candidate.id)).for("update"); + if (!agent) return "removed"; + if (agentInstructionsBundleMode(agent) === "external") return "external_instructions"; + const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id), + inArray(heartbeatRuns.status, ["running", "queued"]), + )).limit(1); + if (run) return "active_execution"; + const [head] = await tx.select({ generation: agentFileRoots.generation }).from(agentFileRoots) + .where(and(eq(agentFileRoots.companyId, agent.companyId), eq(agentFileRoots.agentId, agent.id))); + if (input.apply) { + await adoptAgentFiles(tx, agent); + return "durable"; + } + if (head) return "durable"; + const root = resolveManagedInstructionsRoot(agent); + const stored = agent.adapterConfig.instructionsRootPath; + if (typeof stored === "string" && (path.resolve(stored) !== root + || !await fs.stat(path.join(root, deriveBundleState(agent).entryFile)).then(s => s.isFile()).catch(() => false))) { + return "legacy_files_missing"; + } + if (await fs.stat(root).then(s => s.isDirectory()).catch(() => false)) await inspectAgentDirectory(root, false); + return "needs_import"; + }); + results.push({ agentId: candidate.id, state }); + } + return { results, next: rows.length > limit ? results.at(-1)!.agentId : null, + pageReady: results.every(row => row.state === "durable" || row.state === "removed") }; +} + +/** Bounded, explicit maintenance, with no idle polling. Keep every object for + * an unadopted agent: a rolled-back first save may have a local import receipt. + * The agent row lock prevents a concurrent publisher from reusing an object + * between the current-head check and deletion. Deleted agents have no writer. */ +export async function collectAgentFileObjects(db: Db, provider: Pick, storage: StorageService, input: { + companyId: string; agentId: string; apply: boolean; continuationToken?: string; now?: number; +}) { + if (provider.id !== "s3" || storage.provider !== "s3" || !provider.listObjects) throw new Error("Shared object storage is required"); + const uuid = /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i; + if (!uuid.test(input.companyId) || !uuid.test(input.agentId)) throw new Error("Company and agent UUIDs are required"); + const prefix = `${input.companyId}/agent-files/${input.agentId}/`; + const signal = AbortSignal.timeout(30_000); + const page = await provider.listObjects({ prefix, continuationToken: input.continuationToken, signal }); + let retained = 0, eligible = 0, deleted = 0; + for (const key of page.keys) { + signal.throwIfAborted(); + if (!key.startsWith(prefix) || key.includes("..") || key.includes("\\")) throw new Error("Agent file object scope is invalid"); + await fileStorageDb(db).transaction(async tx => { + const [agent] = await tx.select({ id: agents.id }).from(agents) + .where(and(eq(agents.companyId, input.companyId), eq(agents.id, input.agentId))).for("update"); + const [head] = await tx.select({ entries: agentFileRoots.entries }).from(agentFileRoots) + .where(and(eq(agentFileRoots.companyId, input.companyId), eq(agentFileRoots.agentId, input.agentId))); + if ((agent && !head) || head?.entries.some(entry => entry.kind === "file" && entry.objectKey === key)) { + retained++; return; + } + const object = await storage.headObject(input.companyId, key, { signal }); + if (!object.exists || !object.lastModified || object.lastModified.getTime() + AGENT_FILE_OBJECT_GRACE_MS > (input.now ?? Date.now())) { + retained++; return; + } + eligible++; + if (input.apply) { + await storage.deleteObject(input.companyId, key, { signal, allVersions: true }); + if ((await storage.headObject(input.companyId, key, { signal })).exists) throw new Error("Agent file object remains after deletion"); + deleted++; + } + }); + } + return { retained, eligible, deleted, next: page.continuationToken ?? null }; +} diff --git a/server/src/services/agent-file-store.ts b/server/src/services/agent-file-store.ts index 4f31bf98d35..1cd4a63b1e6 100644 --- a/server/src/services/agent-file-store.ts +++ b/server/src/services/agent-file-store.ts @@ -5,13 +5,14 @@ import { constants } from "node:fs"; import { Readable } from "node:stream"; import path from "node:path"; import { and, desc, eq, isNull, sql } from "drizzle-orm"; -import { agents, activityLog, agentInstructionHeads, agentInstructionRevisions, resourceLifecycleEvents, type Db } from "@paperclipai/db"; +import { agents, activityLog, agentInstructionHeads, agentInstructionRevisions, resourceLifecycleEvents, fileStorageDb, type Db } from "@paperclipai/db"; import { captureDirectorySnapshot, mergeDirectoryWithBaseline, type DirectorySnapshot } from "@paperclipai/adapter-utils/workspace-restore-merge"; import { HttpError, conflict, notFound, unprocessable } from "../errors.js"; import { authorizeInstructionCommit, authorizeInstructionRead } from "./agent-instruction-authorization.js"; import { assertInstructionPathSafe, instructionPath, instructionBytes, materializeInstructionBytes, readInstructionBytes, MAX_INSTRUCTION_BYTES } from "./agent-instruction-files.js"; import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js"; import type { AuthorizationActor } from "./authorization.js"; +import { durableAgentFilesEnabled, invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles } from "./agent-file-durability.js"; type Tx = Parameters[0]>[0]; type Agent = typeof agents.$inferSelect; @@ -60,11 +61,17 @@ export function agentFileTokenFromHash(h: string): string { return `${h.slice(0, 8)}-${h.slice(8, 12)}-8${h.slice(13, 16)}-a${h.slice(17, 20)}-${h.slice(20, 32)}`; } -/** Call under the agent row lock. Import deployed revision heads once, then the - * directory is authoritative. The adoption marker lives OUTSIDE agent files. */ +/** Call under the agent row lock. Restore shared authority first; legacy local + * adoption is allowed only on the server that actually owns the old directory. */ export async function adoptAgentFiles(tx: Tx, agent: Agent): Promise { if (agentInstructionsBundleMode(agent) === "external") throw unprocessable("External instructions must be migrated to managed storage first"); const root = resolveManagedInstructionsRoot(agent); + if (await restoreDurableAgentFiles(tx, agent, root)) return root; + const storedRoot = agent.adapterConfig.instructionsRootPath; + if (durableAgentFilesEnabled() && typeof storedRoot === "string" + && (path.resolve(storedRoot) !== root || !await readInstructionBytes(root, deriveBundleState(agent).entryFile))) { + throw new HttpError(503, "Migrate managed agent files on their current server before replacing its filesystem", { code: "AGENT_FILES_MIGRATION_REQUIRED" }); + } const markerRoot = path.join(path.dirname(root), "file-sync"); const marker = "adopted.json"; await assertInstructionPathSafe(root, deriveBundleState(agent).entryFile); @@ -76,6 +83,7 @@ export async function adoptAgentFiles(tx: Tx, agent: Agent): Promise { }); if (adopted !== null) { if (JSON.parse(adopted).schema !== AGENT_FILES_CONTRACT) throw new Error("Unsupported agent file contract"); + await persistDurableAgentFiles(tx, agent, root, true); return root; } const legacy = await tx.select({ entryFile: agentInstructionHeads.entryFile, contentBase64: agentInstructionRevisions.contentBase64 }) @@ -83,6 +91,7 @@ export async function adoptAgentFiles(tx: Tx, agent: Agent): Promise { .where(and(eq(agentInstructionHeads.companyId, agent.companyId), eq(agentInstructionHeads.agentId, agent.id))); for (const entry of legacy) await materializeInstructionBytes(root, entry.entryFile, Buffer.from(entry.contentBase64, "base64")); await materializeInstructionBytes(markerRoot, marker, Buffer.from(JSON.stringify({ schema: AGENT_FILES_CONTRACT }))); + await persistDurableAgentFiles(tx, agent, root, true); return root; } @@ -178,6 +187,7 @@ export async function isAgentFileSaveRecorded(tx: Tx, agent: Agent, relative: st } export function agentFileStore(db: Db) { + db = fileStorageDb(db); async function locked(companyId: string, agentId: string, actor: AuthorizationActor, write: boolean, fn: (tx: Tx, agent: Agent, root: string, bound: AuthorizationActor) => Promise) { return db.transaction(async tx => { @@ -185,7 +195,10 @@ export function agentFileStore(db: Db) { if (!agent) throw notFound("Agent not found"); const bound = write ? await authorizeInstructionCommit(tx, actor, agent) : await authorizeInstructionRead(tx, actor, agent); const root = await adoptAgentFiles(tx, agent); - return fn(tx, agent, root, bound); + if (write) await invalidateAgentFileProjection(root); + const result = await fn(tx, agent, root, bound); + if (write) await persistDurableAgentFiles(tx, agent, root); + return result; }); } async function audit(tx: Tx, agent: Agent, actor: AuthorizationActor, details: Record) { diff --git a/server/src/services/agent-hiring.ts b/server/src/services/agent-hiring.ts index 0c238907bbc..16253ac025a 100644 --- a/server/src/services/agent-hiring.ts +++ b/server/src/services/agent-hiring.ts @@ -1,3 +1,5 @@ +import { assertEphemeralAgentInstructions } from "./agent-instructions.js"; +import { initializeDurableAgentFiles } from "./agent-file-durability.js"; import { notifyAttentionRetentionWork } from "./retention-work-notifications.js"; import { notifySkillVersionRetention, notifySkillVersionPinChange } from "./skill-version-retention-notifications.js"; import { agentAppearanceSchema, randomAgentAppearance } from "@paperclipai/shared"; @@ -65,6 +67,8 @@ export async function prepareAgentHire(db: Db, companyId: string, input: CreateA export async function initializeAgentHire(txDb: Db, created: AgentHireRecord, options?: CreateAgentOptions) { const tx = txDb; const companyId = created.companyId; + assertEphemeralAgentInstructions({ ...created, name: "" }); + await initializeDurableAgentFiles(txDb, created); await agentIdentityService(txDb).ensureAgentIdentity(companyId, created.id); // New standard agents receive the standard direct grants at activation. // Low-trust and bundled agents keep their explicit, narrower grants. @@ -133,6 +137,7 @@ export async function prepareAgentHireApproval(txDb: Db, existing: AgentHireReco }); } if (patch.adapterConfig !== undefined) await notifySkillVersionPinChange(txDb, existing.companyId, existing.adapterConfig ?? {}, patch.adapterConfig); + assertEphemeralAgentInstructions({ ...existing, ...patch, name: "" }); return patch; } diff --git a/server/src/services/agent-instruction-revisions.ts b/server/src/services/agent-instruction-revisions.ts index f7f3619d3f2..cd60264463e 100644 --- a/server/src/services/agent-instruction-revisions.ts +++ b/server/src/services/agent-instruction-revisions.ts @@ -1,6 +1,7 @@ import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js"; import { createHash } from "node:crypto"; import { adoptAgentFiles, agentFileToken, isAgentFileSaveRecorded } from "./agent-file-store.js"; +import { invalidateAgentFileProjection, persistDurableAgentFiles } from "./agent-file-durability.js"; import { and, desc, eq, getTableColumns, lt, or, sql } from "drizzle-orm"; import { activityLog, @@ -9,6 +10,7 @@ import { pluginManagedResources, agentInstructionHeads as heads, agentInstructionRevisions as revisions, + fileStorageDb, type Db, } from "@paperclipai/db"; import type { @@ -81,6 +83,7 @@ function headOwner(target: InstructionTarget, entryFile: string) { } export function agentInstructionRevisionService(db: Db) { + db = fileStorageDb(db); async function lockTarget( tx: Tx, target: InstructionTarget, @@ -292,7 +295,9 @@ export function agentInstructionRevisionService(db: Db) { }, ); } + await invalidateAgentFileProjection(state.root); await materializeInstructionBytes(state.root, state.entryFile, candidate); + await persistDurableAgentFiles(tx, state.agent, state.root); const row = currentRow(input, input.entryFile, candidate, bound); await recordSave(row); await bindEntry(); diff --git a/server/src/services/agent-instruction-working-copies.ts b/server/src/services/agent-instruction-working-copies.ts index 6cb0aebb3d4..50e61014072 100644 --- a/server/src/services/agent-instruction-working-copies.ts +++ b/server/src/services/agent-instruction-working-copies.ts @@ -195,7 +195,7 @@ export function agentInstructionWorkingCopyService(db: Db, options: { environmen }) : await revisions.readCurrent({ companyId: input.companyId, agentId: input.agentId }, bound); if (!baseline) return null; - const exported = await agentInstructionsService().exportFiles(agent, { rejectSymlinks: true }); + const exported = await agentInstructionsService(db).exportFiles(agent, { rejectSymlinks: true }); const entryFile = instructionPath(baseline?.revision.entryFile ?? exported.entryFile); const content = instructionBytes(baseline?.content ?? exported.files[entryFile] ?? ""); const localRoot = existing?.localRoot ?? expectedLocalRoot; diff --git a/server/src/services/agent-instructions.ts b/server/src/services/agent-instructions.ts index 1729e8be838..d86bd0c8d54 100644 --- a/server/src/services/agent-instructions.ts +++ b/server/src/services/agent-instructions.ts @@ -1,8 +1,9 @@ -import { inspectAgentFile, fileHash, agentFilePath, MAX_AGENT_FILE_BYTES } from "./agent-file-store.js"; +import { adoptAgentFiles, inspectAgentFile, fileHash, agentFilePath, MAX_AGENT_FILE_BYTES } from "./agent-file-store.js"; +import { invalidateAgentFileProjection, persistDurableAgentFiles, restoreDurableAgentFiles } from "./agent-file-durability.js"; import fs from "node:fs/promises"; import path from "node:path"; import { and, eq } from "drizzle-orm"; -import { agentInstructionHeads, agents, type Db } from "@paperclipai/db"; +import { agentInstructionHeads, agents, fileStorageDb, type Db } from "@paperclipai/db"; import { instructionPath, assertInstructionPathSafe, instructionBytes, readInstructionBytes } from "./agent-instruction-files.js"; import { notFound, unprocessable } from "../errors.js"; import { resolveHomeAwarePath, resolvePaperclipInstanceRoot } from "../home-paths.js"; @@ -250,7 +251,13 @@ export function deriveBundleState(agent: AgentLike): BundleState { const legacyInstructionsPath = asString(config[FILE_KEY]); let mode: BundleMode | null = isBundleMode(storedModeRaw) ? storedModeRaw : null; - let rootPath = storedRootRaw ? resolveHomeAwarePath(storedRootRaw) : null; + // A managed root belongs to this process's disposable projection. Saved + // absolute paths describe the old host, never a different storage authority. + const managedSuffix = path.join("companies", agent.companyId, "agents", agent.id, "instructions"); + const portableManagedRoot = mode === "managed" && storedRootRaw + && path.isAbsolute(storedRootRaw) && path.normalize(storedRootRaw).endsWith(`${path.sep}${managedSuffix}`); + let rootPath = portableManagedRoot ? resolveManagedInstructionsRoot(agent) + : storedRootRaw ? resolveHomeAwarePath(storedRootRaw) : null; let entryFile = ENTRY_FILE_DEFAULT; const storedEntryRaw = asString(config[ENTRY_KEY]); @@ -295,6 +302,13 @@ export function deriveBundleState(agent: AgentLike): BundleState { }; } +/** Configuration mutations cannot introduce authority on disposable disk. */ +export function assertEphemeralAgentInstructions(agent: AgentLike) { + if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && agentInstructionsBundleMode(agent) === "external") { + throw unprocessable("Volume-free servers require managed agent instructions"); + } +} + /** Classify the configured bundle without touching the filesystem. */ export function agentInstructionsBundleMode(agent: AgentLike): BundleMode | null { const state = deriveBundleState(agent); @@ -334,7 +348,8 @@ async function recoverManagedBundleState(agent: AgentLike, state: BundleState): if (state.mode === "external") return state; - const resolvedConfiguredRoot = path.resolve(state.rootPath); + const configuredRoot = asString(state.config[ROOT_KEY]) ?? state.rootPath; + const resolvedConfiguredRoot = path.resolve(configuredRoot); const configuredRootMatchesManaged = resolvedConfiguredRoot === managedRootPath; const hasEntryMismatch = recoveredEntryFile !== state.entryFile; @@ -345,7 +360,7 @@ async function recoverManagedBundleState(agent: AgentLike, state: BundleState): const warnings = [...state.warnings]; if (!configuredRootMatchesManaged) { warnings.push( - `Recovered managed instructions from disk at ${managedRootPath}; ignoring stale configured root ${state.rootPath}.`, + `Recovered managed instructions from disk at ${managedRootPath}; ignoring stale configured root ${configuredRoot}.`, ); } if (hasEntryMismatch) { @@ -426,7 +441,8 @@ function buildPersistedBundleConfig( options?: { clearLegacyPromptTemplate?: boolean }, ): Record { const currentRootPath = current.rootPath ? path.resolve(current.rootPath) : null; - const derivedRootPath = derived.rootPath ? path.resolve(derived.rootPath) : null; + const storedRoot = asString(derived.config[ROOT_KEY]) ?? derived.rootPath; + const derivedRootPath = storedRoot ? path.resolve(storedRoot) : null; const configMatchesRecoveredState = derived.mode === current.mode && derivedRootPath !== null @@ -489,6 +505,7 @@ export function syncInstructionsBundleConfigFromFilePath( } export function agentInstructionsService(db?: Db) { + if (db) db = fileStorageDb(db); async function assertUnversionedEntry(agent: AgentLike, entryFile: string, connection: Db | Parameters[0]>[0] | undefined = db) { if (!connection) throw unprocessable("Bundle initialization requires the database-backed instructions service"); const [head] = await connection.select().from(agentInstructionHeads).where(and(eq(agentInstructionHeads.companyId, agent.companyId), @@ -597,6 +614,9 @@ export function agentInstructionsService(db?: Db) { ): Promise<{ bundle: AgentInstructionsBundle; adapterConfig: Record }> { const state = await recoverManagedBundleState(agent, deriveBundleState(agent)); const nextMode = input.mode ?? state.mode ?? "managed"; + if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && nextMode === "external") { + throw unprocessable("Volume-free servers require managed agent instructions"); + } const nextEntryFile = input.entryFile ? normalizeRelativeFilePath(input.entryFile) : state.entryFile; let nextRootPath: string; @@ -717,7 +737,12 @@ export function agentInstructionsService(db?: Db) { // Historical entries also remain projections if the configured entry changes. await assertUnversionedEntry(current, normalizeRelativeFilePath(relativePath), tx); } - return operation(current); + const managed = agentInstructionsBundleMode(current) !== "external"; + const root = managed ? await adoptAgentFiles(tx, current) : null; + if (root) await invalidateAgentFileProjection(root); + const result = await operation(current); + if (root) await persistDurableAgentFiles(tx, current, root); + return result; }); } async function writeFile(...args: Parameters) { @@ -810,7 +835,7 @@ export function agentInstructionsService(db?: Db) { if (!db) throw unprocessable("Bundle initialization requires the database-backed instructions service"); return db.transaction(async (tx) => { const agent = args[0]; - const [owner] = await tx.select({ id: agents.id }).from(agents).where(and(eq(agents.id, agent.id), eq(agents.companyId, agent.companyId))).for("update"); + const [owner] = await tx.select().from(agents).where(and(eq(agents.id, agent.id), eq(agents.companyId, agent.companyId))).for("update"); if (!owner) throw notFound("Agent not found"); await assertUnversionedEntry(agent, args[2]?.entryFile ? normalizeRelativeFilePath(args[2].entryFile) : ENTRY_FILE_DEFAULT, tx); if (args[2]?.replaceExisting) { @@ -818,19 +843,42 @@ export function agentInstructionsService(db?: Db) { .where(and(eq(agentInstructionHeads.companyId, agent.companyId), eq(agentInstructionHeads.agentId, agent.id))).limit(1); if (existingHead) throw unprocessable("Replacing this bundle would remove a versioned entry. Use canonical content commits.", { code: "INSTRUCTION_REVISION_REQUIRED" }); } - return materializeManagedBundleUnversioned(...args); + const root = resolveManagedInstructionsRoot(owner); + if (agentInstructionsBundleMode(owner) === "external") await restoreDurableAgentFiles(tx, owner, root); + else await adoptAgentFiles(tx, owner); + await invalidateAgentFileProjection(root); + const result = await materializeManagedBundleUnversioned(...args); + await persistDurableAgentFiles(tx, owner, root); + return result; + }); + } + + // Public managed reads and configuration changes share the same row lock as + // editor/runtime writes. Internal helpers operate on the already-loaded tree. + async function withProjection(agent: AgentLike, write: boolean, operation: (current: AgentLike) => Promise, managedDestination = false) { + if (!db) return operation(agent); + return db.transaction(async tx => { + const [current] = await tx.select().from(agents).where(and(eq(agents.id, agent.id), eq(agents.companyId, agent.companyId))).for("update"); + if (!current) throw notFound("Agent not found"); + const managed = agentInstructionsBundleMode(current) !== "external"; + const root = managed ? await adoptAgentFiles(tx, current) : managedDestination ? resolveManagedInstructionsRoot(current) : null; + if (!managed && root) await restoreDurableAgentFiles(tx, current, root); + if (write && root) await invalidateAgentFileProjection(root); + const result = await operation(current); + if (write && root) await persistDurableAgentFiles(tx, current, root); + return result; }); } return { - getBundleLocation, - getBundle, - readFile, - updateBundle, + getBundleLocation: (agent: AgentLike) => withProjection(agent, false, getBundleLocation), + getBundle: (agent: AgentLike) => withProjection(agent, false, getBundle), + readFile: (agent: AgentLike, relativePath: string) => withProjection(agent, false, current => readFile(current, relativePath)), + updateBundle: (agent: AgentLike, input: Parameters[1]) => withProjection(agent, true, current => updateBundle(current, input), input.mode === "managed"), writeFile, deleteFile, - exportFiles, - ensureManagedBundle: ensureWritableBundle, + exportFiles: (agent: AgentLike, options?: Parameters[1]) => withProjection(agent, false, current => exportFiles(current, options)), + ensureManagedBundle: (agent: AgentLike, options?: Parameters[1]) => withProjection(agent, true, current => ensureWritableBundle(current, options)), materializeManagedBundle, }; } diff --git a/server/src/services/built-in-agents.ts b/server/src/services/built-in-agents.ts index c248c555580..ecd6a1dd111 100644 --- a/server/src/services/built-in-agents.ts +++ b/server/src/services/built-in-agents.ts @@ -7,12 +7,13 @@ import { fileURLToPath } from "node:url"; import { readPaperclipSkillSyncPreference, writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils"; import { and, desc, eq, ne } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { activityLog, agents, builtInManagedResources, companies, issueThreadInteractions, issues, routines, routineTriggers } from "@paperclipai/db"; +import { activityLog, agents, builtInManagedResources, companies, issueThreadInteractions, issues, routines, routineTriggers, fileStorageDb } from "@paperclipai/db"; import { agentHarnessType, paperclipRunnerProfileForHarness, agentRunner, syncRoutineVariablesWithTemplate, type AgentRunnerChoice } from "@paperclipai/shared"; import type { Agent, Approval, CompanySkill, PermissionKey, Routine, RoutineTrigger, RoutineVariable } from "@paperclipai/shared"; import { conflict, HttpError, notFound, unprocessable } from "../errors.js"; import { logActivity } from "./activity-log.js"; import { adoptAgentFiles, agentFilePath, fileHash, readAgentFile, snapshotAgentFiles } from "./agent-file-store.js"; +import { invalidateAgentFileProjection, persistDurableAgentFiles } from "./agent-file-durability.js"; import { assertInstructionPathSafe, instructionBytes, materializeInstructionBytes } from "./agent-instruction-files.js"; import { agentInstructionRevisionService } from "./agent-instruction-revisions.js"; import type { AuthorizationActor } from "./authorization.js"; @@ -994,7 +995,7 @@ export function builtInAgentService(db: Db) { // Persist only the pending operation's baseline hashes before touching // files. If a write or DB commit fails, the next reconciliation can accept // already-applied bytes and retry. This is not instruction revision history. - const prepared = await db.transaction(async (tx) => { + const prepared = await fileStorageDb(db).transaction(async (tx) => { const [current] = await tx.select().from(agents).where(and(eq(agents.companyId, agent.companyId), eq(agents.id, agent.id))).for("update"); if (!current) throw notFound("Built-in agent not found"); if (agentInstructionsBundleMode(current) !== "managed") return false; @@ -1024,7 +1025,7 @@ export function builtInAgentService(db: Db) { } else { const previousFiles: Record = {}; for (const file of previousPaths) { - try { previousFiles[file] = (await instructionsSvc.readFile(current, file)).content; } + try { previousFiles[file] = (await readAgentFile(root, agentFilePath(file)))?.toString("utf8") ?? null; } catch { previousFiles[file] = null; } } if (stockHash(previousFiles) !== latest.stockHash) return false; @@ -1044,7 +1045,7 @@ export function builtInAgentService(db: Db) { return true; }); if (!prepared) return currentState; - return db.transaction(async (tx) => { + return fileStorageDb(db).transaction(async (tx) => { const [current] = await tx.select().from(agents).where(and(eq(agents.companyId, agent.companyId), eq(agents.id, agent.id))).for("update"); if (!current) throw notFound("Built-in agent not found"); if (agentInstructionsBundleMode(current) !== "managed") return currentState; @@ -1061,10 +1062,12 @@ export function builtInAgentService(db: Db) { const nextHash = incoming[file] ? fileHash(incoming[file]) : null; if (hash !== baseHash && hash !== nextHash) return currentState; } + await invalidateAgentFileProjection(root); for (const [file, bytes] of Object.entries(incoming)) await materializeInstructionBytes(root, file, bytes); for (const file of Object.keys(pending.baseHashes)) { if (!(file in incoming)) await fs.rm(await assertInstructionPathSafe(root, file), { force: true }); } + await persistDurableAgentFiles(tx, current, root); await upsertManagedResourceBinding({ companyId: agent.companyId, bundleKey: definition.key, resourceKind: "instructions", resourceKey: "AGENTS.md", resourceId: agent.id, stockVersion: bundle.stockVersion, stockHash: stock, diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.ts index 184caa27c75..3e1f1b95b2b 100644 --- a/server/src/services/delivery-work-coordinator.ts +++ b/server/src/services/delivery-work-coordinator.ts @@ -127,11 +127,17 @@ export function createDeliveryWorkCoordinator(input: { const ready = Promise.resolve().then(() => { start(); return running; }); return { ready, ...worker }; }, - async stop() { + // Stop dispatch queues first, keeping only already-admitted controller + // commands alive until their execution owners finish the handoff. A later + // stop() closes the retained workers and the shared scheduler as well. + async stop(keepQueues?: ReadonlySet) { stopped = true; - scheduler.stop(); - await Promise.all(Array.from(workers.values(), worker => worker.stop())); - workers.clear(); + if (!keepQueues?.size) scheduler.stop(); + await Promise.all(Array.from(workers, async ([queue, worker]) => { + if (keepQueues?.has(queue)) return; + await worker.stop(); + workers.delete(queue); + })); }, }; } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 15219849e82..7a87dd728dc 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -2,6 +2,10 @@ import { signalDatabaseWork, subscribeDatabaseWork } from "@paperclipai/db/work- /** Topics for existing durable queues; register intent before their writes. */ export const DELIVERY_QUEUES = { + nativeRuntimeResponses: "native-runtime-responses", + nativeCancellation: "native-cancellation", + deploymentNotifications: "deployment-notifications", + liveInvalidations: "live-invalidations", nativeProcessSessions: "native-process-sessions", nativeQuestionCancellations: "native-question-cancellations", activeExecutionRecovery: "active-execution-recovery", diff --git a/server/src/services/deployment-notifications.ts b/server/src/services/deployment-notifications.ts new file mode 100644 index 00000000000..a53a5d918bf --- /dev/null +++ b/server/src/services/deployment-notifications.ts @@ -0,0 +1,100 @@ +import { createDb, type Db } from "@paperclipai/db"; +import { DATABASE_WORK_CHANNEL, enableDatabaseWorkNotifications, publishDatabaseWorkHint, + receiveDatabaseWorkNotification, reconcileDatabaseWorkNotifications, subscribeDatabaseWork } from "@paperclipai/db/work-signals"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { activeNativeExecutionRunIds, subscribeActiveNativeExecutions } from "./native-runtime/native-active-runs.js"; +import { isIdleTaskDrainActive, subscribeTaskDrainChanges } from "./task-admission.js"; +import { subscribeAllCompanyLiveEvents } from "./live-events.js"; + +/** Temporary overlap, not steady-state horizontal scaling. Listen while startup + * and its 120-second handoff window are active, or while this process owns a + * native execution. Browser connections alone never keep PostgreSQL awake. + * Durable queues and their existing claims remain execution authority. */ +export function deploymentNotifications(db: Db, coordinator: ReturnType, input: { + directDatabaseUrl: string; isWarmStandby?: () => boolean; handoffMs?: number; +}) { + enableDatabaseWorkNotifications(db); + let stopped = false, handoffEndsAt: number | null = null; + let client: ReturnType["$client"] | null = null; + const wanted = () => !stopped && !isIdleTaskDrainActive() + && (handoffEndsAt === null || handoffEndsAt > Date.now() || activeNativeExecutionRunIds.size > 0); + async function close() { + const previous = client; client = null; + await previous?.end({ timeout: 1 }); + } + const listener = coordinator.register(DELIVERY_QUEUES.deploymentNotifications, { + retryMs: 1000, runDuringDrain: true, + async run(signal) { + if (!wanted()) { await close(); return; } + if (client) return; + const opened = createDb(input.directDatabaseUrl, { maxConnections: 1, connectTimeoutSeconds: 5, + applicationName: "paperclip-deployment-notifications" }).$client; + client = opened; + const abort = () => { void opened.end({ timeout: 0 }); }; + signal.addEventListener("abort", abort, { once: true }); + try { + await opened.listen(DATABASE_WORK_CHANNEL, payload => receiveDatabaseWorkNotification(db, payload), + () => reconcileDatabaseWorkNotifications(db)); + } catch (error) { if (client === opened) await close(); throw error; } + finally { signal.removeEventListener("abort", abort); } + if (!wanted()) await close(); + }, + hasPending: async () => false, + nextRunAt: () => handoffEndsAt !== null && handoffEndsAt > Date.now() ? handoffEndsAt : null, + }); + const stopActive = subscribeActiveNativeExecutions(listener.wake); + const stopDrain = subscribeTaskDrainChanges(listener.wake); + + // Only company identity crosses the hint channel, never event contents or + // authorization decisions. A receiving tab refreshes through its normal API. + const companies = new Set(); + let flushAt: number | null = null; + const publisher = coordinator.register(DELIVERY_QUEUES.liveInvalidations, { + retryMs: 1000, runDuringDrain: true, + async run() { + if (flushAt === null || Date.now() < flushAt) return; + // Remove before awaiting: a same-company event arriving during the SQL + // write belongs to the next batch and must not be cleared by this one. + flushAt = null; + for (const companyId of [...companies]) { + companies.delete(companyId); + try { await publishDatabaseWorkHint(db, `live:${companyId}`); } + catch (error) { + companies.add(companyId); + flushAt ??= Date.now() + 1000; + throw error; + } + } + }, + hasPending: async () => false, + nextRunAt: () => flushAt, + }); + const stopEvents = subscribeAllCompanyLiveEvents(event => { + if (stopped || !/^[a-f0-9-]{36}$/i.test(event.companyId)) return; + companies.add(event.companyId); + flushAt ??= Date.now() + 1000; + publisher.wake(); + }); + return { + async ready() { + await listener.ready; + // Coordinator startup errors normally retry. A replacement must not + // advertise readiness until the cross-controller channel is connected. + if (!client && !input.isWarmStandby?.()) throw new Error("Deployment notification listener did not become ready"); + }, + markReady() { + handoffEndsAt = Date.now() + (input.handoffMs ?? 120_000); + listener.wake(); + }, + subscribeCompany(companyId: string, resync: () => void) { + return subscribeDatabaseWork(db, `live:${companyId}`, event => { if (event === "settled") resync(); }); + }, + async stop() { + stopped = true; + stopActive(); stopDrain(); stopEvents(); + await close(); + await Promise.all([listener.stop(), publisher.stop()]); + }, + }; +} diff --git a/server/src/services/ephemeral-server-contract.ts b/server/src/services/ephemeral-server-contract.ts new file mode 100644 index 00000000000..f210837f954 --- /dev/null +++ b/server/src/services/ephemeral-server-contract.ts @@ -0,0 +1,76 @@ +import { and, eq, isNotNull, isNull, sql } from "drizzle-orm"; +import { agentFileRoots, agents, executionWorkspaces, heartbeatRuns, providerTraceRecords, type Db } from "@paperclipai/db"; +import type { Config } from "../config.js"; +import { isValidLocalEncryptedMasterKey } from "../secrets/local-encrypted-provider.js"; +import { agentInstructionsBundleMode } from "./agent-instructions.js"; +import { skillInstallationMaintenance } from "./skill-installation-maintenance.js"; + +export const ephemeralServerEnabled = () => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true"; + +/** Validate before consumers can generate a replacement local identity. Error + * messages name settings only, never credential values or connection URLs. */ +export function assertEphemeralServerConfig(config: Pick, env = process.env): string | null { + if (env.PAPERCLIP_EPHEMERAL_SERVER !== "true") return null; + const refuse = (reason: string): never => { throw new Error(`Volume-free server configuration: ${reason}`); }; + if (config.storageProvider !== "s3" || env.PAPERCLIP_DURABLE_AGENT_FILES !== "true" || env.PAPERCLIP_SHARED_UI_ASSETS !== "true") { + refuse("shared S3 storage, durable agent files and shared UI assets are required"); + } + if (!env.PAPERCLIP_SECRETS_MASTER_KEY || !isValidLocalEncryptedMasterKey(env.PAPERCLIP_SECRETS_MASTER_KEY)) { + refuse("a stable PAPERCLIP_SECRETS_MASTER_KEY is required"); + } + if ((env.PAPERCLIP_DECISION_SIGNING_SECRET?.trim().length ?? 0) < 32) { + refuse("a stable PAPERCLIP_DECISION_SIGNING_SECRET of at least 32 characters is required; preserve the existing decision key during migration"); + } + let runtime: URL, listener: URL; + try { runtime = new URL(config.databaseUrl ?? ""); listener = new URL(env.PAPERCLIP_DATABASE_LISTEN_URL ?? ""); } + catch { return refuse("external DATABASE_URL and PAPERCLIP_DATABASE_LISTEN_URL are required"); } + if (![runtime, listener].every(url => ["postgres:", "postgresql:"].includes(url.protocol)) + || runtime.username !== listener.username || runtime.password !== listener.password || runtime.pathname !== listener.pathname + || runtime.hostname.replace(/-pooler(?=\.)/, "") !== listener.hostname + || /pooler|pgbouncer/i.test(listener.hostname)) { + refuse("PAPERCLIP_DATABASE_LISTEN_URL must be a direct connection to the same database using the runtime role"); + } + return listener.href; +} + +/** A readiness gate, not a periodic scan or an implicit migration. Existing + * volume data is adopted by the owning maintenance commands before removal. */ +export async function assertEphemeralServerData(db: Db) { + if (!ephemeralServerEnabled()) return; + const { blockers } = await readEphemeralServerDataEligibility(db); + if (blockers.length) throw new Error(`Volume-free server data is not eligible: ${blockers.join(", ")}`); +} + +export async function readEphemeralServerDataEligibility(db: Db) { + const blockers: string[] = []; + const [missingFiles] = await db.select({ id: agents.id }).from(agents).leftJoin(agentFileRoots, + and(eq(agentFileRoots.companyId, agents.companyId), eq(agentFileRoots.agentId, agents.id))) + .where(isNull(agentFileRoots.agentId)).limit(1); + if (missingFiles) blockers.push("agent_files_require_migration"); + // Only inspect configuration here; filesystem reads must never adopt an + // empty directory on the replacement machine as the old server's content. + const configured = await db.select({ id: agents.id, companyId: agents.companyId, name: agents.name, adapterConfig: agents.adapterConfig }).from(agents); + if (configured.some(agent => agentInstructionsBundleMode(agent) === "external")) blockers.push("external_agent_instructions"); + const [localWorkspace] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and( + isNotNull(executionWorkspaces.cwd), sql`${executionWorkspaces.providerType} <> 'plugin'`, + sql`coalesce(${executionWorkspaces.metadata}->'remoteWorkspace'->>'schema', '') <> 'paperclip.remote-workspace.v1'`, + )).limit(1); + if (localWorkspace) blockers.push("workspaces_require_migration"); + const [localLog] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.logStore, "local_file"), isNotNull(heartbeatRuns.logRef), + )).limit(1); + if (localLog) blockers.push("run_logs_require_migration"); + const [localTrace] = await db.select({ id: providerTraceRecords.id }).from(providerTraceRecords).where(and( + isNull(providerTraceRecords.deletedAt), sql`${providerTraceRecords.traceRef} not like 's3-segments-v1:%'`, + )).limit(1); + if (localTrace) blockers.push("provider_traces_require_migration"); + const skills = skillInstallationMaintenance(db); + let after: string | undefined; + for (;;) { + const page = await skills.eligibility({ after, limit: 50 }); + if (page.results.some(skill => skill.status !== "version_backed")) { blockers.push("skills_require_migration"); break; } + if (page.inspected < 50 || !page.after) break; + after = page.after; + } + return { eligible: blockers.length === 0, blockers }; +} diff --git a/server/src/services/heartbeat/recovery.test.ts b/server/src/services/heartbeat/recovery.test.ts index a92ceff5823..1573f3bdd49 100644 --- a/server/src/services/heartbeat/recovery.test.ts +++ b/server/src/services/heartbeat/recovery.test.ts @@ -1086,6 +1086,29 @@ describePostgres("heartbeat recovery module ownership", () => { } }); + it("notifies a replacement after committing the departing controller's released lease", async () => { + const run = await seedRun(), deps = callbacks(db); + const identity = await currentNativeControllerIdentity(); + await db.update(agents).set({ adapterType: "paperclip_runner" }).where(eq(agents.id, run.agentId)); + await db.update(heartbeatRuns).set({ runtimeMode: "native", nativeIssueId: run.issueId }).where(eq(heartbeatRuns.id, run.id)); + await db.insert(nativeRunFinalizations).values({ runId: run.id, companyId: run.companyId, issueId: run.issueId!, + phase: "observed", controllerBootId: identity.bootId, leaseOwner: "departing-server", leaseExpiresAt: new Date(Date.now() + 60_000) }); + const wake = vi.fn(), unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.nativeProcessSessions, wake); + try { + const before = new Date(); + await createHeartbeatRecovery(db, deps).drainRunningRunsForShutdown("SIGTERM", before, [run.id]); + expect(native.detach).toHaveBeenCalledWith([run.id]); + expect(wake).toHaveBeenCalled(); + const [saved] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)); + expect(saved.leaseExpiresAt).toEqual(before); + expect(saved.recoveryState).toBe("awaiting_runner_reattach"); + } finally { + unsubscribe(); + await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, run.id)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)); + } + }); + it("does not detach a different server's native run during shutdown", async () => { const run = await seedRun(); const deps = callbacks(db); diff --git a/server/src/services/heartbeat/recovery.ts b/server/src/services/heartbeat/recovery.ts index b1f23bcf40a..bc74f44a984 100644 --- a/server/src/services/heartbeat/recovery.ts +++ b/server/src/services/heartbeat/recovery.ts @@ -1,4 +1,5 @@ import { pendingFailedNativeRetryCancellationCondition } from "../native-runtime/native-cancellation-request.js"; +import { notifyLiveRunRecovery } from "../live-run-recovery-notifications.js"; import { registerLiveRecoveryWork } from "./live-recovery-work.js"; import { NATIVE_RETAINED_OWNER_WAIT } from "../native-runtime/native-controller-recovery.js"; import { nativeEnvironmentSettlementDisposition, nativeEnvironmentSettlementWork } from "../native-runtime/native-environment-settlement-work.js"; @@ -1159,9 +1160,9 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD processPid: run.processPid, processStartedAt: run.processStartedAt?.toISOString() ?? null, }); - await db - .update(nativeRunFinalizations) - .set({ + await db.transaction(async tx => { + await notifyLiveRunRecovery(tx, { companyId: run.companyId, runId: run.id }); + await tx.update(nativeRunFinalizations).set({ leaseExpiresAt: now, recoveryState: "awaiting_runner_reattach", recoveryHistory: sql`( @@ -1188,6 +1189,7 @@ export function createHeartbeatRecovery(db: Db, dependencies: HeartbeatRecoveryD isNull(nativeRunFinalizations.resultId), ), ); + }); await appendRunEvent(run, { eventType: "native.recovery.transition", stream: "system", diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index c640732ea77..7b777887f85 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { agents, companies, completionContracts, createDb, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; -import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; import { CHAT_CONTROL_RECOVERY_ADMISSION_KEY } from "../chat-control-recovery-stop.js"; import { buildNativeRuntimeContext } from "../native-runtime/runtime-context.js"; @@ -40,6 +40,7 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = await db.execute(sql`set client_min_messages = warning`); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); afterAll(async () => { await db?.$client.end(); await database?.cleanup(); if (instructionRoot) await rm(instructionRoot, { recursive: true, force: true }); }); + afterEach(() => vi.unstubAllEnvs()); beforeEach(async () => { await db.execute(sql`truncate table companies restart identity cascade`); vi.clearAllMocks(); @@ -327,6 +328,24 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = expect(await coordinators(run.id)).toHaveLength(0); }); + it.each(["warm", "per_turn"] as const)("checks resolved %s policy for ephemeral admission", async mode => { + const { input, run } = await fixture(); + vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", "true"); + // Deliberately disagree with the resolved provider policy in both cases. + input.agent.adapterConfig = { ...input.agent.adapterConfig, ...(mode === "per_turn" ? { lifecycleMode: "warm" } : {}) }; + input.workspace.executionTarget = { ...sandbox, reusableLeaseConfigured: true, + effectiveCapabilities: { reusableLeases: true } as never, + runnerLifecyclePolicy: mode === "warm" ? { mode, idleTimeoutMs: 300_000 } : { mode, idleTimeoutMs: null } }; + if (mode === "per_turn") { + await expect(selectHeartbeatRuntime(db, input)).rejects.toThrow("warm Paperclip Runner"); + expect(await coordinators(run.id)).toHaveLength(0); + } else { + const selected = await selectHeartbeatRuntime(db, input); + expect(selected.selected && selected.nativeExecution?.session.lifecyclePolicy.mode).toBe("warm"); + expect((await read(run.id)).runnerProfileJson).toMatchObject({ nativeExecutionInput: { session: { lifecyclePolicy: { mode: "warm" } } } }); + } + }); + it.each([ { provider: "github" as const, method: "api_key" as const, expected: "warm" }, { provider: "github" as const, method: "subscription" as const, expected: "per_turn" }, diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 28ac44205c2..4f10a467302 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -1,5 +1,5 @@ import { museRunnerBroker } from "../muse-runner-broker.js"; -import { readRemoteWorkspaceAuthority } from "../remote-workspaces.js"; +import { assertEphemeralNativeLifecycle, readRemoteWorkspaceAuthority } from "../remote-workspaces.js"; import { measureRunTransaction } from "../execution-transaction-timing.js"; import { measureRunPhase } from "../execution-preparation-timing.js"; import { @@ -515,6 +515,9 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele (["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend) || managedAiRuntime && !supportsManagedWarmSession ? { mode: "per_turn" as const, idleTimeoutMs: null } : environmentLifecyclePolicy ?? agentLifecyclePolicy); + if (!["openai_dot_mcp", "grok_bot_mcp", "muse_external"].includes(nativeRuntimeResolution.profile.backend)) { + assertEphemeralNativeLifecycle(effectiveLifecyclePolicy.mode); + } if ( effectiveLifecyclePolicy.mode === "warm" && executionTarget?.kind === "remote" && diff --git a/server/src/services/heartbeat/workspace-preparation.ts b/server/src/services/heartbeat/workspace-preparation.ts index 9f3ddac65cc..15c91735f71 100644 --- a/server/src/services/heartbeat/workspace-preparation.ts +++ b/server/src/services/heartbeat/workspace-preparation.ts @@ -1,4 +1,4 @@ -import { readRemoteWorkspaceAuthority, reserveRemoteWorkspace, shouldUseRemoteWorkspaces } from "../remote-workspaces.js"; +import { assertEphemeralWorkspaceExecution, readRemoteWorkspaceAuthority, reserveRemoteWorkspace, shouldUseRemoteWorkspaces } from "../remote-workspaces.js"; import { readFreshWorkspaceReadiness } from "../fresh-workspace-continuation.js"; import { assertNativeSessionPreparationAuthority } from "../native-runtime/native-execution-lease.js"; import { resolveRemoteWorkspaceSource } from "./remote-workspace-source.js"; @@ -237,10 +237,13 @@ async function prepareHeartbeatWorkspaceLocked(db: Db, input: HeartbeatWorkspace let remoteAuthority = !isDotRun && shouldUseRemoteWorkspaces({ adapterType: agent.adapterType, environmentDriver: selectedEnvironmentForConfig?.driver ?? null, - experimentalEnabled: resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled && Boolean(issueId && issueRef) + experimentalEnabled: (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" || resolvedInstanceSettings.experimental.remoteWorkspaceStorageEnabled) && Boolean(issueId && issueRef) && !projectExecutionWorkspacePolicy?.storageProviderPluginId, existingWorkspace: reusableExistingExecutionWorkspace ?? existingExecutionWorkspace, }); + assertEphemeralWorkspaceExecution({ externalConversation: isDotRun, native: run.runtimeMode === "native", + adapterType: agent.adapterType, + pluginTaskWorkspace, remoteAuthority }); const assertRemotePreparationOwned = run.runtimeMode === "native" ? () => assertNativeSessionPreparationAuthority({ db, companyId: agent.companyId, runId: run.id, leaseOwner: runOptions.nativeLeaseOwner ?? runOptions.nativeRestartRecovery?.leaseOwner, diff --git a/server/src/services/native-runtime/native-active-runs.ts b/server/src/services/native-runtime/native-active-runs.ts index b1b6bc2c47a..6339224d977 100644 --- a/server/src/services/native-runtime/native-active-runs.ts +++ b/server/src/services/native-runtime/native-active-runs.ts @@ -1,2 +1,12 @@ export const activeNativeExecutionRunIds = new Set(); +const listeners = new Set<() => void>(); +export function setNativeExecutionActive(runId: string, active: boolean) { + if (active) activeNativeExecutionRunIds.add(runId); + else activeNativeExecutionRunIds.delete(runId); + for (const listener of listeners) listener(); +} +export function subscribeActiveNativeExecutions(listener: () => void) { + listeners.add(listener); + return () => { listeners.delete(listener); }; +} export const NATIVE_PREPARATION_FENCING_VERSION = 1; diff --git a/server/src/services/native-runtime/native-cancellation-delivery.ts b/server/src/services/native-runtime/native-cancellation-delivery.ts new file mode 100644 index 00000000000..636a694c696 --- /dev/null +++ b/server/src/services/native-runtime/native-cancellation-delivery.ts @@ -0,0 +1,88 @@ +import { and, eq, gt, inArray, sql } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { subscribeDatabaseWork } from "@paperclipai/db/work-signals"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { activeNativeExecutionRunIds, subscribeActiveNativeExecutions } from "./native-active-runs.js"; +import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; + +const object = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +export const notifyNativeCancellation = (tx: object) => notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeCancellation); + +export async function nativeCancellationHasRemoteOwner(db: Db, runId: string): Promise { + if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true") return false; + const identity = await currentNativeControllerIdentity(); + const [owner] = await db.select({ bootId: nativeRunFinalizations.controllerBootId }).from(nativeRunFinalizations) + .where(and(eq(nativeRunFinalizations.runId, runId), gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`))).limit(1); + return !!owner?.bootId && owner.bootId !== identity.bootId; +} + +/** The accepting HTTP server waits for the real owner's audited ACK. Subscribe + * before reading so a commit between admission and subscription cannot be lost. + * The existing intent remains recoverable after timeout or process replacement. */ +export function waitForNativeCancellation(db: Db, input: { companyId: string; runId: string; intentId: string }, timeoutMs = 20_000) { + return new Promise<{ dispatched: true; decisionId: string | null; auditId: string }>((resolve, reject) => { + let settled = false, reading = false, dirty = false; + const finish = (error?: unknown, result?: { dispatched: true; decisionId: string | null; auditId: string }) => { + if (settled) return; + settled = true; clearTimeout(timeout); unsubscribe(); + if (error) reject(error); else resolve(result!); + }; + const check = async () => { + dirty = true; + if (reading || settled) return; + reading = true; + try { + do { + dirty = false; + const [run] = await db.select({ result: heartbeatRuns.resultJson }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId))).limit(1); + const intent = object(run?.result?.nativeCancellation); + if (intent.intentId !== input.intentId) throw new Error("native_cancellation_intent_conflict"); + if (intent.dispatchState === "acknowledged" && intent.dispatched === true && typeof intent.acknowledgementAuditId === "string") { + finish(undefined, { dispatched: true, decisionId: typeof intent.decisionId === "string" ? intent.decisionId : null, + auditId: intent.acknowledgementAuditId }); + } + } while (dirty && !settled); + } catch (error) { finish(error); } + finally { reading = false; } + }; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.nativeCancellation, event => { if (event === "settled") void check(); }); + const timeout = setTimeout(() => finish(new Error("native_cancellation_pending_recovery")), timeoutMs); + void check(); + }); +} + +/** Reconcile only this controller's live executions. Intent lives on the run; + * no idle scan, new command table, or competing execution claim is introduced. */ +export function registerNativeCancellationDelivery(db: Db, coordinator: ReturnType) { + const worker = coordinator.register(DELIVERY_QUEUES.nativeCancellation, { + retryMs: 1000, runDuringDrain: true, + async run(signal) { + const ids = [...activeNativeExecutionRunIds]; + if (!ids.length) return; + const identity = await currentNativeControllerIdentity(); + const rows = await db.select({ id: heartbeatRuns.id, result: heartbeatRuns.resultJson }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and( + inArray(heartbeatRuns.id, ids), eq(nativeRunFinalizations.controllerBootId, identity.bootId), + gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`), + sql`${heartbeatRuns.resultJson}->'nativeCancellation'->>'dispatchState' = 'pending'`, + )); + for (const row of rows) { + signal.throwIfAborted(); + if (!activeNativeExecutionRunIds.has(row.id)) continue; + const intent = object(row.result?.nativeCancellation), startup = object(row.result?.startupCancellation); + if (intent.schema !== "paperclip.native-cancellation.v1" || !["run", "issue", "turn"].includes(String(intent.scope))) continue; + const { cancelNativeSession } = await import("./native-session-executor.js"); + await cancelNativeSession(row.id, typeof intent.reason === "string" ? intent.reason : "Operator requested Stop", { + db, scope: intent.scope as "run" | "issue" | "turn", + replacementAccepted: Array.isArray(intent.effects) && intent.effects.includes("accept_replacement_turn"), + ...(typeof startup.cancellationRequestId === "string" ? { cancellationRequestId: startup.cancellationRequestId } : {}), + }); + } + }, + hasPending: async () => false, + }); + const unsubscribe = subscribeActiveNativeExecutions(worker.wake); + return { ready: worker.ready, async stop() { unsubscribe(); await worker.stop(); } }; +} diff --git a/server/src/services/native-runtime/native-runtime-response-delivery.ts b/server/src/services/native-runtime/native-runtime-response-delivery.ts new file mode 100644 index 00000000000..656acd7cd1c --- /dev/null +++ b/server/src/services/native-runtime/native-runtime-response-delivery.ts @@ -0,0 +1,141 @@ +import { createHash, randomUUID } from "node:crypto"; +import { and, eq, gt, inArray, sql } from "drizzle-orm"; +import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import type { HarnessRuntimeRequestResolution } from "../../vendor/paperclip-runner/index.js"; +import { accessService } from "../access.js"; +import { loadConfig } from "../../config.js"; +import { conflict } from "../../errors.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { activeNativeExecutionRunIds, subscribeActiveNativeExecutions } from "./native-active-runs.js"; +import { currentNativeControllerIdentity } from "./native-restart-recovery.js"; +import { assertNativeRuntimeRequestResolverAuthorized, NativeRuntimeRequestResolutionAuthorizationError, + readPendingNativeRuntimeRequest, type NativeRuntimeRequestResolver, type PendingNativeRuntimeRequest } from "./runtime-request-resolution-authority.js"; + +type ResponseCommand = { + commandId: string; + request: PendingNativeRuntimeRequest; + actor: NativeRuntimeRequestResolver; + localImplicit: boolean; + resolution: HarnessRuntimeRequestResolution; + fingerprint: string; + state: "pending" | "queued" | "obsolete" | "denied"; +}; +function commands(result: Record | null): Record { + const value = result?.nativeRuntimeResponses; + return value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} +const requestKey = (id: string) => createHash("sha256").update(id).digest("hex"); + +/** The run owns these commands just as it owns Stop intent. HTTP admission is + * durable; the controller lease and canonical request lifecycle own dispatch. */ +export async function enqueueNativeRuntimeResponse(db: Db, input: { + request: PendingNativeRuntimeRequest; + actor: NativeRuntimeRequestResolver; + localImplicit: boolean; + resolution: HarnessRuntimeRequestResolution; +}) { + assertNativeRuntimeRequestResolverAuthorized(input.request, input.actor); + const key = requestKey(input.request.requestId); + const fingerprint = JSON.stringify({ request: input.request, userId: input.actor.userId, resolution: input.resolution }); + return db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.nativeRuntimeResponses); + const [run] = await tx.select().from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.request.runId), eq(heartbeatRuns.companyId, input.request.companyId), + )).for("update").limit(1); + if (!run || run.runtimeMode !== "native" || run.status !== "running") throw conflict("This runner session is no longer accepting runtime responses."); + const existing = commands(run.resultJson); + const prior = existing[key]; + if (prior) { + if (prior.fingerprint !== fingerprint) throw conflict("A different response was already submitted for this runtime request."); + if (prior.state === "denied" || prior.state === "obsolete") throw conflict("This runtime request is no longer actionable."); + return { commandId: prior.commandId }; + } + const pending = await readPendingNativeRuntimeRequest(tx as unknown as Db, input.request); + if (!pending || pending.turnId !== input.request.turnId || pending.requestKind !== input.request.requestKind) { + throw conflict("This runtime request is stale or is no longer pending."); + } + assertNativeRuntimeRequestResolverAuthorized(pending, input.actor); + const command: ResponseCommand = { ...input, request: pending, commandId: `native-runtime-response:${randomUUID()}`, fingerprint, state: "pending" }; + await tx.update(heartbeatRuns).set({ resultJson: { ...run.resultJson, nativeRuntimeResponses: { ...existing, [key]: command } }, + updatedAt: new Date() }).where(eq(heartbeatRuns.id, run.id)); + return { commandId: command.commandId }; + }); +} + +/** Recheck access at consumption, including revocation during controller + * replacement. Only the configured local-trusted operator bypasses membership. */ +async function authorize(db: Db, command: ResponseCommand) { + const pending = await readPendingNativeRuntimeRequest(db, command.request); + if (!pending || pending.turnId !== command.request.turnId || pending.requestKind !== command.request.requestKind) { + throw new RuntimeResponseObsoleteError(); + } + const access = accessService(db); + const local = command.localImplicit && loadConfig().deploymentMode === "local_trusted"; + const admin = local || await access.isInstanceAdmin(command.actor.userId); + if (!local) { + const membership = await access.getMembership(command.request.companyId, "user", command.actor.userId); + if (membership?.status !== "active" || (!admin && membership.membershipRole === "viewer")) { + throw new NativeRuntimeRequestResolutionAuthorizationError("native_runtime_request_resolver_denied"); + } + } + assertNativeRuntimeRequestResolverAuthorized(pending, { ...command.actor, isInstanceAdmin: admin }); + return { pending, actor: { ...command.actor, isInstanceAdmin: admin } }; +} +class RuntimeResponseObsoleteError extends Error {} + +export function registerNativeRuntimeResponseDelivery(db: Db, coordinator: ReturnType) { + const worker = coordinator.register(DELIVERY_QUEUES.nativeRuntimeResponses, { + retryMs: 1000, runDuringDrain: true, + async run(signal) { + const ids = [...activeNativeExecutionRunIds]; + if (!ids.length) return; + const identity = await currentNativeControllerIdentity(); + const rows = await db.select({ id: heartbeatRuns.id, result: heartbeatRuns.resultJson }).from(heartbeatRuns) + .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and( + inArray(heartbeatRuns.id, ids), eq(nativeRunFinalizations.controllerBootId, identity.bootId), + gt(nativeRunFinalizations.leaseExpiresAt, sql`clock_timestamp()`), + sql`${heartbeatRuns.resultJson} ? 'nativeRuntimeResponses'`, + )); + for (const row of rows) for (const [key, command] of Object.entries(commands(row.result))) { + signal.throwIfAborted(); + if (command.state !== "pending" || !activeNativeExecutionRunIds.has(row.id)) continue; + let state: ResponseCommand["state"] = "queued"; + try { + const { resolveNativeRuntimeRequest, NativeRuntimeRequestResolutionError } = await import("./native-session-executor.js"); + try { + await resolveNativeRuntimeRequest({ runId: row.id, requestId: command.request.requestId, + turnId: command.request.turnId, resolution: command.resolution, commandId: command.commandId, + authorizeBeforeDispatch: async () => { await authorize(db, command); } }); + } catch (error) { + if (error instanceof NativeRuntimeRequestResolutionError && error.code === "runtime_request_stale_turn") { + throw new RuntimeResponseObsoleteError(); + } + if (!(error instanceof NativeRuntimeRequestResolutionError) + || !["native_session_not_active", "runtime_request_resolution_unsupported"].includes(error.code)) throw error; + const { pending, actor } = await authorize(db, command); + const { queueRunnerPrpRuntimeRequestResolution } = await import("../../realtime/runner-prp-ws.js"); + queueRunnerPrpRuntimeRequestResolution({ companyId: pending.companyId, runId: row.id, + pendingRequest: pending, actor, resolution: command.resolution, commandId: command.commandId }); + } + } catch (error) { + if (error instanceof RuntimeResponseObsoleteError) state = "obsolete"; + else if (error instanceof NativeRuntimeRequestResolutionAuthorizationError) state = "denied"; + else throw error; // The concrete pending command remains retryable. + } + await db.transaction(async tx => { + const [current] = await tx.select({ result: heartbeatRuns.resultJson }).from(heartbeatRuns) + .where(eq(heartbeatRuns.id, row.id)).for("update").limit(1); + const saved = commands(current?.result ?? null); + if (saved[key]?.commandId !== command.commandId || saved[key]?.state !== "pending") return; + await tx.update(heartbeatRuns).set({ resultJson: { ...current.result, nativeRuntimeResponses: { + ...saved, [key]: { ...saved[key], state }, + } }, updatedAt: new Date() }).where(eq(heartbeatRuns.id, row.id)); + }); + } + }, + hasPending: async () => false, + }); + const unsubscribe = subscribeActiveNativeExecutions(worker.wake); + return { ready: worker.ready, async stop() { unsubscribe(); await worker.stop(); } }; +} diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index ae3c6a1a493..d663a4801de 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -7847,6 +7847,24 @@ describe("native session cancellation", () => { ).resolves.toBe(false); }); + it("keeps duplicate Stop callers waiting for the same provider cleanup", async () => { + let releaseCleanup!: () => void; + state.cancel.mockReturnValue({ cleanup: new Promise(resolve => { releaseCleanup = resolve; }) }); + const running = executePaperclipNativeSession({ db: leaseDb(), execution, runnerInstanceId: "runner" }); + await vi.waitFor(() => expect(state.release).toBeTypeOf("function")); + const first = cancelNativeSession(execution.binding.runId, "operator Stop"); + await vi.waitFor(() => expect(state.cancel).toHaveBeenCalledOnce()); + let acknowledged = false; + const duplicate = cancelNativeSession(execution.binding.runId, "duplicate Stop").then(result => { acknowledged = true; return result; }); + try { + await new Promise(resolve => setTimeout(resolve, 20)); + expect(acknowledged).toBe(false); + releaseCleanup(); + expect(await Promise.all([first, duplicate])).toEqual([true, true]); + expect(state.cancel).toHaveBeenCalledOnce(); + } finally { releaseCleanup(); state.release?.(); await running; } + }); + it("allows cancellation to be retried when the session dispatch fails", async () => { state.cancel.mockImplementationOnce(() => { throw new Error("transport unavailable"); @@ -16381,7 +16399,7 @@ describe("HTTP Stop during remote runner staging", () => { const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL"); const temporary = externalUrl ? { connectionString: externalUrl, cleanup: () => dbModule.closeRegisteredClients(externalUrl) } - : await dbModule.startEmbeddedPostgresTestDatabase("remote-staging-http-"); + : await (await import("../../__tests__/helpers/embedded-postgres.js")).startEmbeddedPostgresTestDatabase("remote-staging-http-"); const httpDb = dbModule.createDb(temporary.connectionString); const localExecution = { ...execution, binding: { ...execution.binding, companyId: randomUUID(), agentId: randomUUID(), issueId: randomUUID(), runId: randomUUID(), executionWorkspaceId: randomUUID() }, diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index c12cbc1bf52..ce7507ab28f 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,4 @@ +import { nativeCancellationHasRemoteOwner, notifyNativeCancellation, waitForNativeCancellation } from "./native-cancellation-delivery.js"; import { publishNativeHarnessBackup } from "./native-harness-backup-publication.js"; import { rehydratePinnedSkillAssets } from "./runtime-context.js"; import { createNativeUsageDelivery } from "./native-usage-delivery.js"; @@ -78,7 +79,7 @@ import { selectRemotePiCompanion } from "./remote-pi-companion.js"; import { readNativeLocalProcessStop, PROCESS_START_REQUESTED, withReleasedNativeRunProcess } from "../native-local-process-stop.js"; import { remoteLeaseCleanupScope } from "../remote-execution-termination.js"; import { connectorToolDefinitionsForSkills } from "../connector-runtime.js"; -import { activeNativeExecutionRunIds } from "./native-active-runs.js"; +import { setNativeExecutionActive } from "./native-active-runs.js"; import { preserveNativeFailedWorkspace } from "./native-failed-workspace-recovery.js"; import { boundedExecutionCleanup, @@ -7360,6 +7361,8 @@ export async function steerNativeSession(input: { } } +const nativeCancellationsInFlight = new Map>(); + export function cancelNativeSession( runId: string, reason: string, @@ -7397,6 +7400,23 @@ export async function cancelNativeSession( auditId: string | null; } > { + // HTTP and the overlap command worker can observe the same durable intent. + // Join the entire cleanup/ACK lifetime before re-reading that intent, rather + // than treating cancelRequested as proof that provider cleanup has finished. + while (nativeCancellationsInFlight.has(runId)) { + await nativeCancellationsInFlight.get(runId)!.catch(() => undefined); + } + const pending = performNativeCancellation(runId, reason, options); + nativeCancellationsInFlight.set(runId, pending); + try { return await pending; } + finally { nativeCancellationsInFlight.delete(runId); } +} + +async function performNativeCancellation( + runId: string, + reason: string, + options?: { db: Db; scope?: "turn" | "run" | "issue"; replacementAccepted?: boolean; cancellationRequestId?: string }, +) { const runStop = (options?.scope ?? "run") === "run"; let decision: NativeStatusDecision | null = null; let decisionContext: { @@ -7486,6 +7506,7 @@ export async function cancelNativeSession( const effects = cancellationDecision.effects.map((effect) => effect.kind); let intentPublication: Parameters[0] | null = null; const intent = await options.db.transaction(async (tx) => { + await notifyNativeCancellation(tx); const lockedRun = await tx .select({ agentId: heartbeatRuns.agentId, @@ -7688,6 +7709,13 @@ export async function cancelNativeSession( decisionId = intent.decisionId; recoveringCancellationIntent = intent.existing; priorCoordinatorDecisionIdAtIntent = intent.priorCoordinatorDecisionId; + if (!activeNativeSessions.has(runId) && !nativeSessionStartups.has(runId) + && await nativeCancellationHasRemoteOwner(options.db, runId)) { + const acknowledged = await waitForNativeCancellation(options.db, { + companyId: cancellationContext.companyId, runId, intentId: intent.intentId, + }); + return { ...acknowledged, decision }; + } if (intent.acknowledged && (!runStop || intent.dispatched || (!nativeSessionStartups.has(runId) && !activeNativeSessions.has(runId)))) { return { @@ -7784,6 +7812,7 @@ export async function cancelNativeSession( decisionId = committed.decision.id; } const acknowledgement = await options.db.transaction(async (tx) => { + await notifyNativeCancellation(tx); const lockedRun = await tx .select({ agentId: heartbeatRuns.agentId, @@ -8088,7 +8117,7 @@ export async function executePaperclipNativeSession(input: { resolve: session => resolveStartup(session), }; nativeSessionStartups.set(runId, startup); - activeNativeExecutionRunIds.add(runId); + setNativeExecutionActive(runId, true); let preparedInput: typeof input = input; let cleanupStagedAttachments: () => Promise = async () => undefined; let sessionScopeId: string | null = null; @@ -8215,7 +8244,7 @@ export async function executePaperclipNativeSession(input: { startup.settle(); if (nativeSessionStartups.get(runId) === startup) { nativeSessionStartups.delete(runId); - activeNativeExecutionRunIds.delete(runId); + setNativeExecutionActive(runId, false); } if ( ownsSessionScope && diff --git a/server/src/services/native-runtime/runtime-context.ts b/server/src/services/native-runtime/runtime-context.ts index 953d97d8d8a..77e7c3d991f 100644 --- a/server/src/services/native-runtime/runtime-context.ts +++ b/server/src/services/native-runtime/runtime-context.ts @@ -169,7 +169,7 @@ async function materializeInstructionBundle(db: Db, agent: RuntimeAgent, agentFi const entryPath = safeRelativePath(current.revision.entryFile, "instruction entry path"); return { entryPath, bundle: await materializeAsset([{ path: entryPath, content: Buffer.from(current.content, "utf8"), mode: 0o444 }]) }; } - const exported = await agentInstructionsService().exportFiles(agent, { rejectSymlinks: true }); + const exported = await agentInstructionsService(db).exportFiles(agent, { rejectSymlinks: true }); if (agentInstructionsBundleMode(agent) === "managed") { // The database head remains authoritative when a saved revision could not // update its disk projection. Overlay the exact committed snapshot directly; diff --git a/server/src/services/plugin-agent-runtime-state.ts b/server/src/services/plugin-agent-runtime-state.ts index a92efeb4697..1478ba98399 100644 --- a/server/src/services/plugin-agent-runtime-state.ts +++ b/server/src/services/plugin-agent-runtime-state.ts @@ -1,5 +1,5 @@ import { and, desc, eq } from "drizzle-orm"; -import { agents, environments, plugins, pluginCompanySettings, resourceLifecycleEvents, type Db } from "@paperclipai/db"; +import { agents, environments, plugins, pluginCompanySettings, resourceLifecycleEvents, fileStorageDb, type Db } from "@paperclipai/db"; import type { PluginAgentRuntimeState, PluginAgentRuntimeIdentity } from "@paperclipai/plugin-sdk"; import { agentRecordQueries } from "../lib/agent-records.js"; import { conflict, forbidden, unprocessable } from "../errors.js"; @@ -16,9 +16,9 @@ import { environmentSkillConfig } from "./environment-skills.js"; async function readState(db: Db, pluginId: string, companyId: string, agentId: string, identityOnly: true): Promise; async function readState(db: Db, pluginId: string, companyId: string, agentId: string, identityOnly?: false): Promise | null>; async function readState(db: Db, pluginId: string, companyId: string, agentId: string, identityOnly = false): Promise | null> { - return db.transaction(async tx => { + return fileStorageDb(db).transaction(async tx => { const [agent] = await tx.select().from(agents) - .where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))).for("share"); + .where(and(eq(agents.companyId, companyId), eq(agents.id, agentId))).for("update"); if (!agent) return null; const [plugin] = await tx.select().from(plugins).where(eq(plugins.id, pluginId)); const [settings] = await tx.select().from(pluginCompanySettings).where(and(eq(pluginCompanySettings.companyId, companyId), eq(pluginCompanySettings.pluginId, pluginId))); @@ -35,7 +35,7 @@ async function readState(db: Db, pluginId: string, companyId: string, agentId: s lifecycleState: agent.lifecycleState, lifecycleVersion: agent.lifecycleVersion }, instanceSecrets: await agentInstanceSecretService(tx as unknown as Db).read({ companyId, agentId, pluginId }), }; - const instructions = await agentInstructionsService().getBundleLocation(agent); + const instructions = await agentInstructionsService(tx as unknown as Db).getBundleLocation(agent); const records = agentRecordQueries(tx as unknown as Db); const companyAgents = await records.listCompanyAgentEligibilityRows(companyId); return { @@ -72,7 +72,7 @@ export async function pluginAgentRuntimeState(db: Db, pluginId: string, companyI throw unprocessable("An environment skill is unavailable to this agent"); } const connectionAssignments = await pluginAgentConnectionAssignments(db, companyId, agentId, { publicMcp: settings.enablePublicMcp, chatConnectors: settings.enableChatConnectors }); - const fallback = !state.instructionsRootPath ? await agentInstructionsService().exportFiles(state.agent, { rejectSymlinks: true }) : null; + const fallback = !state.instructionsRootPath ? await agentInstructionsService(db).exportFiles(state.agent, { rejectSymlinks: true }) : null; const entryFile = fallback?.entryFile ?? state.instructionsEntryFile; const entryContent = fallback?.files[entryFile]; const current = await readState(db, pluginId, companyId, agentId); diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index 8d42d7e7e38..61776878ae0 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -72,6 +72,30 @@ export function shouldUseRemoteWorkspaces(input: { return configuredRemoteWorkspacePersistence() !== null; } +/** Deployment profile admission belongs beside workspace authority. A warm + * remote controller may replace its local projection; host-owned execution + * cannot make that promise. External conversation providers own no task disk. */ +export function assertEphemeralWorkspaceExecution(input: { + externalConversation: boolean; native: boolean; adapterType: string; + pluginTaskWorkspace: boolean; remoteAuthority: boolean; +}) { + if (process.env.PAPERCLIP_EPHEMERAL_SERVER !== "true" || input.externalConversation) return; + if (!input.native || input.adapterType !== "paperclip_runner" + || (!input.pluginTaskWorkspace && !input.remoteAuthority)) { + throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.", + { code: "ephemeral_execution_configuration_required" }); + } +} + +/** The runtime selector calls this after environment, credentials and saved + * execution policy have resolved. Raw agent settings are not lifecycle authority. */ +export function assertEphemeralNativeLifecycle(mode: "warm" | "per_turn") { + if (process.env.PAPERCLIP_EPHEMERAL_SERVER === "true" && mode !== "warm") { + throw unprocessable("Volume-free servers require warm Paperclip Runner execution with remote workspace storage.", + { code: "ephemeral_execution_configuration_required" }); + } +} + function configuredRemoteWorkspacePersistence(storage?: StorageService) { const config = loadConfig(); const databaseUrl = config.databaseUrl?.trim(); diff --git a/server/src/services/task-admission.ts b/server/src/services/task-admission.ts index 9ae676dd1a2..46b6d28cf2b 100644 --- a/server/src/services/task-admission.ts +++ b/server/src/services/task-admission.ts @@ -10,6 +10,10 @@ const changeListeners = new Set<() => void>(); function notifyDrainChanged() { for (const listener of changeListeners) listener(); } +export function subscribeTaskDrainChanges(listener: () => void) { + changeListeners.add(listener); + return () => { changeListeners.delete(listener); }; +} // The shared coordinator owns this process-local deadline and its shutdown. // Reading admission can also establish expiry, so both paths notify the same diff --git a/server/src/shutdown.test.ts b/server/src/shutdown.test.ts index 754a99fb1e1..754902cc3ba 100644 --- a/server/src/shutdown.test.ts +++ b/server/src/shutdown.test.ts @@ -297,6 +297,20 @@ describe("closeHttpListenerForShutdown", () => { } }); + it("waits for HTTP work without waiting for upgraded sockets to close", async () => { + const server = fakeServer({ listening: true, closeDelayMs: null }); + const requests = deferred(); + let finished = false; + const drain = closeHttpListenerForShutdown({ server, signal: "SIGTERM", log: stubLogger(), + drainRequests: () => requests.promise }).then(result => { finished = true; return result; }); + await Promise.resolve(); + expect(finished).toBe(false); + requests.resolve(); + await expect(drain).resolves.toBe("closed"); + expect(server.close).toHaveBeenCalledOnce(); + expect(server.closeAllConnections).not.toHaveBeenCalled(); + }); + it("does nothing when the listener was never bound", async () => { const server = fakeServer({ listening: false }); await expect( diff --git a/server/src/shutdown.ts b/server/src/shutdown.ts index ea927cbd6e1..45721a5d4a3 100644 --- a/server/src/shutdown.ts +++ b/server/src/shutdown.ts @@ -55,22 +55,28 @@ export async function closeHttpListenerForShutdown(input: { server: ShutdownHttpListener; signal: "SIGINT" | "SIGTERM"; timeoutMs?: number; + drainRequests?: (signal: AbortSignal) => Promise; log: ShutdownLogger; }): Promise<"closed" | "timed_out" | "not_listening"> { if (!input.server.listening) return "not_listening"; const timeoutMs = input.timeoutMs ?? 5_000; let timer: NodeJS.Timeout | null = null; + const controller = new AbortController(); try { + const closed = new Promise<"closed">((resolve) => { + input.server.close((err) => { + if (err && (err as NodeJS.ErrnoException).code !== "ERR_SERVER_NOT_RUNNING") { + input.log.error({ err, signal: input.signal }, "HTTP listener close failed"); + } + resolve("closed"); + }); + input.server.closeIdleConnections?.(); + }); return await Promise.race([ - new Promise<"closed">((resolve) => { - input.server.close((err) => { - if (err && (err as NodeJS.ErrnoException).code !== "ERR_SERVER_NOT_RUNNING") { - input.log.error({ err, signal: input.signal }, "HTTP listener close failed"); - } - resolve("closed"); - }); + input.drainRequests ? input.drainRequests(controller.signal).then(() => { input.server.closeIdleConnections?.(); - }), + return "closed" as const; + }) : closed, new Promise<"timed_out">((resolve) => { timer = setTimeout(() => { input.log.info( @@ -85,6 +91,7 @@ export async function closeHttpListenerForShutdown(input: { ]); } finally { if (timer) clearTimeout(timer); + controller.abort(); } } diff --git a/server/src/startup-recovery-state.ts b/server/src/startup-recovery-state.ts index 5d4b597cf59..77c0a788592 100644 --- a/server/src/startup-recovery-state.ts +++ b/server/src/startup-recovery-state.ts @@ -1,4 +1,4 @@ -export type StartupRecoveryPhase = "starting" | "recovering" | "ready"; +export type StartupRecoveryPhase = "starting" | "recovering" | "ready" | "draining"; let phase: StartupRecoveryPhase = "ready"; let updatedAt = new Date().toISOString(); diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 9576ed463ff..3cfe7626586 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -3274,3 +3274,28 @@ The shared report navigation links the dated, immutable catalog snapshot. Deploying this generator does not rewrite existing published history HTML. Activating the link on an existing page requires a separately reviewed, conditional navigation-only update. The metadata command publishes nothing. +### Control-plane rolling deployment smoke + +`rolling-deployment-smoke.ts` runs actual source servers A/B/C behind a local +HTTP/WebSocket proxy with no request retries. It requires disposable loopback +PostgreSQL and S3-compatible storage, and creates/deletes an owned database and +bucket per attempt. Each server has a separate home and temporary directory. +Private JSON configuration files must be mode 600 and must not be committed: + +- Database configuration: `{ "url": "" }`. +- Storage configuration: `{ "endpoint": "", "accessKeyId": "", "secretAccessKey": "" }`. + +```sh +PAPERCLIP_ROLLING_TEST_DATABASE_CONFIG=/private/tmp/rolling-database.json \ +PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG=/private/tmp/rolling-storage.json \ +node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/rolling-deployment-smoke.ts +``` + +The receipt records source/patch/lockfile identity, individual assertions, timing +and cleanup. It exercises continuous task reads/writes, concurrent file edits, +stale-write conflict, an admitted file save during SIGTERM, S3 failure with both +transfer connections held, and complete deletion of both earlier server homes. +It uses `local_trusted` authentication and no model provider. Its result does not +qualify browser reconnects, active-agent continuity, distinct packaged releases +or Railway. Keep failed receipts alongside successful attempts; the private +server logs and config files may contain credentials and are not shareable evidence. diff --git a/tests/runner-e2e/rolling-deployment-browser.ts b/tests/runner-e2e/rolling-deployment-browser.ts new file mode 100644 index 00000000000..e78437e1fe9 --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-browser.ts @@ -0,0 +1,64 @@ +import assert from "node:assert/strict"; +import { chromium, expect } from "@playwright/test"; +import path from "node:path"; + +/** The actual built board UI, real APIs and ordinary browser navigation. No + * route stubs, response injection, request retries or fabricated UI state. */ +export async function openRollingDeploymentBrowser(input: { origin: string; root: string; taskTitle: string; taskPath: string; omittedAssets: string[] }) { + const browser = await chromium.launch({ headless: true }); + const context = await browser.newContext({ viewport: { width: 1440, height: 1000 } }); + const page = await context.newPage(); + const sockets: { openedAt: number; closedAt?: number; firstFrameAt?: number }[] = []; + const errors: string[] = []; + const assets: string[] = []; + const metrics: { reconnectMs?: number; oldChunkLoaded: boolean; pageErrors: number } = { oldChunkLoaded: false, pageErrors: 0 }; + page.on("pageerror", error => errors.push(error.message)); + page.on("response", response => { if (response.url().includes("/assets/") && response.ok()) assets.push(response.url()); }); + page.on("websocket", socket => { + if (!socket.url().includes("/events/ws")) return; + const record = { openedAt: Date.now() } as typeof sockets[number]; sockets.push(record); + socket.on("framereceived", () => { record.firstFrameAt ??= Date.now(); }); + socket.on("close", () => { record.closedAt = Date.now(); }); + }); + try { + await page.goto(`${input.origin}${input.taskPath}`); + await expect(page.getByRole("heading", { name: input.taskTitle, exact: true }).first()).toBeVisible({ timeout: 30_000 }); + await expect.poll(() => sockets.some(socket => socket.firstFrameAt !== undefined), { timeout: 15_000 }).toBe(true); + assert.equal(assets.some(url => input.omittedAssets.some(asset => url.endsWith(asset))), false, "Old lazy chunk must not have loaded before deployment"); + await page.screenshot({ path: path.join(input.root, "browser-before.png") }); + return { + metrics, + async verifyReplacement() { + try { + await expect.poll(() => sockets.length, { timeout: 5_000 }).toBeGreaterThan(1); + await expect.poll(() => sockets.at(-1)?.firstFrameAt !== undefined, { timeout: 5_000 }).toBe(true); + const closedAt = sockets.find(socket => socket.closedAt !== undefined)?.closedAt; + assert.ok(closedAt); + const reconnected = sockets.find(socket => socket.openedAt >= closedAt && socket.firstFrameAt); + assert.ok(reconnected); + const reconnectMs = reconnected.firstFrameAt! - closedAt; + metrics.reconnectMs = reconnectMs; + assert.ok(reconnectMs <= 5_000, `Browser reconnected after ${reconnectMs}ms`); + // Use the existing tab's router so its original build requests the + // previously unopened page's chunk from B after A has been removed. + const activity = page.getByRole("link", { name: /^(Audit|Activity)$/ }).first(); + await activity.click(); + await expect.poll(() => assets.some(url => input.omittedAssets.some(asset => url.endsWith(asset))), { timeout: 15_000 }).toBe(true); + await expect(page.getByRole("heading", { name: /^(Audit|Activity)$/ }).first()).toBeVisible({ timeout: 15_000 }); + metrics.oldChunkLoaded = true; + metrics.pageErrors = errors.length; + assert.deepEqual(errors, []); + await page.screenshot({ path: path.join(input.root, "browser-after.png") }); + return metrics; + } catch (error) { + await page.screenshot({ path: path.join(input.root, "browser-failure.png") }).catch(() => undefined); + throw error; + } + }, + close: () => browser.close(), + }; + } catch (error) { + await page.screenshot({ path: path.join(input.root, "browser-failure.png") }).catch(() => undefined); + await browser.close(); throw error; + } +} diff --git a/tests/runner-e2e/rolling-deployment-process.ts b/tests/runner-e2e/rolling-deployment-process.ts new file mode 100644 index 00000000000..2951b236452 --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-process.ts @@ -0,0 +1,94 @@ +import { spawn, type ChildProcess } from "node:child_process"; +import { createWriteStream } from "node:fs"; +import { cp, mkdir, readFile, rm, symlink, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; +import { reserveRunnerE2EServerPort } from "./ports.js"; +import { prepareRunnerE2EServerConfig } from "./server-config.js"; +import { createRunnerE2EServerStopper } from "./server-stop.js"; +import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; + +export async function startRollingDeploymentProcess(input: { + root: string; + name: string; + repositoryRoot: string; + databaseUrl: string; + ingressOrigin: string; + sharedEnvironment: NodeJS.ProcessEnv; + browser?: { omitAssets?: string[] }; +}) { + const home = path.join(input.root, input.name); + const temporary = path.join(home, "tmp"); + await mkdir(temporary, { recursive: true, mode: 0o700 }); + const port = await reserveRunnerE2EServerPort(); + const configPath = path.join(home, "instances/default/config.json"); + const reservation = await prepareRunnerE2EServerConfig({ temporaryRoot: home, configPath, serverPort: port }); + await reservation?.close(); + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8"))); + config.database = { ...config.database, mode: "postgres", connectionString: input.databaseUrl }; + config.server.serveUi = Boolean(input.browser); + await writeFile(configPath, JSON.stringify(config), { mode: 0o600 }); + const log = createWriteStream(path.join(input.root, `${input.name}.log`), { mode: 0o600 }); + const spawnErrors = new WeakMap(); + const stopper = createRunnerE2EServerStopper({ + gracefulTimeoutMs: 90_000, forcedTimeoutMs: 5_000, + hasSpawnError: child => spawnErrors.has(child), markExpectedStop() {}, + log: message => log.write(message), + }); + let entry = path.join(input.repositoryRoot, "server/src/index.ts"); + if (input.browser) { + // Independent build directories let the browser test remove A completely + // and omit an old lazy chunk from B without mutating the checkout's build. + const appRoot = path.join(home, "app/server"); + await mkdir(appRoot, { recursive: true }); + await cp(path.join(input.repositoryRoot, "server/dist"), path.join(appRoot, "dist"), { recursive: true }); + await cp(path.join(input.repositoryRoot, "ui/dist"), path.join(appRoot, "ui-dist"), { recursive: true }); + await symlink(path.join(input.repositoryRoot, "server/node_modules"), path.join(appRoot, "node_modules")); + await writeFile(path.join(appRoot, "package.json"), '{"type":"module"}'); + for (const omittedAsset of input.browser.omitAssets ?? []) { + if (!/^[A-Za-z0-9_.-]+\.js$/.test(omittedAsset)) throw new Error("Unsafe omitted asset fixture"); + for (const suffix of ["", ".br", ".gz"]) await rm(path.join(appRoot, "ui-dist/assets", omittedAsset + suffix), { force: true }); + } + entry = path.join(appRoot, "dist/index.js"); + } + const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(input.repositoryRoot, entry), { + cwd: input.repositoryRoot, + env: { + PATH: process.env.PATH, HOME: home, TMPDIR: temporary, + NODE_ENV: "test", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, + PAPERCLIP_INSTANCE_ID: "default", DATABASE_URL: input.databaseUrl, + PAPERCLIP_API_URL: input.ingressOrigin, PAPERCLIP_TELEMETRY_ENABLED: "false", + PAPERCLIP_DISABLE_TELEMETRY: "true", PAPERCLIP_LOG_LEVEL: "warn", + HEARTBEAT_SCHEDULER_ENABLED: "false", ...input.sharedEnvironment, + }, + detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"], + }); + child.on("error", error => spawnErrors.set(child, error)); + child.stdout!.pipe(log, { end: false }); child.stderr!.pipe(log, { end: false }); + await stopper.watch(child); + const origin = `http://127.0.0.1:${port}`; + let stopped = false; + return { + child, home, origin, + async waitForReady() { + const started = performance.now(); + while (performance.now() - started < 180_000) { + if (spawnErrors.has(child) || child.exitCode !== null || child.signalCode !== null) { + throw new Error(`Server ${input.name} exited before readiness; inspect ${input.name}.log`); + } + const health = await fetch(`${origin}/api/ready`, { signal: AbortSignal.timeout(2_000) }) + .then(async response => response.ok ? await response.json() as { status?: string } : null).catch(() => null); + if (health?.status === "ready") return { elapsedMs: Math.round(performance.now() - started), pid: child.pid }; + await delay(100); + } + throw new Error(`Server ${input.name} did not become ready; inspect ${input.name}.log`); + }, + async stop() { + if (stopped) return; + await stopper.stopAll(); + await new Promise(resolve => log.end(resolve)); + stopped = true; + }, + }; +} diff --git a/tests/runner-e2e/rolling-deployment-proxy.test.ts b/tests/runner-e2e/rolling-deployment-proxy.test.ts new file mode 100644 index 00000000000..7b6dd7b6811 --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-proxy.test.ts @@ -0,0 +1,59 @@ +import { createServer } from "node:http"; +import { once } from "node:events"; +import { expect, it } from "vitest"; +import { startRollingDeploymentProxy } from "./rolling-deployment-proxy.js"; + +it("switches new traffic without replaying or cutting an admitted write", async () => { + const writes: string[] = []; + let accept!: () => void; + let finish!: () => void; + const accepted = new Promise(resolve => { accept = resolve; }); + const pending = new Promise(resolve => { finish = resolve; }); + const a = createServer(async (req, res) => { + let body = ""; + for await (const chunk of req) body += chunk; + writes.push(body); accept(); + await pending; + res.end("saved by A"); + }); + const b = createServer((_req, res) => res.end("served by B")); + const proxy = await startRollingDeploymentProxy(); + try { + a.listen(0, "127.0.0.1"); await once(a, "listening"); + b.listen(0, "127.0.0.1"); await once(b, "listening"); + const origin = (server: typeof a) => `http://127.0.0.1:${(server.address() as { port: number }).port}`; + proxy.routeTo(origin(a)); + const saved = fetch(proxy.origin, { method: "POST", body: "one accepted mutation" }); + await accepted; + proxy.routeTo(origin(b)); + expect(await (await fetch(proxy.origin)).text()).toBe("served by B"); + finish(); + expect(await (await saved).text()).toBe("saved by A"); + expect(writes).toEqual(["one accepted mutation"]); + } finally { + finish(); + await proxy.close(); + await Promise.all([a, b].map(server => new Promise(resolve => { + server.close(() => resolve()); server.closeAllConnections(); + }))); + } +}); + +it("reports a failed backend write without retrying it", async () => { + let accepted = 0; + const backend = createServer(async (req, res) => { + for await (const _chunk of req) { /* consume the accepted body */ } + accepted++; + res.destroy(); + }); + const proxy = await startRollingDeploymentProxy(); + try { + backend.listen(0, "127.0.0.1"); await once(backend, "listening"); + proxy.routeTo(`http://127.0.0.1:${(backend.address() as { port: number }).port}`); + expect((await fetch(proxy.origin, { method: "POST", body: "accepted then disconnected" })).status).toBe(502); + expect(accepted).toBe(1); + } finally { + await proxy.close(); + await new Promise(resolve => { backend.close(() => resolve()); backend.closeAllConnections(); }); + } +}); diff --git a/tests/runner-e2e/rolling-deployment-proxy.ts b/tests/runner-e2e/rolling-deployment-proxy.ts new file mode 100644 index 00000000000..dc910f61d15 --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-proxy.ts @@ -0,0 +1,80 @@ +import { Agent, createServer, request } from "node:http"; +import { once } from "node:events"; +import type { Socket } from "node:net"; + +/** Test-owned ingress. Each request keeps the backend selected at admission; + * switching traffic never retries a request or terminates its existing stream. */ +export async function startRollingDeploymentProxy() { + let target: URL | undefined; + const agent = new Agent({ keepAlive: true }); + const sockets = new Set(); + const server = createServer((incoming, outgoing) => { + const selected = target; + if (!selected) { outgoing.writeHead(503).end(); return; } + const upstream = request(selected, { + path: incoming.url ?? "/", + method: incoming.method, + headers: { ...incoming.headers, host: selected.host }, + agent, + }, response => { + outgoing.writeHead(response.statusCode ?? 502, response.headers); + response.pipe(outgoing); + response.on("error", () => outgoing.destroy()); + }); + upstream.on("error", () => { + if (!outgoing.headersSent) outgoing.writeHead(502).end(); + else outgoing.destroy(); + }); + incoming.on("aborted", () => upstream.destroy()); + outgoing.on("close", () => { if (!outgoing.writableFinished) upstream.destroy(); }); + incoming.pipe(upstream); + }); + server.on("connection", socket => { + sockets.add(socket); + socket.on("close", () => sockets.delete(socket)); + }); + server.on("upgrade", (incoming, socket, head) => { + if (!target) { socket.destroy(); return; } + const upstream = request(target, { + path: incoming.url ?? "/", + headers: { ...incoming.headers, host: target.host }, agent: false, + }); + socket.on("error", () => upstream.destroy()); + upstream.on("upgrade", (response, backend, backendHead) => { + const headers = response.rawHeaders; + socket.write(`HTTP/1.1 ${response.statusCode} ${response.statusMessage}\r\n` + + headers.reduce((text, item, index) => text + (index % 2 ? `${item}\r\n` : `${item}: `), "") + "\r\n"); + if (backendHead.length) socket.write(backendHead); + if (head.length) backend.write(head); + backend.on("error", () => socket.destroy()); + socket.on("error", () => backend.destroy()); + backend.on("close", () => socket.destroy()); + socket.on("close", () => backend.destroy()); + backend.pipe(socket).pipe(backend); + }); + upstream.on("response", response => { response.resume(); socket.destroy(); }); + upstream.on("error", () => socket.destroy()); + upstream.end(); + }); + server.listen(0, "127.0.0.1"); + await once(server, "listening"); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Proxy address unavailable"); + return { + origin: `http://127.0.0.1:${address.port}`, + routeTo(origin: string) { + const next = new URL(origin); + if (next.protocol !== "http:" || next.hostname !== "127.0.0.1" || next.username || next.password) { + throw new Error("Rolling deployment fixture backends must use loopback HTTP"); + } + target = next; + }, + async close() { + const closed = once(server, "close"); + server.close(); + agent.destroy(); + for (const socket of sockets) socket.destroy(); + await closed; + }, + }; +} diff --git a/tests/runner-e2e/rolling-deployment-smoke.ts b/tests/runner-e2e/rolling-deployment-smoke.ts new file mode 100644 index 00000000000..1485814d2b6 --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-smoke.ts @@ -0,0 +1,301 @@ +/** Credential-free full-server overlap characterization. No model or Runner + * continuity is claimed. Requires an explicitly selected disposable S3 service. + * node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/rolling-deployment-smoke.ts */ +import assert from "node:assert/strict"; +import { createHash, randomBytes, randomUUID } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { readFile, readdir, rm, writeFile } from "node:fs/promises"; +import { createRequire } from "node:module"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { startEmbeddedPostgresTestDatabase } from "../../packages/db/src/index.js"; +import { createExternalControllerDatabase } from "./external-controller-database.js"; +import { reserveRunnerE2EServerPort } from "./ports.js"; +import { createRunnerE2ETemporaryRoot } from "./server-config.js"; +import { openRollingDeploymentBrowser } from "./rolling-deployment-browser.js"; +import { startRollingDeploymentProcess } from "./rolling-deployment-process.js"; +import { startRollingDeploymentProxy } from "./rolling-deployment-proxy.js"; +import { startRollingStorageProxy } from "./rolling-deployment-storage-proxy.js"; + +// Exercise the SDK version owned by the server under test. +const { S3Client, CreateBucketCommand, ListObjectsV2Command, DeleteObjectsCommand, DeleteBucketCommand } = + createRequire(new URL("../../server/package.json", import.meta.url))("@aws-sdk/client-s3"); +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); +const patchHash = createHash("sha256").update(execFileSync("git", ["diff", "--binary", "HEAD"], { cwd: repositoryRoot, maxBuffer: 32 * 1024 * 1024 })); +for (const filename of execFileSync("git", ["ls-files", "--others", "--exclude-standard", "-z"], { cwd: repositoryRoot, encoding: "utf8" }).split("\0").filter(Boolean).sort()) { + patchHash.update(filename).update("\0").update(await readFile(path.join(repositoryRoot, filename))); +} +const root = await createRunnerE2ETemporaryRoot(); +const reportPath = path.join(root, "rolling-deployment-evidence.json"); +const checks: { name: string; passed: boolean; error?: string }[] = []; +const report: Record = { + schema: "paperclip.e2e.rolling-deployment-smoke/v1", startedAt: new Date().toISOString(), + sourceRevision: execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8" }).trim(), + sourceDirty: execFileSync("git", ["status", "--porcelain"], { cwd: repositoryRoot, encoding: "utf8" }).trim().length > 0, + sourcePatchSha256: patchHash.digest("hex"), nodeVersion: process.version, + lockfileSha256: createHash("sha256").update(await readFile(path.join(repositoryRoot, "pnpm-lock.yaml"))).digest("hex"), + artifactKind: "source", authentication: "local_trusted", readinessProbe: "/api/ready", + activeAgentContinuityQualified: false, browserQualified: false, checks, +}; +console.info(JSON.stringify({ started: true, evidence: reportPath })); +const saveReport = () => writeFile(reportPath, `${JSON.stringify(report, null, 2)}\n`, { mode: 0o600 }); +const processes: Awaited>[] = []; +let database: Awaited> | undefined; +let proxy: Awaited> | undefined; +let browser: Awaited> | undefined; +let storageProxy: Awaited> | undefined; +let s3: InstanceType | undefined; +const bucket = `rolling-deploy-${randomUUID()}`; +let bucketCreated = false; +let probing = false; +let probes: Promise | undefined; +const requestErrors: string[] = []; +const acceptedWrites: string[] = []; +let reads = 0; +async function api(origin: string, route: string, method = "GET", body?: object) { + const response = await fetch(`${origin}/api${route}`, { + method, headers: { "content-type": "application/json", origin }, + body: body ? JSON.stringify(body) : undefined, signal: AbortSignal.timeout(10_000), + }); + if (!response.ok) throw new Error(`${method} ${route} returned ${response.status}`); + return response.json() as Promise>; +} +async function check(name: string, run: () => Promise) { + try { await run(); checks.push({ name, passed: true }); } + catch (error) { checks.push({ name, passed: false, error: error instanceof Error ? error.message : String(error) }); } + await saveReport(); +} +try { + const storageFile = process.env.PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG; + if (!storageFile) throw new Error("Set PAPERCLIP_ROLLING_TEST_STORAGE_CONFIG to private disposable S3 connection JSON"); + const storage = JSON.parse(await readFile(storageFile, "utf8")) as { + endpoint: string; accessKeyId: string; secretAccessKey: string; + }; + const endpoint = new URL(storage.endpoint); + if (endpoint.protocol !== "http:" || !["127.0.0.1", "localhost", "[::1]"].includes(endpoint.hostname) + || endpoint.username || endpoint.password || !storage.accessKeyId || !storage.secretAccessKey) { + throw new Error("Rolling smoke requires an explicit loopback S3 service with disposable credentials"); + } + s3 = new S3Client({ region: "us-east-1", endpoint: storage.endpoint, forcePathStyle: true, + credentials: { accessKeyId: storage.accessKeyId, secretAccessKey: storage.secretAccessKey } }); + await s3.send(new CreateBucketCommand({ Bucket: bucket })); bucketCreated = true; + storageProxy = await startRollingStorageProxy(storage.endpoint); + const setupStarted = performance.now(); + if (process.env.PAPERCLIP_ROLLING_TEST_DATABASE_CONFIG) { + const { url } = JSON.parse(await readFile(process.env.PAPERCLIP_ROLLING_TEST_DATABASE_CONFIG, "utf8")) as { url: string }; + const configPath = path.join(root, "database-owner/config.json"); + const owned = await createExternalControllerDatabase({ adminUrl: url, temporaryRoot: root, + configPath, serverPort: await reserveRunnerE2EServerPort() }); + // Register cleanup before reading the generated configuration. + database = { connectionString: "", cleanup: owned.cleanup }; + const saved = JSON.parse(await readFile(configPath, "utf8")) as { database: { connectionString: string } }; + database.connectionString = saved.database.connectionString; + } else { + database = await startEmbeddedPostgresTestDatabase("rolling-deployment-"); + } + proxy = await startRollingDeploymentProxy(); + const sharedEnvironment = { + PAPERCLIP_DURABLE_AGENT_FILES: "true", + PAPERCLIP_SHARED_UI_ASSETS: "true", + PAPERCLIP_DATABASE_LISTEN_URL: database.connectionString, + PAPERCLIP_SECRETS_MASTER_KEY: randomBytes(32).toString("base64"), + PAPERCLIP_AGENT_JWT_SECRET: randomUUID(), PAPERCLIP_DECISION_SIGNING_SECRET: randomUUID(), + PAPERCLIP_TOOL_ACTION_SIGNING_SECRET: randomUUID(), BETTER_AUTH_SECRET: randomUUID(), + PAPERCLIP_STORAGE_PROVIDER: "s3", PAPERCLIP_STORAGE_S3_BUCKET: bucket, + PAPERCLIP_STORAGE_S3_REGION: "us-east-1", PAPERCLIP_STORAGE_S3_ENDPOINT: storageProxy.endpoint, + PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE: "true", AWS_ACCESS_KEY_ID: storage.accessKeyId, + AWS_SECRET_ACCESS_KEY: storage.secretAccessKey, AWS_EC2_METADATA_DISABLED: "true", + RUN_LOG_S3_BUCKET: bucket, RUN_LOG_S3_ENDPOINT: storage.endpoint, + RUN_LOG_S3_REGION: "us-east-1", RUN_LOG_S3_FORCE_PATH_STYLE: "true", + }; + const browserEnabled = process.env.PAPERCLIP_ROLLING_TEST_BROWSER === "true"; + report.artifactKind = browserEnabled ? "compiled-server-with-workspace-dependencies" : "source"; + const omittedAssets = browserEnabled ? (await readdir(path.join(repositoryRoot, "ui/dist/assets"))).filter(name => /^CompanyActivity(?:\.production)?-[^.]+\.js$/.test(name)) : []; + if (browserEnabled && !omittedAssets.length) throw new Error("Build the UI before browser qualification"); + const start = async (name: string, durableAgentFiles = true) => { + const server = await startRollingDeploymentProcess({ root, name, repositoryRoot, + databaseUrl: database!.connectionString, ingressOrigin: proxy!.origin, + ...(browserEnabled ? { browser: name === "A" ? {} : { omitAssets: omittedAssets } } : {}), + sharedEnvironment: { ...sharedEnvironment, PAPERCLIP_EPHEMERAL_SERVER: String(durableAgentFiles), PAPERCLIP_DURABLE_AGENT_FILES: String(durableAgentFiles) } }); + processes.push(server); + report[`${name}Ready`] = await server.waitForReady(); + return server; + }; + const a = await start("A"); proxy.routeTo(a.origin); + report.setupElapsedMs = Math.round(performance.now() - setupStarted); + const company = await api(proxy.origin, "/companies", "POST", { name: "Rolling deployment qualification" }); + const task = await api(proxy.origin, `/companies/${company.id}/issues`, "POST", { title: "Persist through deployment" }); + const agent = await api(proxy.origin, `/companies/${company.id}/agents`, "POST", { + name: "Storage qualification", role: "engineer", adapterType: "process", + adapterConfig: { command: "/usr/bin/true" }, + }); + const fileRoute = `/agents/${agent.id}/instructions-bundle/file`; + await api(proxy.origin, fileRoute, "PUT", { path: "AGENTS.md", content: "Synthetic deployment fixture.", baseRevisionId: null }); + let content = `Acknowledged file ${randomUUID()}`; + await api(proxy.origin, fileRoute, "PUT", { path: "reference.txt", content, baseHash: null }); + probing = true; + probes = (async () => { + let sequence = 0; + while (probing) { + try { + const saved = await api(proxy!.origin, `/issues/${task.id}`); assert.equal(saved.id, task.id); reads++; + if (sequence++ % 5 === 0) { + const key = randomUUID(); + const comment = await api(proxy!.origin, `/issues/${task.id}/comments`, "POST", { body: key, clientRequestId: key }); + acceptedWrites.push(String(comment.id)); + } + } catch (error) { requestErrors.push(error instanceof Error ? error.message : String(error)); } + await delay(100); + } + })(); + if (browserEnabled) browser = await openRollingDeploymentBrowser({ origin: proxy.origin, root, taskTitle: task.title, + taskPath: `/issues/${task.id}`, omittedAssets }); + const b = await start("B"); + await check("B reads acknowledged task while A is alive", async () => { + assert.equal((await api(b.origin, `/issues/${task.id}`)).title, task.title); + }); + await check("B reads acknowledged agent file while A filesystem exists", async () => { + assert.equal((await api(b.origin, `${fileRoute}?path=reference.txt`)).content, content); + }); + await check("Concurrent servers observe file saves and enforce stale-write conflicts", async () => { + const before = await api(a.origin, `${fileRoute}?path=reference.txt`); + content = `Saved on B ${randomUUID()}`; + await api(b.origin, fileRoute, "PUT", { path: "reference.txt", content, baseHash: before.contentHash }); + const onA = await api(a.origin, `${fileRoute}?path=reference.txt`); + assert.equal(onA.content, content); + const conflict = await fetch(`${a.origin}/api${fileRoute}`, { method: "PUT", + headers: { "content-type": "application/json", origin: a.origin }, + body: JSON.stringify({ path: "reference.txt", content: "Stale write must fail", baseHash: before.contentHash }), + signal: AbortSignal.timeout(10_000) }); + assert.equal(conflict.status, 409); await conflict.arrayBuffer(); + content = `Saved on A after B cached its files ${randomUUID()}`; + await api(a.origin, fileRoute, "PUT", { path: "reference.txt", content, baseHash: onA.contentHash }); + assert.equal((await api(b.origin, `${fileRoute}?path=reference.txt`)).content, content); + }); + const switchedAt = performance.now(); + await check("A finishes an admitted save after traffic switches and SIGTERM starts draining", async () => { + storageProxy!.holdUploads(); + let acknowledged = false; + const saving = api(proxy!.origin, fileRoute, "PUT", { path: "drained-save.txt", content: "Saved during HTTP drain", baseHash: null }) + .then(result => { acknowledged = true; return result; }); + void saving.catch(() => undefined); + let stopping: Promise | undefined; + try { + const deadline = Date.now() + 8_000; + while (storageProxy!.heldUploads < 1 && Date.now() < deadline) await delay(25); + assert.equal(storageProxy!.heldUploads, 1); + proxy!.routeTo(b.origin); + stopping = a.stop(); + await delay(100); + assert.equal(acknowledged, false); + storageProxy!.resumeUploads(); + assert.equal((await saving).content, "Saved during HTTP drain"); + } finally { + storageProxy!.resumeUploads(); + await saving.catch(() => undefined); + await stopping; + } + }); + proxy.routeTo(b.origin); + await delay(500); + await a.stop(); + await rm(a.home, { recursive: true, force: true }); + await delay(1_000); + report.handoffElapsedMs = Math.round(performance.now() - switchedAt); + if (browser) await check("Existing browser reconnects and opens an old lazy chunk after A removal", async () => { + report.browser = browser!.metrics; + await browser!.verifyReplacement(); report.browserQualified = true; + }); + probing = false; await probes; + report.reads = reads; report.acknowledgedWrites = acceptedWrites.length; + await check("HTTP requests succeed across overlap and shutdown", async () => { + assert.ok(reads > 0); assert.ok(acceptedWrites.length > 0); assert.deepEqual(requestErrors, []); + }); + await check("Every acknowledged write survives A removal exactly once", async () => { + const comments = await api(b.origin, `/issues/${task.id}/comments`) as unknown as { id: string }[]; + for (const id of acceptedWrites) assert.equal(comments.filter(comment => comment.id === id).length, 1); + }); + await check("Agent file survives A filesystem removal", async () => { + assert.equal((await api(b.origin, `${fileRoute}?path=reference.txt`)).content, content); + assert.equal((await api(b.origin, `${fileRoute}?path=drained-save.txt`)).content, "Saved during HTTP drain"); + }); + await check("Object-store outage preserves files and leaves ordinary requests responsive", async () => { + const fixtureAgents = await Promise.all([0, 1, 2].map(async index => { + const fixture = await api(b.origin, `/companies/${company.id}/agents`, "POST", { + name: `Storage failure ${index}`, role: "engineer", adapterType: "process", adapterConfig: { command: "/usr/bin/true" }, + }); + // Exercise both previously committed files and an agent's first save. + if (index < 2) await api(b.origin, `/agents/${fixture.id}/instructions-bundle/file`, "PUT", { + path: "AGENTS.md", content: "Durable original", baseRevisionId: null, + }); + return { id: String(fixture.id), seeded: index < 2 }; + })); + storageProxy!.holdUploads(); + const writes = Promise.all(fixtureAgents.map(fixture => fetch(`${b.origin}/api/agents/${fixture.id}/instructions-bundle/file`, { + method: "PUT", headers: { "content-type": "application/json", origin: b.origin }, + body: JSON.stringify({ path: "failed.txt", content: "Must never become authoritative", baseHash: null }), + signal: AbortSignal.timeout(40_000), + }).then(async response => { await response.arrayBuffer(); return response.status; }))); + // Keep a rejection handler attached even if an earlier assertion fails. + void writes.catch(() => undefined); + try { + const deadline = Date.now() + 10_000; + while (storageProxy!.heldUploads < 2 && Date.now() < deadline) await delay(25); + assert.equal(storageProxy!.heldUploads, 2, "Both file-transfer connections must be occupied before probing ordinary HTTP"); + const began = performance.now(); + assert.equal((await api(b.origin, `/issues/${task.id}`)).id, task.id); + report.storageOutageReadMs = Math.round(performance.now() - began); + assert.ok(Number(report.storageOutageReadMs) < 2_000); + storageProxy!.rejectUploads(); + assert.ok((await writes).every(status => status >= 500)); + } finally { + storageProxy!.rejectUploads(); + await writes.catch(() => undefined); + storageProxy!.resumeUploads(); + } + for (const fixture of fixtureAgents) { + const route = `/agents/${fixture.id}/instructions-bundle/file`; + const response = await fetch(`${b.origin}/api${route}?path=failed.txt`, { signal: AbortSignal.timeout(10_000) }); + const body = await response.text(); + assert.equal(response.status, 404, `Failed-write read returned ${response.status}: ${body.slice(0, 300)}`); + if (fixture.seeded) assert.equal((await api(b.origin, `${route}?path=AGENTS.md`)).content, "Durable original"); + } + }); + await b.stop(); await rm(b.home, { recursive: true, force: true }); + const c = await start("C", false); proxy.routeTo(c.origin); + await check("Fresh C restores files after both homes are deleted, even with adoption opt-in disabled", async () => { + assert.equal((await api(c.origin, `${fileRoute}?path=reference.txt`)).content, content); + assert.equal((await api(c.origin, `/issues/${task.id}`)).id, task.id); + }); +} catch (error) { + report.error = error instanceof Error ? error.message : String(error); +} finally { + probing = false; + await probes; + const cleanupErrors: string[] = []; + const clean = async (run: () => Promise) => { + try { await run(); } catch (error) { cleanupErrors.push(error instanceof Error ? error.message : String(error)); } + }; + if (browser) await clean(() => browser!.close()); + for (const server of processes) await clean(() => server.stop()); + if (proxy) await clean(() => proxy!.close()); + if (storageProxy) await clean(() => storageProxy!.close()); + if (database) await clean(() => database!.cleanup()); + if (s3 && bucketCreated) await clean(async () => { + while (true) { + const objects = await s3!.send(new ListObjectsV2Command({ Bucket: bucket })); + if (!objects.Contents?.length) break; + const deleted = await s3!.send(new DeleteObjectsCommand({ Bucket: bucket, + Delete: { Objects: objects.Contents.map((object: { Key: string }) => ({ Key: object.Key })) } })); + if (deleted.Errors?.length) throw new Error("Owned S3 bucket cleanup could not delete all objects"); + } + await s3!.send(new DeleteBucketCommand({ Bucket: bucket })); + }); + s3?.destroy(); + report.cleanupConfirmed = cleanupErrors.length === 0; + if (cleanupErrors.length) report.cleanupErrors = cleanupErrors; + report.passed = !report.error && checks.length > 0 && checks.every(item => item.passed) && report.cleanupConfirmed; + await saveReport(); + console.info(JSON.stringify({ passed: report.passed, checks, evidence: reportPath, cleanupConfirmed: report.cleanupConfirmed })); +} +if (!report.passed) process.exitCode = 1; diff --git a/tests/runner-e2e/rolling-deployment-storage-proxy.ts b/tests/runner-e2e/rolling-deployment-storage-proxy.ts new file mode 100644 index 00000000000..9761bfa2285 --- /dev/null +++ b/tests/runner-e2e/rolling-deployment-storage-proxy.ts @@ -0,0 +1,58 @@ +import { Agent, createServer, request, type ServerResponse } from "node:http"; +import { once } from "node:events"; +import type { Socket } from "node:net"; + +/** Transparent, non-retrying S3 fault boundary. Preserve signed Host headers. */ +export async function startRollingStorageProxy(endpoint: string) { + const target = new URL(endpoint); + if (target.protocol !== "http:" || !["127.0.0.1", "localhost", "[::1]"].includes(target.hostname)) { + throw new Error("Storage fault injection requires a loopback service"); + } + let uploads: "pass" | "hold" | "reject" = "pass"; + const held = new Map void>(); + const sockets = new Set(); + const agent = new Agent({ keepAlive: true }); + const server = createServer((incoming, outgoing) => { + if (incoming.method === "PUT" && uploads === "reject") { + incoming.resume(); + outgoing.writeHead(503).end(); + return; + } + const upstream = request(target, { method: incoming.method, path: incoming.url, headers: incoming.headers, agent }, response => { + response.on("error", () => outgoing.destroy()); + const release = (reject: boolean) => { + held.delete(outgoing); + if (reject) { response.resume(); outgoing.writeHead(503).end(); } + else { outgoing.writeHead(response.statusCode ?? 502, response.headers); response.pipe(outgoing); } + }; + if (incoming.method === "PUT" && uploads === "hold") { + held.set(outgoing, release); + outgoing.on("close", () => { held.delete(outgoing); response.destroy(); }); + } else release(false); + }); + upstream.on("error", () => { if (!outgoing.headersSent) outgoing.writeHead(502).end(); else outgoing.destroy(); }); + incoming.on("aborted", () => upstream.destroy()); + outgoing.on("close", () => { if (!outgoing.writableFinished) upstream.destroy(); }); + incoming.pipe(upstream); + }); + server.on("connection", socket => { sockets.add(socket); socket.on("close", () => sockets.delete(socket)); }); + server.listen(0, "127.0.0.1"); await once(server, "listening"); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Storage proxy did not bind"); + return { + endpoint: `http://127.0.0.1:${address.port}`, + get heldUploads() { return held.size; }, + holdUploads() { uploads = "hold"; }, + rejectUploads() { + uploads = "reject"; + for (const release of held.values()) release(true); + held.clear(); + }, + resumeUploads() { uploads = "pass"; for (const release of held.values()) release(false); }, + async close() { + const closed = once(server, "close"); server.close(); agent.destroy(); + for (const socket of sockets) socket.destroy(); + await closed; + }, + }; +}