From 59822467710a7c968347e3d8fe905359a348df2a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:17:01 -0500 Subject: [PATCH 01/41] fix(server): schedule custom-image setup expiry from committed changes (#17443) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages AI agent work and its execution environments. > - Interactive image setup sessions must expire even when users leave. > - The server currently scans these sessions on every maintenance tick. > - Empty scans keep the instance and database active. > - Existing session rows already record status and expiry. > - This pull request uses transaction signals and the shared scheduler for those rows. > - Empty and completed setup history no longer create recurring cleanup queries. ## Linked Issues or Issue Description **What happened?** The server checks custom-image setup expiry every heartbeat maintenance tick, even when no active session exists. **What did you expect?** Session writes should update the next cleanup deadline. Existing provider cancellation and cleanup ownership must remain intact. Empty state should stay quiet. **Steps to reproduce** Run an instance with heartbeat scheduling enabled and no custom-image setup sessions. Observe repeated setup-expiry reads. Refs #17145. This is the first PR in the recurring server sleep stack. Its base is master. The full requested scope and verification contract are recorded in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. Follow-up PRs target their predecessor and each own a separate reason for recurring work. ## What Changed - Carry the historical cleanup retry-observation fixture repair and its existing external test-database hook at the stack foundation. Runtime scheduling is unchanged. - Carry the previously verified login-cleanup fixture settlement correction at the stack foundation so every intermediate CI run receives it. This is test-only maintenance; custom-image expiry behavior is unchanged. - Add a custom-image expiry worker to the existing delivery coordinator. Remove its maintenance-tick scan. - Signal initial creation, provider refresh, cancellation/failure, and promotion inside their owning transaction. - Keep the active-status predicate, earliest saved expiry, and provider behavior in the custom-image service. Shared code gains only a queue topic. - Preserve startup recovery, bounded retries, concurrent write generations, and drain deferral. No tables or external alarm changes. - Add PostgreSQL lifecycle tests and repair this suite's database cleanup callback. A dedicated external test URL permits disposable PostgreSQL when the host cannot start another embedded cluster. ## Verification Current head: `3507c6be1e054fdd93353d5dcf77d4899b44518f`. This remains a single-duty PR stacked on master. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/3507c6be1e054fdd93353d5dcf77d4899b44518f/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover supported service producers sharing the DB owner. Independent SQL and another process require the existing restart/reconciliation contract. - Failed provider cancellation retains the existing failed-session and historical-cleanup behavior. This PR changes scheduling, not teardown authority. - Active rows without an expiry preserve the existing behavior; no deadline is invented. Cloud sleep eligibility remains conservative. - The worker remains in the existing heartbeat-enabled startup path. It does not change disabled-scheduler behavior or prove deployed suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Review follow-up: Added a real producer commit during a paused deadline read. Updated startup assertions for coordinator ownership and removed added code comments. The three affected suites pass (102 tests). The separate login-cleanup suite also passes locally (16 tests); its prior CI consume test timed out, and the fresh complete CI run passed. External PostgreSQL tests now use the repository guard for disposable databases ending in `_test`. --------- Co-authored-by: Paperclip --- doc/idle-environment-cleanup.md | 45 ++++ doc/idle-login-cleanup.md | 17 ++ ...-instance-sleep-and-scheduler-inventory.md | 9 +- ...2026-10-24-recurring-server-sleep-stack.md | 63 ++++++ ...-10-26-retained-workspace-slot-fairness.md | 53 +++++ packages/db/src/migrations/meta/_journal.json | 2 +- .../tests/native_provider_backend.rs | 2 +- .../custom-image-cleanup-work.test.ts | 75 +++++++ .../environment-cleanup-work.test.ts | 16 +- .../environment-custom-images-service.test.ts | 199 +++++++++++++++++- .../src/__tests__/login-cleanup-work.test.ts | 8 +- .../server-startup-feedback-export.test.ts | 12 +- server/src/index.ts | 22 +- .../src/services/custom-image-cleanup-work.ts | 35 +++ .../services/delivery-work-notifications.ts | 1 + .../src/services/environment-custom-images.ts | 33 ++- .../src/services/workspace-recovery.test.ts | 22 +- server/src/services/workspace-recovery.ts | 16 +- 18 files changed, 569 insertions(+), 61 deletions(-) create mode 100644 doc/plans/2026-10-24-recurring-server-sleep-stack.md create mode 100644 doc/plans/2026-10-26-retained-workspace-slot-fairness.md create mode 100644 server/src/__tests__/custom-image-cleanup-work.test.ts create mode 100644 server/src/services/custom-image-cleanup-work.ts diff --git a/doc/idle-environment-cleanup.md b/doc/idle-environment-cleanup.md index 9070703ef8c..a8d2ecdff02 100644 --- a/doc/idle-environment-cleanup.md +++ b/doc/idle-environment-cleanup.md @@ -101,3 +101,48 @@ moves here. Other maintenance duties, external probes, and connected transports still affect idle traffic. No Cloud alarm coverage or host-sleep exemption changes. These tests prove the changed workers become quiet and recover; they do not prove deployed suspension or fleet savings. + +## Custom-image setup expiry + +Setup expiry uses a separate `custom-image-expiry` worker. The custom-image +service owns the active-status predicate, the earliest saved `expiresAt`, and +provider cancellation. Startup registers the worker in the existing enabled +heartbeat housekeeping path; the maintenance tick no longer calls its sweep. + +All supported setup producers live in `environment-custom-images.ts`: initial +creation, provider refresh, cancellation/failure status, and promotion. They +register intent in their owning transaction. A provider refresh can move the +deadline earlier; a terminal transition removes it. Historical verification +metadata does not change active expiry eligibility and does not need this signal. +Environment deletion can only remove obligations; an already scheduled pass +observes the removal. Independent SQL and other DB roots are outside the existing +process-local notification contract. + +No active dated session means no recurring reads or provider calls. Overdue +batches and failures retain a bounded retry, while first writes and failed +reconciliation survive drain. A missing expiry retains the existing behavior +(no inferred expiration); sleep safety still treats unsupported live resources +conservatively. Provider cancellation, failure state, and historical resource +verification keep their existing semantics. No Cloud alarm coverage changes. + +## Historical retry test observation + +The historical verification worker intentionally does not delay server readiness. +Its provider sweep finishes before the following database pending-state read +registers a retry deadline. The regression test now waits for that future +deadline before inspecting it; sweep completion alone is not scheduler settlement. +The runtime worker, retry interval, and readiness contract are unchanged. + +A controlled 150ms delay before the real pending-state query reproduced the +current CI failure: the immediate assertion observed a null deadline. The +corrected assertion passed with the same delay. The temporary delay was removed, +and all 14 cleanup tests passed against an isolated PostgreSQL database. The +existing external test-database hook is carried at the stack foundation so +intermediate branches can use this same verification path. + +An initial attempt at the foundation could not start embedded PostgreSQL because +that earlier branch lacked the external fixture hook. A broader fixture wait for +all idle holds to clear then failed three cases that legitimately retain pending +cleanup obligations. That wait was removed. Only the historical deadline assertion +changed; its requirement is now a deadline strictly in the future. The complete +suite passes with the original readiness and idle-hold behavior preserved. diff --git a/doc/idle-login-cleanup.md b/doc/idle-login-cleanup.md index 813cf34eb85..5bc3ebf62ac 100644 --- a/doc/idle-login-cleanup.md +++ b/doc/idle-login-cleanup.md @@ -95,3 +95,20 @@ cover the surrounding login contracts. The fake-clock database fixture warms its connections and disables pool idle/lifetime expiry so time jumps test the worker rather than racing socket connection timers. Database suspension and reconnect behavior require separate qualification. + +## CI fixture settlement across the migration stack + +The setup-token consume fixture can finish its initial fake-clock jump while +cleanup is still performing database work. A write during that pass can request +another generation whose coalesced deadline lies beyond the jump. Waiting for +zero tracked work without advancing to that registered deadline then times out, +even though cleanup has a valid wake and can finish. + +The existing fixture repair from `59f8c8b3e5` is now carried at the earliest stack +branch so every intermediate PR receives it. The helper advances to registered +deadlines only while work remains tracked, preserving its existing five-second +observation limit and all production behavior. All 16 login-cleanup cases passed +on this branch against a dedicated PostgreSQL database; the prior later-branch +verification and failed CI history remain documented in the GitHub review-check +migration audit. No runtime change or weakened quiet assertion is included. +Evidence: `/tmp/sleep-login-settlement-owning-tests.log` and its zero status file. diff --git a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md index 2e5bb5c9ca5..b92d7520ec0 100644 --- a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md +++ b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md @@ -634,10 +634,11 @@ Longer polling intervals alone do not complete an item. [idle-login-cleanup.md](../idle-login-cleanup.md) for producer coverage, verification and remaining provider-orphan boundaries. -- [ ] **JOB-19 — Custom-image setup expiry.** Current: **30s** active-session - expiry scans. Change: register `expiresAt` and cleanup attempts. **Complete - when:** no sessions means no SQL and expired provider resources are released. - Source: [environment-custom-images.ts](../../server/src/services/environment-custom-images.ts). +- [x] **JOB-19 — Custom-image setup expiry.** Uses committed setup-session + changes and the earliest existing active-session expiry. Startup reconstructs + deadlines; empty and terminal history disarm. Failed reconciliation and first + writes retain their wake through drain. Provider teardown and historical cleanup + remain with the existing services. See [idle-environment-cleanup.md](../idle-environment-cleanup.md#custom-image-setup-expiry). - [ ] **JOB-20 — Tool connection health.** Current: **30s** DB inventory; provider probes are conditional. Change: check on use/configuration change, diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md new file mode 100644 index 00000000000..50f73222265 --- /dev/null +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -0,0 +1,63 @@ +# Recurring server work migration stack + +Baseline: `785d54636e1c8513ee5503d602daf6a5ee1970d6` (2026-10-24). + +The requested outcome is a reviewed PR stack for every remaining recurring server +work item below. A PR owns one reason for background activity. Split compound +items into separate PRs as their producer and lifecycle audits identify distinct +owners. Each PR includes behavior, producer audit, tests, and documentation. +Do not mark an item done from a timer removal or a green check alone. + +## Scope and completion ledger + +| ID | Original checklist duty | Required slices / disposition | PR / evidence | +| --- | --- | --- | --- | +| 01 | Custom-image setup expiry | Session writes, existing expiry, startup/retry/drain | In progress | +| 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | +| 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | +| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | +| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | +| 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Pending | +| 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | +| 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | +| 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | +| 11 | Status-card evaluation | Settings/card mutations and evaluation deadlines | Pending | +| 12 | Plugin scheduled jobs | Registration/update/removal and actual job deadlines | Pending | +| 13 | Routine, agent-timer and issue-monitor scheduling | Remove periodic wake backstops after complete producer coverage; routine execution-status reconciliation separately | Pending | +| 14 | Skill-version retention and native-harness backup expiration | Separate retention owners and existing retention guarantees | Pending | +| 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Pending | +| 16 | Aggregate maintenance tick | Retire enabled and disabled branches after all constituent owners migrate | Pending | + +## Stack and verification contract + +The first PR targets master. Each following PR targets its predecessor branch so +its diff shows only its own reason. Parent changes must be propagated before +final verification. Record the exact checked head and base for every PR. All PRs +must finish with passing checks, fresh 5/5 Greptile review, and no unresolved +findings. This request does not authorize merging the stack. + +Reuse the delivery coordinator, transaction notifications, and shared scheduler. +Existing durable rows own work; no table per loop. Keep domain/provider behavior +in its service and wiring in startup. A write signal is not durable truth. Preserve +outer commit/rollback and uncertain-outcome reconciliation. Audit every supported +producer, including alternate routes and recovery writers. + +For each slice verify applicable empty/configured-idle/history quiet states, +first real producer, actual deadlines, rollback/outer commit, restart, failures, +changes during execution, drain/resume, and shutdown. Use real PostgreSQL where +transaction boundaries matter and controlled providers where external behavior +is under test. Record failures and subsequent repair evidence. + +External-source reconciliation may remain only for a concrete supported obligation +with a defined cadence and stopping condition, using the shared scheduler. Never +remove the only missed-event backstop without replacing its recovery contract. + +## Boundaries + +Cloud sleep eligibility and durable alarm coverage remain conservative. An +in-process deadline cannot wake a stopped host. Browser traffic, persistent +connector transport architecture, probes, telemetry/exporter qualification, and +end-to-end provider suspension are outside this requested recurring-server stack; +none are claimed solved by these PRs. Required active-operation lease renewals and +request timeouts remain with their owners. diff --git a/doc/plans/2026-10-26-retained-workspace-slot-fairness.md b/doc/plans/2026-10-26-retained-workspace-slot-fairness.md new file mode 100644 index 00000000000..682bf991baa --- /dev/null +++ b/doc/plans/2026-10-26-retained-workspace-slot-fairness.md @@ -0,0 +1,53 @@ +# Retained workspace recovery: fill a free slot across the cursor + +This is a shared CI repair carried at the sleep-stack root. It fixes the existing +retained-workspace owner; it does not add another background duty or migrate the +remaining run/task recovery inventory. No PR is merged by this repair. + +## Failure and ownership + +The retained-workspace worker permits two physical exports. A held export stays +in the durable unfinished candidates, even though this process already owns it. +Discovery paginates by recovery-action ID and wraps only when its query returns +no rows. A page containing only a held export was nonempty, but its only row was +skipped in memory. Newly committed work before the cursor therefore waited for +another 15-second recovery pass despite a free slot. + +The CI failure at PR #17446, run 40423925338 / job 121925042581, leaves the third +export `scheduled` at the existing 20-second assertion. The older #17450 failure +has the same observable result. A deterministic real-PostgreSQL/filesystem +regression reproduces it by starting the highest action ID, holding its provider +resume, completing a middle action, then requesting the lowest action. The +before-fix test failed at that same third-export assertion. + +## Change and preserved boundaries + +The existing candidate query accepts the locally in-flight run IDs and excludes +them before pagination on both unfinished and passive discovery paths. The +coordinator reads this set after its existing two-slot guard, so the exclusion +contains at most one run for an admitted discovery pass. An empty page can now +wrap in the same pass and find new work before the cursor. The in-memory duplicate +guard remains to handle changes while a query is running. + +The retained-workspace module owns candidate selection. Its callers, SQL source +eligibility, physical ownership lock, retry policy, concurrency limit, completion +callbacks, and shutdown joins retain their existing contracts. Other-process +locks are still handled by the existing bounded cursor rotation. No new timer, +queue, table, provider branch, or retry authority is added. Held work remains +tracked as active and is not treated as settled or safe to sleep through. + +## Verification + +At the integrated pre-repair tip, the deterministic test failed with `scheduled` +instead of `complete`. After the repair, that regression and the existing +other-worker physical-lock rotation test both pass with real files and database +state. The free-slot test keeps the original 20-second completion bound, verifies +preserved files, no extra agent run, at most two concurrent resume operations, +and shutdown waiting for the held export before its final completion. + +At the stack root, all 56 retained-workspace recovery cases pass on a dedicated +migrated PostgreSQL test database. The full suite also exercises retries, +exhaustion, stop verification, cleanup authorization, source bindings and saved +file recovery. Integrated workspace typecheck/build and fresh remote checks must +be recorded in the PR after parent propagation. This is not a full repository or +deployed-sleep claim. diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index b6d2fe1f6bd..96c505941fd 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2656,4 +2656,4 @@ "breakpoints": true } ] -} \ No newline at end of file +} diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs index 2e99a0bb35c..eebe8fb4c16 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs @@ -1160,7 +1160,7 @@ fn acpx_transport_failure_retains_its_error_without_reopening_the_provider() { assert_eq!(snapshot["status"], "closed"); assert_eq!(snapshot["activeProviderTurnId"], Value::Null); assert!(executor.poll_events().unwrap().is_empty()); - executor.shutdown().unwrap(); + shutdown_recovered_acpx_fixture(&mut executor, &directory); fs::remove_dir_all(directory).unwrap(); } diff --git a/server/src/__tests__/custom-image-cleanup-work.test.ts b/server/src/__tests__/custom-image-cleanup-work.test.ts new file mode 100644 index 00000000000..6a6b5b021d9 --- /dev/null +++ b/server/src/__tests__/custom-image-cleanup-work.test.ts @@ -0,0 +1,75 @@ +import { createDb } from "@paperclipai/db"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { registerCustomImageCleanupWork } from "../services/custom-image-cleanup-work.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; + +describe("custom-image expiry deadline precision", () => { + let db: ReturnType; + let coordinator: ReturnType; + beforeEach(() => { vi.useFakeTimers(); vi.setSystemTime(new Date("2026-10-26T12:00:00Z")); + db = createDb("postgres://localhost:1/deadline_precision_test"); }); + afterEach(async () => { await coordinator?.stop(); await db.$client.end({ timeout: 1 }); vi.useRealTimers(); }); + + async function start(delay: number, failFirst = false) { + let due: number | null = Date.now() + delay; + let expired = 0; + const errors: unknown[] = []; + const cleanup = vi.fn(async () => { + if (due !== null && due <= Date.now()) { expired++; due = null; } + return { scanned: 0, timedOut: 0, failed: 0 }; + }); + if (failFirst) cleanup.mockRejectedValueOnce(new Error("temporary database failure")); + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, + onError: error => errors.push(error) }); + const worker = registerCustomImageCleanupWork(coordinator, { + cleanupExpiredSetupSessions: cleanup, + nextSetupCleanupAt: async () => due, + }, () => true); + await worker.ready; + await vi.advanceTimersByTimeAsync(0); + return { cleanup, errors, worker, expired: () => expired, move: (at: number) => { due = at; worker.wake(); } }; + } + + it("expires at a saved five-second startup deadline and then becomes quiet", async () => { + const due = Date.now() + 5_000; + const work = await start(5_000); + expect(coordinator.nextWakeAt()).toBe(due); + await vi.advanceTimersByTimeAsync(4_999); + expect(work.expired()).toBe(0); + await vi.advanceTimersByTimeAsync(1); + expect(work.expired()).toBe(1); + expect(coordinator.nextWakeAt()).toBeNull(); + const calls = work.cleanup.mock.calls.length; + await vi.advanceTimersByTimeAsync(120_000); + expect(work.cleanup).toHaveBeenCalledTimes(calls); + }); + + it("replaces a future deadline with an earlier wake inside the retry window", async () => { + const work = await start(60_000); + await vi.advanceTimersByTimeAsync(1_000); + const due = Date.now() + 5_000; + work.move(due); + await vi.advanceTimersByTimeAsync(0); + expect(coordinator.nextWakeAt()).toBe(due); + await vi.advanceTimersByTimeAsync(4_999); + expect(work.expired()).toBe(0); + await vi.advanceTimersByTimeAsync(1); + expect(work.expired()).toBe(1); + }); + + it("retains failure retry spacing even when another wake arrives", async () => { + const retryAt = Date.now() + 30_000; + const work = await start(5_000, true); + expect(work.errors).toHaveLength(1); + work.worker.wake(); + await vi.advanceTimersByTimeAsync(0); + expect(coordinator.nextWakeAt()).toBe(retryAt); + await vi.advanceTimersByTimeAsync(29_999); + expect(work.cleanup).toHaveBeenCalledTimes(1); + expect(work.expired()).toBe(0); + await vi.advanceTimersByTimeAsync(1); + expect(work.expired()).toBe(1); + expect(work.cleanup).toHaveBeenCalledTimes(2); + expect(coordinator.nextWakeAt()).toBeNull(); + }); +}); diff --git a/server/src/__tests__/environment-cleanup-work.test.ts b/server/src/__tests__/environment-cleanup-work.test.ts index 7f6b1d64c12..6a251dabdd0 100644 --- a/server/src/__tests__/environment-cleanup-work.test.ts +++ b/server/src/__tests__/environment-cleanup-work.test.ts @@ -17,17 +17,20 @@ import { historicalCleanupVerificationService } from "../services/historical-cle import { heartbeatService } from "../services/heartbeat.js"; import { environmentRuntimeService } from "../services/environment-runtime.js"; -const support = await getEmbeddedPostgresTestSupport(); +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_ENVIRONMENT_CLEANUP_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); describe.skipIf(!support.supported)("environment cleanup scheduling", () => { - let database: Awaited>, db: ReturnType; + let database: Awaited> | undefined, db: ReturnType; let companyId: string, environmentId: string, draining: boolean, failing: boolean; let coordinator: ReturnType | undefined; let worker: ReturnType | undefined; let releases: ReturnType, inspections: ReturnType, reads: ReturnType; const errors: unknown[] = [], disposers: Array<() => void> = []; beforeAll(async () => { - database = await startEmbeddedPostgresTestDatabase("environment-cleanup-work-"); - db = createDb(database.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + if (!external) database = await startEmbeddedPostgresTestDatabase("environment-cleanup-work-"); + db = createDb(external ?? database!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); }, 90_000); beforeEach(async () => { @@ -230,7 +233,10 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => { it("keeps unknown historical resources scheduled until provider confirmation", async () => { const lease = await pending(); await environmentService(db).releaseLease(lease.id, "failed", { cleanupStatus: "success" }); inspections.mockResolvedValueOnce({ providerLeaseId: lease.providerLeaseId, state: "unknown" }); - await start(true); expect(coordinator!.nextWakeAt()).not.toBeNull(); + await start(true); + // The provider sweep finishes before its pending-state read registers the + // retry. Readiness intentionally does not wait for historical verification. + await vi.waitFor(() => expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now())); await settle(300_000); expect(inspections).toHaveBeenCalledTimes(2); await quiet(); }); }); diff --git a/server/src/__tests__/environment-custom-images-service.test.ts b/server/src/__tests__/environment-custom-images-service.test.ts index 6422cec2f91..d4cc612963d 100644 --- a/server/src/__tests__/environment-custom-images-service.test.ts +++ b/server/src/__tests__/environment-custom-images-service.test.ts @@ -1,6 +1,10 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { registerCustomImageCleanupWork } from "../services/custom-image-cleanup-work.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; import { randomUUID } from "node:crypto"; -import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; -import { desc, eq } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { desc, eq, sql } from "drizzle-orm"; import { activityLog, companies, @@ -27,7 +31,8 @@ import { startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalTestDatabase = resolveExternalTestDatabaseUrl("PAPERCLIP_CUSTOM_IMAGE_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalTestDatabase ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -153,9 +158,10 @@ describeEmbeddedPostgres("environmentCustomImageService", () => { let db!: ReturnType; beforeAll(async () => { + if (externalTestDatabase) { db = createDb(externalTestDatabase, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); return; } const started = await startEmbeddedPostgresTestDatabase("environment-custom-images"); - stopDb = started.stop; - db = createDb(started.connectionString); + stopDb = started.cleanup; + db = createDb(started.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); }); afterEach(async () => { @@ -167,6 +173,7 @@ describeEmbeddedPostgres("environmentCustomImageService", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await stopDb?.(); }); @@ -200,6 +207,176 @@ describeEmbeddedPostgres("environmentCustomImageService", () => { return { companyId, environmentId }; } + describe("expiry scheduling", () => { + let coordinator: ReturnType | undefined; + let service: ReturnType; + let provider: ReturnType; + let draining: boolean; + const errors: unknown[] = []; + + beforeEach(async () => { + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + draining = false; + errors.length = 0; + provider = createWorkerManager(); + service = environmentCustomImageService(db, { pluginWorkerManager: provider }); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); + coordinator = undefined; + vi.useRealTimers(); + vi.restoreAllMocks(); + expect(errors).toEqual([]); + }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerCustomImageCleanupWork(coordinator, service, () => !draining).ready; + } + async function settle(ms = 60_000) { + await vi.advanceTimersByTimeAsync(ms); + if (!draining) await vi.waitFor(() => expect(idleWorkSnapshot().active === 0 || coordinator!.nextWakeAt() !== null).toBe(true), { timeout: 5000 }); + } + async function quiet() { + await settle(); + expect(coordinator!.nextWakeAt()).toBeNull(); + const reads = vi.spyOn(db, "select"), calls = provider.call.mock.calls.length; + await settle(30 * 60_000); + expect(reads).not.toHaveBeenCalled(); + expect(provider.call).toHaveBeenCalledTimes(calls); + reads.mockRestore(); + } + async function create() { + const fixture = await seed(); + return service.startSetupSession({ environmentId: fixture.environmentId, + actor: { userId: "fixture-user" }, ttlSeconds: 3600 }); + } + const cancellations = () => provider.call.mock.calls.filter(([, method]) => method === "environmentCancelInteractiveSetup"); + + it("is quiet when empty, wakes from the real producer, expires once, and keeps terminal history quiet", async () => { + await start(); await quiet(); + const created = await create(); + await settle(); + expect(coordinator!.nextWakeAt()).toBe(new Date(created.session.expiresAt!).getTime()); + const reads = vi.spyOn(db, "select"); + await settle(30 * 60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + expect(cancellations()).toHaveLength(0); + await settle(30 * 60_000); + expect(cancellations()).toHaveLength(1); + expect(await service.getSessionById(created.session.id)).toMatchObject({ status: "timed_out" }); + await quiet(); + }); + + it("honors a committed expiry inside the retry window", async () => { + await start(); + const created = await create(); + await settle(); + const earlier = new Date(Date.now() + 5_000); + provider.call.mockResolvedValueOnce({ providerLeaseId: created.session.providerLeaseId, + status: "waiting_for_user", expiresAt: earlier.toISOString() } as never); + await service.refreshSetupSession({ sessionId: created.session.id }); + await vi.waitFor(() => expect(coordinator!.nextWakeAt()).toBe(earlier.getTime())); + expect(cancellations()).toHaveLength(0); + await settle(Math.max(0, earlier.getTime() - Date.now())); + await vi.waitFor(() => expect(cancellations()).toHaveLength(1)); + expect(await service.getSessionById(created.session.id)).toMatchObject({ status: "timed_out" }); + await quiet(); + }); + + it("replaces an expiry when the provider moves it earlier and disarms on user cancellation", async () => { + await start(); + const created = await create(); await settle(); + const earlier = new Date(Date.now() + 5 * 60_000); + provider.call.mockResolvedValueOnce({ providerLeaseId: created.session.providerLeaseId, + status: "waiting_for_user", expiresAt: earlier.toISOString() } as never); + await service.refreshSetupSession({ sessionId: created.session.id }); + await settle(); + expect(coordinator!.nextWakeAt()).toBe(earlier.getTime()); + await service.cancelSetupSession({ sessionId: created.session.id }); + await quiet(); + expect(cancellations()).toHaveLength(1); + }); + + it.each([false, true])("waits for the producer's outer transaction (rollback %s)", async rollback => { + const fixture = await seed(); await start(); await quiet(); + let release!: () => void, written!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const staged = new Promise(resolve => { written = resolve; }); + const writing = db.transaction(async tx => { + const scoped = environmentCustomImageService(tx as unknown as typeof db, { pluginWorkerManager: provider }); + await scoped.startSetupSession({ environmentId: fixture.environmentId, actor: { userId: "fixture-user" }, ttlSeconds: 1 }); + written(); await gate; + if (rollback) throw new Error("rollback fixture"); + }).catch(error => { if (!rollback) throw error; }); + await staged; + try { + await vi.advanceTimersByTimeAsync(60_000); + expect(cancellations()).toHaveLength(0); + } finally { release(); } + await writing; await settle(); + expect(cancellations()).toHaveLength(rollback ? 0 : 1); + await quiet(); + }); + + it("retains a producer commit that arrives during the deadline read", async () => { + await start(); const created = await create(); await settle(); + await coordinator!.stop(); + let release!: () => void, reading!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const entered = new Promise(resolve => { reading = resolve; }); + const readDeadline = service.nextSetupCleanupAt; + vi.spyOn(service, "nextSetupCleanupAt").mockImplementationOnce(async () => { + const stale = await readDeadline(); + reading(); await gate; + return stale; + }); + const starting = start(); + await entered; + const earlier = new Date(Date.now() + 5 * 60_000); + try { + provider.call.mockResolvedValueOnce({ providerLeaseId: created.session.providerLeaseId, + status: "waiting_for_user", expiresAt: earlier.toISOString() } as never); + await service.refreshSetupSession({ sessionId: created.session.id }); + } finally { release(); } + await starting; await settle(); + await vi.waitFor(() => expect(coordinator!.nextWakeAt()).toBe(earlier.getTime())); + await settle(5 * 60_000); + await vi.waitFor(() => expect(cancellations()).toHaveLength(1)); + await quiet(); + }); + + it("reconstructs a saved expiry after restart", async () => { + await start(); const created = await create(); await settle(); + await coordinator!.stop(); + await start(); + expect(coordinator!.nextWakeAt()).toBe(new Date(created.session.expiresAt!).getTime()); + await settle(60 * 60_000); + expect(cancellations()).toHaveLength(1); await quiet(); + }); + + it.each(["startup", "first_write", "failed_pass"])("retains %s work through drain without passive SQL", async mode => { + if (mode === "startup") draining = true; + await start(); + if (mode === "first_write") { await quiet(); draining = true; } + await create(); + if (mode === "failed_pass") { + vi.spyOn(service, "nextSetupCleanupAt").mockRejectedValueOnce(new Error("database unavailable")); + await settle(30_000); + await vi.waitFor(() => expect(errors).toHaveLength(1)); errors.length = 0; + draining = true; + } + await settle(); + const reads = vi.spyOn(db, "select"); + await settle(2 * 60 * 60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + expect(cancellations()).toHaveLength(0); + draining = false; await settle(); + expect(cancellations()).toHaveLength(1); await quiet(); + }); + }); + it.each(["supported", "old_worker", "changed_config"])("inspects historical setup read-only with exact config and capability gates: %s", async mode => { const fixture = await seed(); const worker = createWorkerManager({ workerMethods: [...ALL_CUSTOM_IMAGE_WORKER_METHODS, @@ -838,9 +1015,10 @@ describeEmbeddedPostgres("environmentCustomImageService reconciliation", () => { let db!: ReturnType; beforeAll(async () => { + if (externalTestDatabase) { db = createDb(externalTestDatabase, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); return; } const started = await startEmbeddedPostgresTestDatabase("environment-custom-images-reconcile"); - stopDb = started.stop; - db = createDb(started.connectionString); + stopDb = started.cleanup; + db = createDb(started.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); }); afterEach(async () => { @@ -852,6 +1030,7 @@ describeEmbeddedPostgres("environmentCustomImageService reconciliation", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await stopDb?.(); }); @@ -1145,9 +1324,10 @@ describeEmbeddedPostgres("environmentCustomImageService relink", () => { let db!: ReturnType; beforeAll(async () => { + if (externalTestDatabase) { db = createDb(externalTestDatabase, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); return; } const started = await startEmbeddedPostgresTestDatabase("environment-custom-images-relink"); - stopDb = started.stop; - db = createDb(started.connectionString); + stopDb = started.cleanup; + db = createDb(started.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); }); afterEach(async () => { @@ -1160,6 +1340,7 @@ describeEmbeddedPostgres("environmentCustomImageService relink", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await stopDb?.(); }); diff --git a/server/src/__tests__/login-cleanup-work.test.ts b/server/src/__tests__/login-cleanup-work.test.ts index c21d04c97d3..081fc6eece6 100644 --- a/server/src/__tests__/login-cleanup-work.test.ts +++ b/server/src/__tests__/login-cleanup-work.test.ts @@ -116,7 +116,13 @@ describe.skipIf(!support.supported)("durable login cleanup", () => { }); async function settle(ms = 30_000) { await vi.advanceTimersByTimeAsync(ms); - await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0), { timeout: 5000 }); + await vi.waitFor(async () => { + if (idleWorkSnapshot().active > 0) { + const deadlines = coordinators.flatMap(c => c.nextWakeAt() === null ? [] : [c.nextWakeAt()!]); + if (deadlines.length) await vi.advanceTimersByTimeAsync(Math.max(0, Math.min(...deadlines) - Date.now())); + } + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); await vi.advanceTimersByTimeAsync(1); await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0), { timeout: 5000 }); expect(errors).toEqual([]); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index db22ce7b2c2..81c689366c3 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -129,6 +129,7 @@ const { }; const issueThreadInteractionServiceFactoryMock = vi.fn(() => issueThreadInteractionServiceMock); const environmentCustomImagesServiceMock = { + nextSetupCleanupAt: vi.fn(async (): Promise => null), cleanupExpiredSetupSessions: vi.fn(async () => ({ scanned: 0, timedOut: 0, failed: 0 })), }; const environmentCustomImagesServiceFactoryMock = vi.fn(() => environmentCustomImagesServiceMock); @@ -843,7 +844,8 @@ describe("startServer feedback export wiring", () => { expect(issueThreadInteractionServiceMock.sweepMergedPullRequestConfirmations).toHaveBeenCalledTimes(1); expect(executionWorkspaceServiceMock.sweepTerminalWorkspaces).toHaveBeenCalledTimes(1); expect(routineServiceMock.tickScheduledTriggers).toHaveBeenCalledTimes(1); - expect(environmentCustomImagesServiceMock.cleanupExpiredSetupSessions).toHaveBeenCalledTimes(2); + expect(environmentCustomImagesServiceMock.cleanupExpiredSetupSessions).toHaveBeenCalledTimes(1); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.customImageExpiry)).toBe(true); } finally { setIntervalSpy.mockRestore(); } @@ -960,10 +962,10 @@ describe("startServer feedback export wiring", () => { try { await expect(startServer()).resolves.toBeDefined(); await settleStartupHousekeeping(); - expect(vi.mocked(logger.error)).toHaveBeenCalledWith( - expect.objectContaining({ startupStep: "custom_image_setup_cleanup" }), - "startup housekeeping step failed", - ); + const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.customImageExpiry); + expect(registration).toBeDefined(); + expect(await registration![1].hasPending!()).toBe(true); + expect(environmentCustomImagesServiceMock.nextSetupCleanupAt).not.toHaveBeenCalled(); // The scheduler starts only after every housekeeping step settled. expect(heartbeatIntervals).toHaveLength(1); } finally { diff --git a/server/src/index.ts b/server/src/index.ts index 23b13375283..411dda3f87f 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerCustomImageCleanupWork } from "./services/custom-image-cleanup-work.js"; import { registerWorkspaceRecoveryWork } from "./services/workspace-recovery-work.js"; import { registerEnvironmentCleanupWork } from "./services/environment-cleanup-work.js"; import { loginCleanupSchedule } from "./services/login-cleanup-state.js"; @@ -1790,12 +1791,9 @@ async function startServerWithDatabaseTeardown( work().catch((err: unknown) => { logger.error({ err, startupStep: name }, "startup housekeeping step failed"); }); - await step("custom_image_setup_cleanup", async () => { - const setupCleanup = await environmentCustomImages.cleanupExpiredSetupSessions(); - if (setupCleanup.timedOut > 0 || setupCleanup.failed > 0) { - logger.warn({ ...setupCleanup }, "startup environment customImage setup cleanup changed sessions"); - } - }); + await step("custom_image_setup_cleanup", () => registerCustomImageCleanupWork(deliveryWork, environmentCustomImages, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped, + config.heartbeatSchedulerIntervalMs).ready); await step("tool_connection_health", async () => { const toolHealthSweep = await tools.sweepConnectionHealth(); if (toolHealthSweep.failed > 0) { @@ -1877,18 +1875,6 @@ async function startServerWithDatabaseTeardown( logger.error({ err }, "status-card scheduler tick failed"); })); - if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; - trackHeartbeatSchedulerWork(environmentCustomImages - .cleanupExpiredSetupSessions() - .then((result) => { - if (result.timedOut > 0 || result.failed > 0) { - logger.warn({ ...result }, "environment customImage setup cleanup changed sessions"); - } - }) - .catch((err) => { - logger.error({ err }, "environment customImage setup cleanup failed"); - })); - if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; trackHeartbeatSchedulerWork(tools .sweepConnectionHealth() diff --git a/server/src/services/custom-image-cleanup-work.ts b/server/src/services/custom-image-cleanup-work.ts new file mode 100644 index 00000000000..9ecd4e73612 --- /dev/null +++ b/server/src/services/custom-image-cleanup-work.ts @@ -0,0 +1,35 @@ +import { logger } from "../middleware/logger.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import type { environmentCustomImageService } from "./environment-custom-images.js"; + +export function registerCustomImageCleanupWork( + coordinator: ReturnType, + service: Pick, "cleanupExpiredSetupSessions" | "nextSetupCleanupAt">, + canRun: () => boolean, + retryMs = 30_000, +) { + let requested = 1, reconciled = 0; + let next: number | null = null; + let earliest = 0; + return coordinator.register(DELIVERY_QUEUES.customImageExpiry, { + retryMs, + pausePassiveMaintenance: true, + onSettled: () => { requested++; }, + run: async () => { + if (!canRun()) { requested++; return; } + if (Date.now() < earliest) { next = earliest; return; } + const generation = ++requested; + earliest = Date.now() + retryMs; + const result = await service.cleanupExpiredSetupSessions(); + if (result.timedOut > 0 || result.failed > 0) logger.warn(result, "environment customImage setup cleanup changed sessions"); + const deadline = await service.nextSetupCleanupAt(); + // Back off failed or still-overdue work, not a successfully read future deadline. + if (deadline === null || deadline > Date.now()) earliest = 0; + next = deadline === null ? null : Math.max(earliest, deadline); + reconciled = generation; + }, + hasPending: async () => requested !== reconciled, + nextRunAt: () => next, + }); +} diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 66f14532600..f6d50c71930 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + customImageExpiry: "custom-image-expiry", historicalCleanup: "historical-cleanup", adapterLoginCleanup: "adapter-login-cleanup", setupTokenCleanup: "setup-token-cleanup", diff --git a/server/src/services/environment-custom-images.ts b/server/src/services/environment-custom-images.ts index 324065e84d5..9ff4fd1c1df 100644 --- a/server/src/services/environment-custom-images.ts +++ b/server/src/services/environment-custom-images.ts @@ -1,7 +1,8 @@ -import { writeEnvironmentCleanup, notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js"; +import { notifyEnvironmentCleanup } from "./environment-cleanup-notifications.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { randomUUID } from "node:crypto"; import { withoutCleanupVerification } from "./cleanup-verification-proof.js"; -import { and, desc, eq, inArray, lte } from "drizzle-orm"; +import { and, desc, eq, inArray, lte, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { environmentCustomImageSetupSessions, @@ -290,6 +291,14 @@ export function environmentCustomImageService( const environments = environmentService(db); const secrets = secretService(db); + function writeSetup(write: (tx: Db) => PromiseLike): Promise { + return db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.customImageExpiry); + await notifyEnvironmentCleanup(tx); + return write(tx as unknown as Db); + }); + } + async function getTemplateById(id: string): Promise { const row = await db .select() @@ -595,7 +604,7 @@ export function environmentCustomImageService( ): Promise { const status = normalizePersistedStatus(providerSession.status, fallbackStatus); const now = new Date(); - const row = await writeEnvironmentCleanup(db, tx => tx + const row = await writeSetup(tx => tx .update(environmentCustomImageSetupSessions) .set({ providerLeaseId: providerSession.providerLeaseId ?? null, @@ -620,7 +629,7 @@ export function environmentCustomImageService( metadata?: Record | null; }): Promise { const now = new Date(); - const row = await writeEnvironmentCleanup(db, tx => tx + const row = await writeSetup(tx => tx .update(environmentCustomImageSetupSessions) .set({ status: input.status, @@ -834,7 +843,7 @@ export function environmentCustomImageService( [SOURCE_ENVIRONMENT_CONFIG_FINGERPRINT_METADATA_KEY]: fingerprint, [SETUP_RPC_COMPANY_ID_METADATA_KEY]: provider.rpcCompanyId, }; - await db.insert(environmentCustomImageSetupSessions).values({ + await writeSetup(tx => tx.insert(environmentCustomImageSetupSessions).values({ id: sessionId, environmentId: input.environmentId, templateId: selectedTemplate?.id ?? null, @@ -847,7 +856,7 @@ export function environmentCustomImageService( metadata: setupMetadata, createdAt: now, updatedAt: now, - }); + })); try { const providerSession = await callProviderStart({ @@ -1020,6 +1029,7 @@ export function environmentCustomImageService( )); } await notifyEnvironmentCleanup(tx); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.customImageExpiry); await tx .update(environmentCustomImageSetupSessions) .set({ @@ -1355,6 +1365,17 @@ export function environmentCustomImageService( return template; }, + nextSetupCleanupAt: async (): Promise => { + const [row] = await db.select({ nextAt: sql`min(${environmentCustomImageSetupSessions.expiresAt})` }) + .from(environmentCustomImageSetupSessions) + .where(inArray(environmentCustomImageSetupSessions.status, [...ACTIVE_SETUP_STATUSES])); + if (!row || row.nextAt === undefined) throw new Error("Setup expiry query returned no result"); + if (row.nextAt === null) return null; + const deadline = new Date(row.nextAt).getTime(); + if (!Number.isFinite(deadline)) throw new Error("Setup expiry query returned an invalid deadline"); + return deadline; + }, + cleanupExpiredSetupSessions: async (input: { now?: Date; limit?: number; diff --git a/server/src/services/workspace-recovery.test.ts b/server/src/services/workspace-recovery.test.ts index 91b5e3e5b6f..a0a462e6056 100644 --- a/server/src/services/workspace-recovery.test.ts +++ b/server/src/services/workspace-recovery.test.ts @@ -298,9 +298,16 @@ describe("automatic retained workspace recovery", () => { } finally { await coordinator.stop(); await worker.stop(); } }, 30_000); - it("fills a free export slot while another export is held and joins both at shutdown", async () => { - const held = await seed(), fast = await seed(); - const fixtures = [held, fast]; + it("fills a free export slot across the candidate cursor while another export is held and joins both at shutdown", async () => { + const orderedSeed = async (position: number) => { + const fixture = await seed(); + const id = `00000002-000${position}-4000-8000-${randomUUID().slice(-12)}` as const; + await db.update(issueRecoveryActions).set({ id }).where(eq(issueRecoveryActions.id, fixture.actionId)); + fixture.actionId = id; fixture.input.actionId = id; + return fixture; + }; + const held = await orderedSeed(2); + const fixtures = [held]; const runtimeFor = (runId: string) => fixtures.find(fixture => fixture.runId === runId)!.environmentRuntime; const runtime = { ...held.environmentRuntime, resumeWorkspaceRecoveryLease: (input: Parameters[0]) => @@ -317,15 +324,20 @@ describe("automatic retained workspace recovery", () => { try { if (pause) await gate; return await resume(input); } finally { active -= 1; } }); }; - countExport(held, true); countExport(fast); + countExport(held, true); const errors = vi.fn(); const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: errors }); const worker = registerWorkspaceRecoveryWork(db, coordinator, runtime, async () => {}); try { await expect.poll(() => held.runtime.resumeRunLease.mock.calls.length, { timeout: 20_000 }).toBe(1); + // Start the middle candidate only after the highest candidate occupies + // one slot. The next producer then sorts before that saved cursor. + const fast = await orderedSeed(1); fixtures.push(fast); countExport(fast); + await requestTaskRetry({ db, companyId: fast.companyId, issueId: fast.issueId, actorId: "board", + observedRunId: fast.runId, idempotencyKey: randomUUID() }); await expect.poll(async () => readSystemRecovery((await action(fast.actionId)).evidence)?.state, { timeout: 20_000 }).toBe("complete"); - const third = await seed(); fixtures.push(third); countExport(third); + const third = await orderedSeed(0); fixtures.push(third); countExport(third); await requestTaskRetry({ db, companyId: third.companyId, issueId: third.issueId, actorId: "board", observedRunId: third.runId, idempotencyKey: randomUUID() }); await expect.poll(async () => readSystemRecovery((await action(third.actionId)).evidence)?.state, diff --git a/server/src/services/workspace-recovery.ts b/server/src/services/workspace-recovery.ts index 179ddc478ad..0cf699ab239 100644 --- a/server/src/services/workspace-recovery.ts +++ b/server/src/services/workspace-recovery.ts @@ -2,7 +2,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio import { beginIdleTrackedWork } from "./task-admission.js"; import { writeEnvironmentCleanup } from "./environment-cleanup-notifications.js"; import { resumePreservedResultFinalization } from "./preserved-result-finalization.js"; -import { and, asc, eq, gt, inArray, ne, or, sql } from "drizzle-orm"; +import { and, asc, eq, gt, inArray, ne, notInArray, or, sql } from "drizzle-orm"; import { agentWakeupRequests, environmentLeases, heartbeatRuns, issues, issueRecoveryActions, workspaceOperations, type Db } from "@paperclipai/db"; import type { Environment, EnvironmentLease } from "@paperclipai/shared"; import { conflict } from "../errors.js"; @@ -313,7 +313,7 @@ export async function scheduleRecoveredWorkspaceCleanup(db: Db, run: typeof hear sql`${environmentLeases.metadata} = ${JSON.stringify(lease.metadata)}::jsonb`, sql`${environmentLeases.metadata}->>'workspaceRecoveryCleanup' is null`))); } -export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false) { +export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedOnly = false, obligationsOnly = false, inFlightRunIds: string[] = []) { const cleanup = sql`(${heartbeatRuns.runnerProfileJson}#>>'{nativeWorkspaceSync,resourceDisposition}' = 'destroy' and exists ( select 1 from ${environmentLeases} where ${environmentLeases.companyId} = ${heartbeatRuns.companyId} and ${environmentLeases.heartbeatRunId} = ${heartbeatRuns.id} @@ -339,6 +339,7 @@ export function retainedWorkspaceCandidates(db: Db, cursor?: string, unfinishedO sql`(${issueRecoveryActions.evidence}#>>'{workspaceRecovery,status}' is distinct from 'recovered' or ${cleanup} or ${waitingInput})`, cursor ? gt(issueRecoveryActions.id, cursor) : undefined, + inFlightRunIds.length ? notInArray(heartbeatRuns.id, inFlightRunIds) : undefined, )).orderBy(asc(issueRecoveryActions.id)).limit(5); } @@ -351,11 +352,14 @@ export async function reconcileRetainedWorkspaces(db: Db, runtime: EnvironmentRu let pending = workInFlight.get(db); if (!pending) { pending = new Set(); workInFlight.set(db, pending); } if (pending.size >= 2) return; - let rows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true); - if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, true); + // Locally running exports already own their slot. Exclude them before + // pagination so a page containing only held work cannot prevent wrapping. + const inFlightRunIds = [...pending]; + let rows = await retainedWorkspaceCandidates(db, unfinishedCursors.get(db), true, false, inFlightRunIds); + if (!rows.length) rows = await retainedWorkspaceCandidates(db, undefined, true, false, inFlightRunIds); const unfinished = rows.length > 0; - if (!rows.length) rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly); - if (!rows.length && !passiveSweep) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly); + if (!rows.length) rows = await retainedWorkspaceCandidates(db, passiveSweep ? passiveSweep.cursor : cursors.get(db), false, obligationsOnly, inFlightRunIds); + if (!rows.length && !passiveSweep) rows = await retainedWorkspaceCandidates(db, undefined, false, obligationsOnly, inFlightRunIds); const work: Promise[] = []; let hasUnvisited = false; for (const { action, run } of rows) { From 623f1dc289a14a699ebbe3f09b94179d38fc2ae2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:36:28 -0500 Subject: [PATCH 02/41] fix(server): schedule secret-proposal expiry from committed changes (#17446) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents can propose secrets and bindings for approval. > - Pending proposals already contain their expiry and encrypted value. > - The server scans for expired proposals on every maintenance tick, even when none exist. > - Committed proposal changes and saved expiry times provide the required triggers. > - This PR uses those triggers and preserves scrubbing, approval checks, and recovery. > - Empty and completed proposal history no longer causes recurring expiry queries. ## Linked Issues or Issue Description Refs #17443. This is the second PR in the sleep migration stack and targets its branch. **What problem does this solve?** The periodic proposal scan contacts PostgreSQL when no proposal needs maintenance. That obstructs idle database behavior across tenant instances. **What is the proposed solution?** Use transaction-aware notifications for creation and resolution. Reconstruct the earliest pending expiry at startup. Run cleanup at that deadline, or retry while reconciliation remains unfinished. **Why is this useful?** It removes an idle scan while retaining security checks and physical ciphertext cleanup. No queue table or migration is required. ## What Changed - Added intent notifications before secret/binding creation, approval, terminal transitions, and linked interaction resolution. The existing outer transaction owns settlement. - Kept expiry selection, ciphertext scrubbing, dependent rejection, and authorization in the secret-proposal service. - Registered a startup and deadline worker in place of the aggregate maintenance sweep. Empty/terminal states disarm; overdue batches and failures retain retries. - Reused shared scoped deadline dispatch for secret expiry. The service owns company-filtered SQL and expiry policy. Custom-image expiry continues to use `registerDeadlineWork`. - Added real PostgreSQL producer, outer-transaction, concurrent-write, expiry, restart, interaction resolution, and drain/resume tests. Updated startup wiring fixtures and documented the supported writer/lifecycle contract. ## Verification - Landing reconciliation at `c10eb1bcf752b058713e05d3345e5cf38dd3fa72`: 160 tests pass across the complete secret-proposal, scoped deadline, company batch, coordinator, startup, custom-image service and custom-image worker suites; no cases skipped in the accepted runs. Server typecheck passes. This integrates master after #17443 landed; fresh remote CI and Ponytail review are required before merge. Current head: `5bbe17ad9ebc4573b08ed1a9323644ef9608ca09`. This remains a single-duty PR stacked on #17443. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover supported writers sharing the database signal owner. Independent SQL writers and other process roots require a separate ingress contract; startup recovery alone is not live replica support. - The shared coordinator and scoped worker also serve later stack duties. Their defaults remain intact; existing voice, chat, native, and startup consumers are included in integrated verification. - Startup now checks overdue proposals immediately. During drain, physical cleanup waits; approval still rejects expired proposals synchronously. - Each due company pass remains bounded to 100 rows. Failed scopes back off from 30 seconds to five minutes; healthy overdue work keeps draining. Self-generated write hints coalesce and do not erase a failed sweep's backoff. - Heartbeat-disabled behavior and Cloud sleep-safety gates remain unchanged. This does not establish deployed suspension or external wake coverage. Tool review and gateway token cleanup remain separate slices. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-secret-proposal-expiry.md | 90 +++++ ...-instance-sleep-and-scheduler-inventory.md | 6 +- ...2026-10-24-recurring-server-sleep-stack.md | 4 +- .../src/__tests__/company-work-batch.test.ts | 29 ++ .../delivery-work-coordinator.test.ts | 52 +++ .../__tests__/scoped-deadline-work.test.ts | 39 +++ .../__tests__/secret-proposals-routes.test.ts | 314 +++++++++++++++++- .../server-startup-feedback-export.test.ts | 25 +- server/src/index.ts | 11 +- server/src/services/company-deletion.ts | 1 + server/src/services/company-work-batch.ts | 28 ++ .../src/services/custom-image-cleanup-work.ts | 21 +- server/src/services/deadline-work.ts | 36 ++ .../src/services/delivery-work-coordinator.ts | 16 +- .../services/delivery-work-notifications.ts | 1 + .../src/services/issue-thread-interactions.ts | 2 + server/src/services/scoped-deadline-work.ts | 99 ++++++ .../services/secret-proposal-expiry-work.ts | 29 ++ server/src/services/secret-proposals.ts | 30 +- 19 files changed, 792 insertions(+), 41 deletions(-) create mode 100644 doc/idle-secret-proposal-expiry.md create mode 100644 server/src/__tests__/company-work-batch.test.ts create mode 100644 server/src/__tests__/scoped-deadline-work.test.ts create mode 100644 server/src/services/company-work-batch.ts create mode 100644 server/src/services/deadline-work.ts create mode 100644 server/src/services/scoped-deadline-work.ts create mode 100644 server/src/services/secret-proposal-expiry-work.ts diff --git a/doc/idle-secret-proposal-expiry.md b/doc/idle-secret-proposal-expiry.md new file mode 100644 index 00000000000..a66747421fb --- /dev/null +++ b/doc/idle-secret-proposal-expiry.md @@ -0,0 +1,90 @@ +# Secret-proposal expiry and idle behavior + +Secret proposals already store `status`, `expiresAt`, and encrypted pending +values. This worker uses those records; it adds no queue table or migration. + +## Ownership and activation + +`secret-proposals.ts` still owns quotas, authority checks, approval, expiry, +ciphertext scrubbing, dependent binding rejection, audit records, and interaction +reflection. `expiryDeadlines(companyId?)` returns one saved deadline per pending +company. Normal write notifications reload only the affected company. Startup +reconstruction loads all companies once. The manual `nextExpiryAt()` query remains +available to existing callers. +Terminal history does not create a deadline. Approval still rejects an expired +proposal synchronously, even when maintenance is paused. + +The supported producers register `secretProposalExpiry` intent before their +writes in the owning transaction: + +- `createWithinQuota`: both secret and binding creation, after quota admission. +- `markApproved`: ordinary approval and prerequisite approval during a cascade. +- `transition`: rejection, withdrawal, expiry, and dependent binding rejection. +- Interaction resolution: rejection, withdrawal/cancellation, expiry, and failed + acceptance execution that resolves a linked pending binding. +- Company deletion: a company hint removes its cached deadline after deletion commits. + +Each producer includes its company ID. The remaining proposal write only attaches +an interaction ID and does not change expiry eligibility. Independent SQL writers +or another DB root are not supported live producers of these process-local signals; +startup reconstructs their persisted rows, but it is not a live replica contract. + +The shared database signal boundary defers notification until the outer transaction +settles. Nested savepoints, rollback, and uncertain commit outcomes retain the +existing protocol. Callers do not pass recovery counters or reconnect the database. + +## Scheduling and lifecycle + +`registerSecretProposalExpiryWork` runs once at startup while heartbeat scheduling +is enabled. It replaces the secret-proposal sweep in the aggregate maintenance +tick. Disabled heartbeat scheduling keeps its existing behavior. + +The worker reloads the affected company's saved minimum expiry after a committed +change. It does not sweep future work. At a due deadline it sweeps at most 100 +expired proposals for that company, then reads that company's next deadline. +Companies rotate fairly between bounded passes. Retained history creates no work. +No pending proposal, unresolved write, or failed reconciliation means no recurring +query. Unscoped legacy hints still reconstruct all companies. + +`scoped-deadline-work.ts` owns scope invalidation, cached deadlines, fair dispatch, +and failure backoff. It owns no SQL or proposal policy. Failed loads and failed +company sweeps back off independently from 30 seconds to five minutes. A new +write reloads that company without erasing its failed sweep's backoff. Other +companies keep their own deadlines, including already-due work. The coordinator's +optional `nextRunAtOnError` contract permits immediate eligible work only after +the worker has recorded per-item backoffs. Failed transaction reconciliation and +failed pending-state inspection retain the coordinator's normal retry. + +Secret expiry opts out of the shared helper's default one-second pass spacing so +newly learned short deadlines are not clamped. Other scoped consumers retain that +default. Custom-image expiry continues to use `deadline-work.ts` unchanged. No +shared scheduler or startup wiring interprets secret-proposal policy. + +A request stays pending until reconciliation succeeds. Changes arriving during a +pass, first writes during drain, and failures followed by drain retain a request. +Passive drain performs no proposal SQL. Resume reconciles retained requests; +restart reconstructs deadlines. Coordinator shutdown unsubscribes and waits for +active work. Deferred physical cleanup does not extend approval authorization. + +## Verification and limits + +The PostgreSQL route suite exercises actual service producers, nested outer commit +and rollback, quiet empty and terminal states, quiet future deadlines, automatic +ciphertext scrubbing and dependent-card resolution, startup reconstruction, +withdrawal, interaction rejection/failed execution, failure retry, and drain/resume. +It also verifies that expired approval fails while cleanup is paused. The existing +coordinator suite verifies fallback retry and uncertainty boundaries. New cases +record actual SQL to prove normal writes and due expiry stay company-scoped; they +also cover five-second deadlines, earlier replacement deadlines, company deletion, +and another company making progress while one sweep fails. A deterministic +shared-worker test covers a 250ms replacement deadline. + +The worker tests use a controlled clock and a real PostgreSQL database. Existing +route tests cover quotas, isolation, authorization, concurrency, and lifecycle +semantics. The test-only database URL must refer to a disposable migrated database: +the suite deletes its fixture tables. Ordinary CI uses its isolated embedded DB. + +This is worker quiet, not proof of host or database suspension. No Cloud alarm or +sleep-eligibility exemption is added. Existing sleep-safety checks remain in force; +a local expiry timer cannot wake a stopped host. Tool review expiry and gateway +session cleanup are separate migrations. diff --git a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md index b92d7520ec0..9141878caa8 100644 --- a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md +++ b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md @@ -647,8 +647,10 @@ Longer polling intervals alone do not complete an item. visible within the agreed freshness contract. Source: [index.ts](../../server/src/index.ts), `tools.sweepConnectionHealth`. -- [ ] **JOB-21 — Secret proposals and expired action tokens.** Current: **30s** - expiry/cleanup queries. Change: enforce expiry synchronously on use and +- [ ] **JOB-21 — Secret proposals and expired action tokens (partially migrated).** + Secret-proposal expiry now uses transaction signals and saved expiry deadlines; + see [the lifecycle contract](../idle-secret-proposal-expiry.md). Tool review + recovery and gateway token cleanup still use **30s** queries. Change: enforce expiry synchronously on use and register required scrub/cleanup deadlines. **Complete when:** idle cleanup does not extend authorization or secret-retention periods. Sources: [secret-proposals.ts](../../server/src/services/secret-proposals.ts), diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 50f73222265..20221274e70 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -12,13 +12,13 @@ Do not mark an item done from a timer removal or a green check alone. | ID | Original checklist duty | Required slices / disposition | PR / evidence | | --- | --- | --- | --- | -| 01 | Custom-image setup expiry | Session writes, existing expiry, startup/retry/drain | In progress | +| 01 | Custom-image setup expiry | Session writes, existing expiry, startup/retry/drain | [#17443](https://github.com/paperclipai/paperclip/pull/17443), merged after CI and passing reviews | | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | -| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Pending | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret-proposal expiry in progress; tool reviews and gateway tokens pending | | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | diff --git a/server/src/__tests__/company-work-batch.test.ts b/server/src/__tests__/company-work-batch.test.ts new file mode 100644 index 00000000000..9fe1abc5a1d --- /dev/null +++ b/server/src/__tests__/company-work-batch.test.ts @@ -0,0 +1,29 @@ +import { describe, expect, it } from "vitest"; +import { takeCompanyWorkBatch } from "../services/company-work-batch.js"; + +describe("company work batch", () => { + it("shares capacity while preserving the owner's order within each company", () => { + const entries = [ + { companyId: "a", id: "a-oldest" }, { companyId: "a", id: "a-next" }, + { companyId: "a", id: "a-later" }, { companyId: "b", id: "b-due" }, + ]; + expect(takeCompanyWorkBatch(entries, 3)).toEqual({ + entries: [entries[0], entries[3], entries[1]], lastCompany: "a", + }); + }); + it("rotates between batches when more companies are due than fit in one batch", () => { + const entries = ["a", "b", "c", "d"].flatMap(companyId => [ + { companyId, id: `${companyId}-first` }, { companyId, id: `${companyId}-second` }, + ]); + const first = takeCompanyWorkBatch(entries, 2); + const remaining = entries.filter(entry => !first.entries.includes(entry)); + const second = takeCompanyWorkBatch(remaining, 2, first.lastCompany); + expect([...first.entries, ...second.entries].map(entry => entry.companyId)).toEqual(["a", "b", "c", "d"]); + expect(takeCompanyWorkBatch(remaining, 1, "z").entries[0].companyId).toBe("a"); + }); + it("finishes an empty or exhausted batch without inventing work", () => { + expect(takeCompanyWorkBatch([], 25, "gone")).toEqual({ entries: [], lastCompany: "gone" }); + const entry = { companyId: "only" }; + expect(takeCompanyWorkBatch([entry], 25).entries).toEqual([entry]); + }); +}); diff --git a/server/src/__tests__/delivery-work-coordinator.test.ts b/server/src/__tests__/delivery-work-coordinator.test.ts index 805aa3db4c4..2ec8054336f 100644 --- a/server/src/__tests__/delivery-work-coordinator.test.ts +++ b/server/src/__tests__/delivery-work-coordinator.test.ts @@ -59,6 +59,58 @@ describe("delivery work coordinator", () => { expect(s.onError).not.toHaveBeenCalled(); } finally { clock.mockRestore(); } }); + it("honors a future backlog deadline after failure while preserving the fallback retry", async () => { + const s = setup(), t = task(); + let next: number | null = Date.now() + 1000; + t.run.mockRejectedValueOnce(new Error("One item failed")); + await s.coordinator.register(DELIVERY_QUEUES.feedback, { ...t, nextRunAt: () => next }).ready; + expect(s.coordinator.nextWakeAt()).toBe(next); + expect(s.onError).toHaveBeenCalledOnce(); + next = null; + await vi.advanceTimersByTimeAsync(1000); + expect(t.run).toHaveBeenCalledTimes(2); + expect(s.coordinator.nextWakeAt()).toBeNull(); + }); + it("dispatches another due item after a worker records scoped failure backoffs", async () => { + const s = setup(), t = task(), now = Date.now(); + t.run.mockRejectedValueOnce(new Error("One scoped item failed")); + const nextRunAtOnError = vi.fn(() => now); + await s.coordinator.register(DELIVERY_QUEUES.feedback, { ...t, nextRunAtOnError }).ready; + expect(nextRunAtOnError).toHaveBeenCalledOnce(); + expect(s.coordinator.nextWakeAt()).toBe(now); + await vi.advanceTimersByTimeAsync(1); + expect(t.run).toHaveBeenCalledTimes(2); + expect(s.coordinator.nextWakeAt()).toBeNull(); + }); + it.each(["reconcile", "pending"])("does not consume scoped retry deadlines for a %s failure", async mode => { + const s = setup(), t = task(), now = Date.now(); + if (mode === "reconcile") { + s.failCommit(); + await expect(s.enqueue()).rejects.toThrow("lost commit reply"); + s.execute.mockRejectedValueOnce(new Error("Database unavailable")); + } else t.hasPending.mockRejectedValueOnce(new Error("Pending state unavailable")); + const nextRunAtOnError = vi.fn(() => now); + await s.coordinator.register(DELIVERY_QUEUES.feedback, { ...t, nextRunAtOnError }).ready; + expect(nextRunAtOnError).not.toHaveBeenCalled(); + expect(s.coordinator.nextWakeAt()).toBe(now + 5000); + expect(s.onError).toHaveBeenCalledOnce(); + await vi.advanceTimersByTimeAsync(4999); + expect(s.onError).toHaveBeenCalledOnce(); + }); + it.each(["overdue", "later", "unreadable", "draining"])("keeps failure backoff for a %s deadline", async mode => { + const s = setup(), t = task(), now = Date.now(); + t.run.mockImplementation(async () => { + if (mode === "draining") { s.canRun.mockReturnValue(false); s.canReconcile.mockReturnValue(true); } + throw new Error("Unavailable"); + }); + await s.coordinator.register(DELIVERY_QUEUES.feedback, { ...t, nextRunAt: () => { + if (mode === "unreadable") throw new Error("Deadline unavailable"); + return mode === "overdue" ? now : now + (mode === "later" ? 60_000 : 1000); + } }).ready; + expect(s.coordinator.nextWakeAt()).toBe(now + 5000); + await vi.advanceTimersByTimeAsync(4999); + expect(t.run).toHaveBeenCalledOnce(); + }); it("schedules maintenance without holding idle intent, and lets new work preempt it", async () => { const s = setup(), t = task(); let next: number | null = Date.now() + 60_000; diff --git a/server/src/__tests__/scoped-deadline-work.test.ts b/server/src/__tests__/scoped-deadline-work.test.ts new file mode 100644 index 00000000000..fda914dd13e --- /dev/null +++ b/server/src/__tests__/scoped-deadline-work.test.ts @@ -0,0 +1,39 @@ +import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js"; +import { afterEach, expect, it, vi } from "vitest"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { registerScopedDeadlineWork } from "../services/scoped-deadline-work.js"; + +let coordinator: ReturnType | undefined; +afterEach(async () => { await coordinator?.stop(); vi.useRealTimers(); }); + +it("preserves a subsecond replacement deadline when the owner does not require pass spacing", async () => { + vi.useFakeTimers(); + const onError = vi.fn(); + const owner = installDatabaseWorkSignals({ execute: vi.fn(async () => []) }); + coordinator = createDeliveryWorkCoordinator({ owner, canRun: () => true, onError }); + const now = Date.now(); + let dueAt = now + 60_000; + const check = vi.fn(async () => null); + const worker = registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.secretProposalExpiry, + minimumPassIntervalMs: 0, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: (entry, companyId?: string) => companyId === undefined || entry.companyId === companyId, + load: async () => [{ id: "company", companyId: "company", dueAt }], + check, + }, () => true); + await worker.ready; + dueAt = now + 250; + worker.invalidate("company"); + await vi.advanceTimersByTimeAsync(1); + expect(coordinator.nextWakeAt()).toBe(dueAt); + await vi.advanceTimersByTimeAsync(248); + expect(check).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + expect(check).toHaveBeenCalledOnce(); + expect(coordinator.nextWakeAt()).toBeNull(); + await vi.advanceTimersByTimeAsync(60_000); + expect(check).toHaveBeenCalledOnce(); + expect(onError).not.toHaveBeenCalled(); +}); diff --git a/server/src/__tests__/secret-proposals-routes.test.ts b/server/src/__tests__/secret-proposals-routes.test.ts index 3354345e4f4..935fdb84f89 100644 --- a/server/src/__tests__/secret-proposals-routes.test.ts +++ b/server/src/__tests__/secret-proposals-routes.test.ts @@ -1,3 +1,8 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { registerSecretProposalExpiryWork } from "../services/secret-proposal-expiry-work.js"; +import { deleteCompany } from "../services/company-deletion.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; import { randomUUID } from "node:crypto"; import { mkdirSync, rmSync } from "node:fs"; import os from "node:os"; @@ -5,7 +10,7 @@ import path from "node:path"; import express from "express"; import request from "supertest"; import { and, eq, sql } from "drizzle-orm"; -import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { activityLog, agents, @@ -38,7 +43,8 @@ import { startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalTestDatabase = resolveExternalTestDatabaseUrl("PAPERCLIP_SECRET_PROPOSAL_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalTestDatabase ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; describeEmbeddedPostgres("secret proposal routes", () => { @@ -50,9 +56,10 @@ describeEmbeddedPostgres("secret proposal routes", () => { beforeAll(async () => { mkdirSync(secretsTmpDir, { recursive: true }); process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = path.join(secretsTmpDir, "master.key"); + if (externalTestDatabase) { db = createDb(externalTestDatabase, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); return; } const started = await startEmbeddedPostgresTestDatabase("secret-proposal-routes"); stopDb = started.cleanup; - db = createDb(started.connectionString); + db = createDb(started.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); }); afterEach(async () => { @@ -74,14 +81,14 @@ describeEmbeddedPostgres("secret proposal routes", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await stopDb?.(); if (previousKeyFile === undefined) delete process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE; else process.env.PAPERCLIP_SECRETS_MASTER_KEY_FILE = previousKeyFile; rmSync(secretsTmpDir, { recursive: true, force: true }); }); - async function seedRun() { - const companyId = randomUUID(); + async function seedRun(identifier = "SEC-1", companyId = randomUUID()) { const agentId = randomUUID(); const heartbeatRunId = randomUUID(); const issueId = randomUUID(); @@ -113,7 +120,7 @@ describeEmbeddedPostgres("secret proposal routes", () => { id: issueId, companyId, title: "Needs credential", - identifier: "SEC-1", + identifier, status: "in_progress", responsibleUserId: "user-1", executionRunId: heartbeatRunId, @@ -221,6 +228,301 @@ describeEmbeddedPostgres("secret proposal routes", () => { return app; } + describe("expiry scheduling", () => { + let coordinator: ReturnType | undefined; + let proposals: ReturnType; + let fixture: Awaited>; + let draining: boolean; + const errors: unknown[] = []; + const expiryMs = 14 * 24 * 60 * 60_000; + + beforeEach(async () => { + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + fixture = await seedRun(); + proposals = createSecretProposalsService(db); + draining = false; + errors.length = 0; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); + coordinator = undefined; + vi.useRealTimers(); + expect(errors).toEqual([]); + }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerSecretProposalExpiryWork(coordinator, proposals, () => !draining).ready; + } + async function settle(ms = 60_000) { + await vi.advanceTimersByTimeAsync(ms); + if (!draining) await vi.waitFor(() => expect(idleWorkSnapshot().active === 0).toBe(true), { timeout: 5000 }); + } + async function quiet() { + await settle(); + expect(coordinator!.nextWakeAt()).toBeNull(); + const reads = vi.spyOn(db, "select"); + await settle(60 * 60_000); + expect(reads).not.toHaveBeenCalled(); + reads.mockRestore(); + } + const create = (service = proposals) => service.createSecret({ ...fixture, registerForRedaction: () => {} }, { + name: "dev/scheduled/secret", value: "expiry-test-only", justification: "Test durable expiry", + }); + const bind = (secretProposalId: string) => proposals.createBinding(fixture, { + secretProposalId, configPath: "access.SCHEDULED", justification: "Test dependent expiry", + bindingTargetPolicy: "self_and_reports", + }); + + async function createWithExpiry(expiresAt: Date) { + return db.transaction(async tx => { + const proposal = await create(createSecretProposalsService(tx as unknown as typeof db)); + const [updated] = await tx.update(companySecretProposals).set({ expiresAt }) + .where(eq(companySecretProposals.id, proposal.id)).returning(); + return updated; + }); + } + + it("honors a saved five-second expiry without clamping it to recovery spacing", async () => { + const delay = 5000; + await start(); await quiet(); + const expiresAt = new Date(Date.now() + delay); + const proposal = await createWithExpiry(expiresAt); await settle(1); + expect(coordinator!.nextWakeAt()).toBe(expiresAt.getTime()); + await settle(delay / 2); + expect(await proposals.getById(fixture.companyId, proposal.id)).toMatchObject({ status: "pending" }); + await settle(delay); + expect(await proposals.getById(fixture.companyId, proposal.id)).toMatchObject({ status: "expired", valueCiphertext: null }); + await quiet(); + }); + + it("moves to an earlier committed proposal deadline without losing the later one", async () => { + await start(); await quiet(); + const later = await createWithExpiry(new Date(Date.now() + 120_000)); await settle(1); + expect(coordinator!.nextWakeAt()).toBe(later.expiresAt.getTime()); + const earlier = await createWithExpiry(new Date(Date.now() + 5000)); await settle(1); + expect(coordinator!.nextWakeAt()).toBe(earlier.expiresAt.getTime()); + await settle(6000); + expect(await proposals.getById(fixture.companyId, earlier.id)).toMatchObject({ status: "expired", valueCiphertext: null }); + expect(await proposals.getById(fixture.companyId, later.id)).toMatchObject({ status: "pending" }); + expect(coordinator!.nextWakeAt()).toBe(later.expiresAt.getTime()); + await settle(120_000); await quiet(); + }); + + it("keeps sequential tenant writes from scanning another company's proposals", async () => { + const other = await seedRun("OTHER-1"); + const foreign = await proposals.createSecret({ ...other, registerForRedaction: () => {} }, { + name: "dev/other/secret", value: "other-expiry-only", justification: "Another company's deadline", + }); + await start(); await settle(1); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query, params) => { + if (/^select/i.test(query) && query.includes('"company_secret_proposals"') && + (query.includes('min(') || query.includes('"expires_at" <='))) reads.push({ query, params: [...params] }); + }; + try { + for (let i = 0; i < 3; i++) { await create(); await settle(1000); } + } finally { db.$client.options.debug = debug; } + expect(reads.length).toBeGreaterThan(0); + for (const read of reads) { + expect(read.query).toContain('"company_secret_proposals"."company_id" ='); + expect(read.params).toContain(fixture.companyId); + expect(read.params).not.toContain(other.companyId); + } + expect(await proposals.getById(other.companyId, foreign.id)).toMatchObject({ status: "pending", valueCiphertext: foreign.valueCiphertext }); + }); + + it("expires only the company whose saved deadline is due", async () => { + const other = await seedRun("OTHER-1"); + const foreign = await proposals.createSecret({ ...other, registerForRedaction: () => {} }, { + name: "dev/other/secret", value: "other-expiry-only", justification: "Another company's deadline", + }); + const due = await createWithExpiry(new Date(Date.now() + 5000)); + await start(); await settle(1); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query, params) => { + if (/^select/i.test(query) && query.includes('"company_secret_proposals"') && + (query.includes('min(') || query.includes('"expires_at" <='))) reads.push({ query, params: [...params] }); + }; + try { await settle(6000); } finally { db.$client.options.debug = debug; } + expect(reads.some(read => read.query.includes('"expires_at" <='))).toBe(true); + for (const read of reads) { + expect(read.query).toContain('"company_secret_proposals"."company_id" ='); + expect(read.params).toContain(fixture.companyId); + expect(read.params).not.toContain(other.companyId); + } + expect(await proposals.getById(fixture.companyId, due.id)).toMatchObject({ status: "expired", valueCiphertext: null }); + expect(await proposals.getById(other.companyId, foreign.id)).toMatchObject({ status: "pending", valueCiphertext: foreign.valueCiphertext }); + expect(coordinator!.nextWakeAt()).toBe(foreign.expiresAt.getTime()); + }); + + it.each([0, 3000])("keeps a failed company on backoff while another company expires on time (offset %s)", async offset => { + // Make the failing company first in the fair dispatch order. + const other = await seedRun("OTHER-1", "ffffffff-ffff-ffff-ffff-ffffffffffff"); + const failing = await createWithExpiry(new Date(Date.now() + 5000)); + const healthy = await db.transaction(async tx => { + const proposal = await createSecretProposalsService(tx as unknown as typeof db).createSecret( + { ...other, registerForRedaction: () => {} }, + { name: "dev/other/secret", value: "other-expiry-only", justification: "Independent expiry" }); + const [saved] = await tx.update(companySecretProposals).set({ expiresAt: new Date(failing.expiresAt.getTime() + offset) }) + .where(eq(companySecretProposals.id, proposal.id)).returning(); + return saved; + }); + const sweep = proposals.sweepExpiredForCompany; + let unavailable = true; + const attempts = vi.spyOn(proposals, "sweepExpiredForCompany").mockImplementation(async companyId => { + if (unavailable && companyId === fixture.companyId) throw new Error("company expiry unavailable"); + return sweep(companyId); + }); + await start(); await settle(10_000); + expect(errors).toHaveLength(1); errors.length = 0; + expect(await proposals.getById(other.companyId, healthy.id)).toMatchObject({ status: "expired", valueCiphertext: null }); + expect(await proposals.getById(fixture.companyId, failing.id)).toMatchObject({ status: "pending" }); + // A new write must refresh this company without erasing its failed sweep's backoff. + await create(); await settle(1000); + expect(attempts.mock.calls.filter(([id]) => id === fixture.companyId)).toHaveLength(1); + unavailable = false; await settle(30_000); + expect(await proposals.getById(fixture.companyId, failing.id)).toMatchObject({ status: "expired", valueCiphertext: null }); + await settle(expiryMs); await quiet(); + }); + + it("removes the cached expiry when the company deletion owner commits", async () => { + await create(); await start(); await settle(1); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, fixture.heartbeatRunId)); + await db.update(issues).set({ executionRunId: null }).where(eq(issues.id, fixture.issueId)); + // The deletion fixture has no live resources; its agent has already + // finished termination before entering the company deletion transaction. + await db.update(agents).set({ status: "terminated", lifecycleState: "terminated" }) + .where(eq(agents.id, fixture.agentId)); + expect(await deleteCompany(db, fixture.companyId)).toMatchObject({ id: fixture.companyId }); + await quiet(); + expect(await proposals.expiryDeadlines()).toEqual([]); + }); + + it("stays quiet until the real proposal deadline, scrubs ciphertext, and leaves terminal history quiet", async () => { + await start(); await quiet(); + const secret = await create(); + const binding = await bind(secret.id); + expect(secret.valueCiphertext).not.toBeNull(); + await settle(); + expect(coordinator!.nextWakeAt()).toBe(secret.expiresAt.getTime()); + const reads = vi.spyOn(db, "select"); + await settle(24 * 60 * 60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await settle(expiryMs); + expect(await proposals.getById(fixture.companyId, secret.id)).toMatchObject({ + status: "expired", valueCiphertext: null, ciphertextScrubbedAt: expect.any(Date), + }); + expect(await proposals.getById(fixture.companyId, binding.id)).toMatchObject({ status: "rejected" }); + expect(await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, binding.interactionId!))) + .toEqual([expect.objectContaining({ status: "rejected" })]); + await quiet(); + }); + + it.each([false, true])("only observes the producer after outer settlement (rollback %s)", async rollback => { + await start(); await quiet(); + let release!: () => void, written!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const staged = new Promise(resolve => { written = resolve; }); + const rollbackError = new Error("rollback fixture"); + const writing = db.transaction(async tx => { + await create(createSecretProposalsService(tx as unknown as typeof db)); + written(); await gate; + if (rollback) throw rollbackError; + }).catch(error => { if (error !== rollbackError) throw error; }); + await staged; + try { + await vi.advanceTimersByTimeAsync(60_000); + expect(await proposals.nextExpiryAt()).toBeNull(); + } finally { release(); } + await writing; await settle(); + expect(await proposals.nextExpiryAt()).toEqual(rollback ? null : expect.any(Number)); + if (rollback) await quiet(); + else expect(coordinator!.nextWakeAt()).toBe(await proposals.nextExpiryAt()); + }); + + it("retains a proposal committed while an empty deadline read is finishing", async () => { + let release!: () => void, reading!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const entered = new Promise(resolve => { reading = resolve; }); + const readDeadline = proposals.expiryDeadlines; + vi.spyOn(proposals, "expiryDeadlines").mockImplementationOnce(async () => { + const stale = await readDeadline(); + reading(); await gate; + return stale; + }); + const starting = start(); + await entered; + let secret!: Awaited>; + try { secret = await create(); } finally { release(); } + await starting; await settle(); + expect(coordinator!.nextWakeAt()).toBe(secret.expiresAt.getTime()); + await proposals.transition(fixture.companyId, secret.id, "withdrawn", { proposerAgentId: fixture.agentId }); + await quiet(); + }); + + it("reconstructs a deadline after restart and disarms when the proposer withdraws", async () => { + const secret = await create(); + await start(); await settle(); + expect(coordinator!.nextWakeAt()).toBe(secret.expiresAt.getTime()); + await proposals.transition(fixture.companyId, secret.id, "withdrawn", { proposerAgentId: fixture.agentId }); + await quiet(); + expect(await proposals.getById(fixture.companyId, secret.id)).toMatchObject({ valueCiphertext: null }); + }); + + it.each(["startup", "first_write", "failed_pass"])("retains %s work through drain and still rejects expired approval on use", async mode => { + if (mode === "startup") draining = true; + await start(); + if (mode === "first_write") { await quiet(); draining = true; } + const secret = await create(); + if (mode === "failed_pass") { + vi.spyOn(proposals, "expiryDeadlines").mockRejectedValueOnce(new Error("database unavailable")); + await vi.advanceTimersByTimeAsync(30_000); + await vi.waitFor(() => expect(errors).toHaveLength(1)); errors.length = 0; + draining = true; + } + await settle(); + vi.setSystemTime(Date.now() + expiryMs); + const reads = vi.spyOn(db, "select"); + await settle(2 * 60 * 60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + expect(await proposals.getById(fixture.companyId, secret.id)).toMatchObject({ status: "pending" }); + await expect(proposals.approve(fixture.companyId, secret.id, { resolvedByUserId: "board-user" })) + .rejects.toThrow("Expired proposals cannot be approved"); + draining = false; await settle(); + expect(await proposals.getById(fixture.companyId, secret.id)).toMatchObject({ status: "expired", valueCiphertext: null }); + await quiet(); + }); + + it.each(["rejected", "execution_failed"])("reschedules when the interaction owner resolves a binding: %s", async mode => { + const secret = await create(); + await start(); + const approved = await proposals.approve(fixture.companyId, secret.id, { resolvedByUserId: "board-user" }); + await quiet(); + const remaining = await proposals.createBinding(fixture, { secretId: approved.createdSecretId, + configPath: "access.SECOND", justification: "Independent binding", bindingTargetPolicy: "self_and_reports" }); + await settle(); + expect(coordinator!.nextWakeAt()).toBe(remaining.expiresAt.getTime()); + const interactions = issueThreadInteractionService(db); + if (mode === "rejected") { + await interactions.rejectInteraction({ id: fixture.issueId, companyId: fixture.companyId, status: "in_progress" }, + remaining.interactionId!, { reason: "No longer required" }, { userId: "board-user" }); + } else { + await db.update(issueThreadInteractions).set({ status: "accepted", result: { version: 1, outcome: "accepted" }, + resolvedByUserId: "board-user", resolvedAt: new Date() }).where(eq(issueThreadInteractions.id, remaining.interactionId!)); + await interactions.recordSecretProposalExecutionResult({ id: fixture.issueId, companyId: fixture.companyId }, + remaining.interactionId!, remaining.id, { status: "failed", errorCode: "permission_denied" }); + } + expect(await proposals.getById(fixture.companyId, remaining.id)).toMatchObject({ status: "rejected" }); + await quiet(); + }); + }); + it("requires company admin access to reject secret proposals", async () => { const fixture = await seedRun(); const proposed = await request(createAgentApp(fixture)) diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 81c689366c3..f3d7171c03a 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,3 +1,4 @@ +import { createSecretProposalsService } from "../services/secret-proposals.js"; import { logger } from "../middleware/logger.js"; import { getStartupRecoveryState } from "../startup-recovery-state.js"; import { idleWorkSnapshot } from "../services/task-admission.js"; @@ -29,6 +30,7 @@ const { detectPortMock, deriveAuthTrustedOriginsMock, deliveryWorkRegisterMock, + deliveryWorkErrors, environmentCustomImagesServiceMock, environmentCustomImagesServiceFactoryMock, executionWorkspaceServiceFactoryMock, @@ -53,11 +55,12 @@ const { }; const caseHandoffDeliveryServiceFactoryMock = vi.fn(() => caseHandoffDeliveryServiceMock); const completionSweepMock = vi.fn(async () => undefined); - const deliveryWorkRegisterMock = vi.fn((_queue: string, task: { + const deliveryWorkErrors: Array<{ queue: string; error: unknown }> = []; + const deliveryWorkRegisterMock = vi.fn((queue: string, task: { retryMs?: number; run: (signal?: AbortSignal) => Promise; hasPending?: () => Promise; - }) => ({ ready: task.run(new AbortController().signal).catch(() => undefined), wake: vi.fn() })); + }) => ({ ready: task.run(new AbortController().signal).catch(error => { deliveryWorkErrors.push({ queue, error }); }), wake: vi.fn() })); const createAppMock = vi.fn(async () => Object.assign((_: unknown, __: unknown) => {}, { locals: { fastResponses: { hasPending: vi.fn(async () => false), sweepPending: vi.fn(async () => undefined) }, @@ -181,6 +184,7 @@ const { detectPortMock, deriveAuthTrustedOriginsMock, deliveryWorkRegisterMock, + deliveryWorkErrors, environmentCustomImagesServiceMock, environmentCustomImagesServiceFactoryMock, executionWorkspaceServiceFactoryMock, @@ -442,6 +446,9 @@ vi.mock("../services/native-runtime/native-question-bridge.js", () => ({ vi.mock("../services/secret-proposals.js", () => ({ createSecretProposalsService: vi.fn(() => ({ + expiryDeadlines: vi.fn(async () => []), + sweepExpiredForCompany: vi.fn(async () => 0), + nextExpiryAt: vi.fn(async () => null), sweepExpired: vi.fn(async () => 0), })), })); @@ -490,6 +497,7 @@ async function settleStartupHousekeeping() { describe("startServer feedback export wiring", () => { beforeEach(() => { vi.clearAllMocks(); + deliveryWorkErrors.length = 0; process.env.PAPERCLIP_DECISION_SIGNING_SECRET = "fedcba9876543210fedcba9876543210"; process.env.PAPERCLIP_AGENT_JWT_SECRET = "0123456789abcdef0123456789abcdef"; loadConfigMock.mockReturnValue(buildTestConfig()); @@ -791,6 +799,19 @@ describe("startServer feedback export wiring", () => { } }); + it("loads secret expiry deadlines successfully during enabled startup", async () => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true })); + await startServer(); + await settleStartupHousekeeping(); + const index = deliveryWorkRegisterMock.mock.calls.findIndex(([queue]) => queue === DELIVERY_QUEUES.secretProposalExpiry); + expect(index).toBeGreaterThanOrEqual(0); + await deliveryWorkRegisterMock.mock.results[index].value.ready; + expect(deliveryWorkErrors.filter(({ queue }) => queue === DELIVERY_QUEUES.secretProposalExpiry)).toEqual([]); + const service = vi.mocked(createSecretProposalsService).mock.results[0].value; + expect(service.expiryDeadlines).toHaveBeenCalledExactlyOnceWith(undefined); + expect(service.sweepExpiredForCompany).not.toHaveBeenCalled(); + }); + it("registers native replacement recovery without an execution-control interval", async () => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: true })); const setIntervalSpy = vi.spyOn(globalThis, "setInterval"); diff --git a/server/src/index.ts b/server/src/index.ts index 411dda3f87f..ca51efe4345 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerSecretProposalExpiryWork } from "./services/secret-proposal-expiry-work.js"; import { registerCustomImageCleanupWork } from "./services/custom-image-cleanup-work.js"; import { registerWorkspaceRecoveryWork } from "./services/workspace-recovery-work.js"; import { registerEnvironmentCleanupWork } from "./services/environment-cleanup-work.js"; @@ -1791,6 +1792,8 @@ async function startServerWithDatabaseTeardown( work().catch((err: unknown) => { logger.error({ err, startupStep: name }, "startup housekeeping step failed"); }); + await step("secret_proposal_expiry", () => registerSecretProposalExpiryWork(deliveryWork, secretProposals, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready); await step("custom_image_setup_cleanup", () => registerCustomImageCleanupWork(deliveryWork, environmentCustomImages, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped, config.heartbeatSchedulerIntervalMs).ready); @@ -1887,14 +1890,6 @@ async function startServerWithDatabaseTeardown( logger.error({ err }, "periodic tool connection health sweep failed"); })); - trackHeartbeatSchedulerWork(secretProposals.sweepExpired() - .then((expired) => { - if (expired > 0) logger.warn({ expired }, "periodic secret proposal expiry scrubbed proposals"); - }) - .catch((err) => { - logger.error({ err }, "periodic secret proposal expiry sweep failed"); - })); - trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews().catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); trackHeartbeatSchedulerWork(app.locals.toolGateway.cleanupExpiredSessions().catch((err: unknown) => logger.error({ err }, "gateway token cleanup failed"))); diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 432cad76e7a..7e4a41a95d7 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -61,6 +61,7 @@ export async function deleteCompany(db: Db, id: string) { .where(eq(companies.id, id)).for("no key update"); if (!existing) return null; await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.secretProposalExpiry, id); await agentObservabilityService(tx as unknown as Db).removeForCompanyDeletion(id); // Finance can reference costs and both can reference runs. Incidents // reference policies and approvals; delete these dependents first. diff --git a/server/src/services/company-work-batch.ts b/server/src/services/company-work-batch.ts new file mode 100644 index 00000000000..f83d2dcd5a5 --- /dev/null +++ b/server/src/services/company-work-batch.ts @@ -0,0 +1,28 @@ +export function takeCompanyWorkBatch( + ordered: readonly T[], + limit: number, + afterCompany: string | null = null, +): { entries: T[]; lastCompany: string | null } { + const grouped = new Map(); + for (const entry of ordered) { + const group = grouped.get(entry.companyId); + if (group) group.push(entry); else grouped.set(entry.companyId, [entry]); + } + const companies = [...grouped.keys()].sort(); + const next = afterCompany === null ? 0 : companies.findIndex(id => id > afterCompany); + const start = next < 0 ? 0 : next; + const rotation = [...companies.slice(start), ...companies.slice(0, start)]; + const entries: T[] = []; + let lastCompany = afterCompany; + for (let round = 0; entries.length < limit; round++) { + let added = false; + for (const companyId of rotation) { + const entry = grouped.get(companyId)![round]; + if (!entry) continue; + entries.push(entry); lastCompany = companyId; added = true; + if (entries.length === limit) break; + } + if (!added) break; + } + return { entries, lastCompany }; +} diff --git a/server/src/services/custom-image-cleanup-work.ts b/server/src/services/custom-image-cleanup-work.ts index 9ecd4e73612..feee4ebefa7 100644 --- a/server/src/services/custom-image-cleanup-work.ts +++ b/server/src/services/custom-image-cleanup-work.ts @@ -1,5 +1,6 @@ import { logger } from "../middleware/logger.js"; import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { registerDeadlineWork } from "./deadline-work.js"; import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import type { environmentCustomImageService } from "./environment-custom-images.js"; @@ -9,27 +10,13 @@ export function registerCustomImageCleanupWork( canRun: () => boolean, retryMs = 30_000, ) { - let requested = 1, reconciled = 0; - let next: number | null = null; - let earliest = 0; - return coordinator.register(DELIVERY_QUEUES.customImageExpiry, { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.customImageExpiry, { retryMs, - pausePassiveMaintenance: true, - onSettled: () => { requested++; }, + canRun, run: async () => { - if (!canRun()) { requested++; return; } - if (Date.now() < earliest) { next = earliest; return; } - const generation = ++requested; - earliest = Date.now() + retryMs; const result = await service.cleanupExpiredSetupSessions(); if (result.timedOut > 0 || result.failed > 0) logger.warn(result, "environment customImage setup cleanup changed sessions"); - const deadline = await service.nextSetupCleanupAt(); - // Back off failed or still-overdue work, not a successfully read future deadline. - if (deadline === null || deadline > Date.now()) earliest = 0; - next = deadline === null ? null : Math.max(earliest, deadline); - reconciled = generation; }, - hasPending: async () => requested !== reconciled, - nextRunAt: () => next, + next: () => service.nextSetupCleanupAt(), }); } diff --git a/server/src/services/deadline-work.ts b/server/src/services/deadline-work.ts new file mode 100644 index 00000000000..9f1de1b2a24 --- /dev/null +++ b/server/src/services/deadline-work.ts @@ -0,0 +1,36 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import type { DeliveryQueue } from "./delivery-work-notifications.js"; + +export function registerDeadlineWork( + coordinator: ReturnType, + queue: DeliveryQueue, + task: { + run: () => Promise; + next: () => Promise; + canRun: () => boolean; + retryMs: number; + }, +) { + let requested = 1, reconciled = 0; + let next: number | null = null; + let earliest = 0; + return coordinator.register(queue, { + retryMs: task.retryMs, + pausePassiveMaintenance: true, + onSettled: () => { requested++; }, + run: async () => { + if (!task.canRun()) { requested++; return; } + if (Date.now() < earliest) { next = earliest; return; } + const generation = ++requested; + earliest = Date.now() + task.retryMs; + await task.run(); + const deadline = await task.next(); + // Back off failed or still-overdue work, not a successfully read future deadline. + if (deadline === null || deadline > Date.now()) earliest = 0; + next = deadline === null ? null : Math.max(earliest, deadline); + reconciled = generation; + }, + hasPending: async () => requested !== reconciled, + nextRunAt: () => next, + }); +} diff --git a/server/src/services/delivery-work-coordinator.ts b/server/src/services/delivery-work-coordinator.ts index 0284e06aba2..0d1a88c9f59 100644 --- a/server/src/services/delivery-work-coordinator.ts +++ b/server/src/services/delivery-work-coordinator.ts @@ -26,6 +26,9 @@ export function createDeliveryWorkCoordinator(input: { onSettled?: (hints?: ReadonlySet) => void; // Optional maintenance/retry deadline restored from durable state. nextRunAt?: () => number | null; + // Opt in only when run has recorded per-item failure backoffs. Other + // eligible items may already be due, unlike an unchanged failed deadline. + nextRunAtOnError?: () => number | null; // An empty queue's maintenance may wait until dispatch admission reopens. // Notifications and unresolved producer writes still reconcile in drain. pausePassiveMaintenance?: boolean; @@ -65,9 +68,10 @@ export function createDeliveryWorkCoordinator(input: { dirty = false; const finish = beginIdleTrackedWork(); const attempt = new AbortController(); + let runFailed = false; controller = attempt; running = Promise.resolve().then(() => reconcileDatabaseWork(input.owner, queue)) - .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal)).then(() => workerStopped ? false : task.hasPending()).then(pending => { + .then(() => workerStopped || !(input.canRun() || (task.runDuringDrain && input.canReconcile?.())) ? undefined : task.run(attempt.signal).catch(error => { runFailed = true; throw error; })).then(() => workerStopped ? false : task.hasPending()).then(pending => { pendingWork = pending || databaseWorkPending(input.owner, queue); if (pendingWork) schedule(task.retryMs); else { finishIntent?.(); finishIntent = undefined; } @@ -78,9 +82,17 @@ export function createDeliveryWorkCoordinator(input: { } }).catch(error => { schedule(task.retryMs); + let failure = error; + try { + const scopedRetry = runFailed && task.nextRunAtOnError; + const next = input.canRun() ? (scopedRetry ? scopedRetry() : task.nextRunAt?.()) : null; + if (next != null && (scopedRetry || next > Date.now())) schedule(0, Math.max(Date.now(), next)); + } catch (deadlineError) { + failure = new AggregateError([error, deadlineError], "Delivery failure deadline could not be read"); + } // Logging must never turn a recoverable sweep failure into an // unhandled rejection or prevent other queues from running. - try { if (!workerStopped) input.onError(error, queue); } catch { /* Recovery remains scheduled. */ } + try { if (!workerStopped) input.onError(failure, queue); } catch { /* Recovery remains scheduled. */ } }).finally(() => { running = null; controller = null; diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index f6d50c71930..b6bed1d0f25 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + secretProposalExpiry: "secret-proposal-expiry", customImageExpiry: "custom-image-expiry", historicalCleanup: "historical-cleanup", adapterLoginCleanup: "adapter-login-cleanup", diff --git a/server/src/services/issue-thread-interactions.ts b/server/src/services/issue-thread-interactions.ts index bcfbe2882a9..74e8d8216db 100644 --- a/server/src/services/issue-thread-interactions.ts +++ b/server/src/services/issue-thread-interactions.ts @@ -1159,6 +1159,7 @@ async function resolveLinkedSecretProposal( ) { const proposalId = linkedSecretProposalId(interaction); if (!proposalId) return; + await notifyDeliveryWork(db, DELIVERY_QUEUES.secretProposalExpiry, interaction.companyId); const [proposal] = await db .update(companySecretProposals) .set({ @@ -3181,6 +3182,7 @@ export function issueThreadInteractionService( : execution.status; if (executionStatus === "failed" && proposal.status === "pending") { const resolutionReason = `Interaction acceptance failed: ${execution.errorCode ?? "secret_proposal_execution_failed"}`; + await notifyDeliveryWork(tx, DELIVERY_QUEUES.secretProposalExpiry, proposal.companyId); await tx .update(companySecretProposals) .set({ diff --git a/server/src/services/scoped-deadline-work.ts b/server/src/services/scoped-deadline-work.ts new file mode 100644 index 00000000000..6de9d3895da --- /dev/null +++ b/server/src/services/scoped-deadline-work.ts @@ -0,0 +1,99 @@ +import { takeCompanyWorkBatch } from "./company-work-batch.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import type { DeliveryQueue } from "./delivery-work-notifications.js"; + +type ScheduledEntry = { id: string; companyId: string; dueAt: number }; + +/** Coalesced scope reconstruction and fair deadline dispatch. Domain owners + * supply persistence, eligibility, and execution; this helper owns no SQL. */ +export function registerScopedDeadlineWork( + coordinator: ReturnType, + options: { + queue: DeliveryQueue; + batchSize?: number; + minimumPassIntervalMs?: number; + decodeScope(hint: string): Scope | undefined; + matches(entry: Entry, scope?: Scope): boolean; + load(scope?: Scope, signal?: AbortSignal): Promise; + check(entry: Entry, signal?: AbortSignal): Promise; + }, + canRun: () => boolean, +) { + const dirty = new Set(["*"]); + const entries = new Map(); + const failures = new Map(); + const scopeFailures = new Map(); + let earliest = 0; + let lastCompany: string | null = null; + function failed(map: typeof failures, id: string) { + const count = Math.min((map.get(id)?.count ?? 0) + 1, 5); + map.set(id, { count, retryAt: Date.now() + Math.min(300_000, 30_000 * 2 ** (count - 1)) }); + } + function dirtyScopes() { + return [...dirty].map(hint => { try { return options.decodeScope(hint); } catch { return undefined; } }); + } + function next() { + if (!canRun()) return dirty.size > 0 || entries.size > 0 ? Date.now() + 30_000 : null; + let nextAt: number | null = null; + const take = (at: number) => { nextAt = nextAt === null ? at : Math.min(nextAt, at); }; + for (const hint of dirty.has("*") ? ["*"] : dirty) take(Math.max(earliest, scopeFailures.get(hint)?.retryAt ?? 0)); + const unavailable = dirtyScopes(); + for (const entry of entries.values()) if (!unavailable.some(scope => options.matches(entry, scope))) + take(Math.max(earliest, entry.dueAt, failures.get(entry.id)?.retryAt ?? 0)); + return nextAt; + } + const worker = coordinator.register(options.queue, { + retryMs: 30_000, + pausePassiveMaintenance: true, + onSettled: hints => { for (const hint of hints ?? ["*"]) dirty.add(hint); }, + run: async signal => { + if (!canRun() || Date.now() < earliest) return; + earliest = Date.now() + (options.minimumPassIntervalMs ?? 1000); + const errors: unknown[] = []; + const requested = (dirty.has("*") ? ["*"] : [...dirty]).filter(hint => (scopeFailures.get(hint)?.retryAt ?? 0) <= Date.now()); + const covered = requested.includes("*") ? [...dirty] : requested; + for (const hint of covered) dirty.delete(hint); + for (let index = 0; index < requested.length; index++) { + if (signal.aborted || !canRun()) { for (const hint of requested.includes("*") ? covered : requested.slice(index)) dirty.add(hint); return; } + const hint = requested[index]; + let scope: Scope | undefined; + let decoded = false; + try { + scope = options.decodeScope(hint); decoded = true; + const rows = await options.load(scope, signal), retained = new Set(rows.map(row => row.id)); + for (const entry of entries.values()) if (options.matches(entry, scope) && !retained.has(entry.id)) { + entries.delete(entry.id); failures.delete(entry.id); + } + for (const entry of rows) entries.set(entry.id, entry); + if (!scope) scopeFailures.clear(); else scopeFailures.delete(hint); + } catch (error) { + const retryHint = decoded ? hint : "*"; + dirty.add(retryHint); failed(scopeFailures, retryHint); errors.push(error); + } + } + const unavailable = dirtyScopes(); + const due = [...entries.values()].filter(entry => !unavailable.some(scope => options.matches(entry, scope)) + && Math.max(entry.dueAt, failures.get(entry.id)?.retryAt ?? 0) <= Date.now()) + .sort((a, b) => a.dueAt - b.dueAt || a.id.localeCompare(b.id)); + const batch = takeCompanyWorkBatch(due, options.batchSize ?? 25, lastCompany); lastCompany = batch.lastCompany; + for (const entry of batch.entries) { + if (signal.aborted || !canRun()) break; + try { + const current = await options.check(entry, signal); + if (current) entries.set(entry.id, current); else entries.delete(entry.id); + failures.delete(entry.id); + } catch (error) { failed(failures, entry.id); errors.push(error); } + } + if (errors.length) throw new AggregateError(errors, `${options.queue} could not settle`); + }, + // Retain scopes/deadlines for resume without retaining a settled write's + // idle hold during drain. The coordinator still fences unresolved XIDs. + hasPending: async () => canRun() && dirty.size > 0, + nextRunAt: next, + nextRunAtOnError: next, + }); + return { ...worker, invalidate(hint = "*") { + dirty.add(hint); + worker.wake(); + } }; +} diff --git a/server/src/services/secret-proposal-expiry-work.ts b/server/src/services/secret-proposal-expiry-work.ts new file mode 100644 index 00000000000..1a019047c92 --- /dev/null +++ b/server/src/services/secret-proposal-expiry-work.ts @@ -0,0 +1,29 @@ +import { logger } from "../middleware/logger.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerScopedDeadlineWork } from "./scoped-deadline-work.js"; +import type { createSecretProposalsService } from "./secret-proposals.js"; + +export function registerSecretProposalExpiryWork( + coordinator: ReturnType, + service: Pick, "sweepExpiredForCompany" | "expiryDeadlines">, + canRun: () => boolean, +) { + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.secretProposalExpiry, + batchSize: 1, + // Saved expiry deadlines are authoritative, including deadlines less than + // a second away. Only actual failures require a recovery backoff. + minimumPassIntervalMs: 0, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: (entry, companyId?: string) => companyId === undefined || entry.companyId === companyId, + load: async (companyId?: string) => (await service.expiryDeadlines(companyId)) + .map(row => ({ ...row, id: row.companyId })), + check: async entry => { + const expired = await service.sweepExpiredForCompany(entry.companyId); + if (expired > 0) logger.warn({ companyId: entry.companyId, expired }, "secret proposal expiry scrubbed proposals"); + const [next] = await service.expiryDeadlines(entry.companyId); + return next ? { ...next, id: next.companyId } : null; + }, + }, canRun); +} diff --git a/server/src/services/secret-proposals.ts b/server/src/services/secret-proposals.ts index fbefd53d5fd..05f5af71d91 100644 --- a/server/src/services/secret-proposals.ts +++ b/server/src/services/secret-proposals.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { updateAgentConfigurationInTransaction } from "./agent-configuration-transaction.js"; import { withAgentAppearance } from "@paperclipai/shared"; import { and, count, desc, eq, gte, inArray, lte, or, sql } from "drizzle-orm"; @@ -144,6 +145,7 @@ export function createSecretProposalsService(db: Db) { }); return { denial, value: null }; } + await notifyDeliveryWork(txDb, DELIVERY_QUEUES.secretProposalExpiry, input.companyId); return { denial: null, value: await create(txDb) }; }); if (result.denial) throw unprocessable(result.denial.message); @@ -619,6 +621,7 @@ export function createSecretProposalsService(db: Db) { appliedBindingConfigPath?: string | null; }) { const now = new Date(); + await notifyDeliveryWork(txDb, DELIVERY_QUEUES.secretProposalExpiry, proposal.companyId); const updated = await txDb.update(companySecretProposals).set({ status: "approved", resolvedByUserId: input.resolvedByUserId, @@ -794,6 +797,7 @@ export function createSecretProposalsService(db: Db) { const now = new Date(); return db.transaction(async (tx) => { const txDb = tx as unknown as Db; + await notifyDeliveryWork(tx, DELIVERY_QUEUES.secretProposalExpiry, companyId); const updated = await tx.update(companySecretProposals).set({ status, resolvedByUserId: input.resolvedByUserId ?? null, @@ -868,10 +872,12 @@ export function createSecretProposalsService(db: Db) { limit = DEFAULT_EXPIRY_SWEEP_LIMIT, expireProposal: (companyId: string, proposalId: string) => Promise = (companyId, proposalId) => transition(companyId, proposalId, "expired", { reason: "Pending proposal expired" }), + companyId?: string, ) { const expired = await db.select({ id: companySecretProposals.id, companyId: companySecretProposals.companyId }) .from(companySecretProposals) - .where(and(eq(companySecretProposals.status, "pending"), lte(companySecretProposals.expiresAt, now))) + .where(and(eq(companySecretProposals.status, "pending"), lte(companySecretProposals.expiresAt, now), + companyId === undefined ? undefined : eq(companySecretProposals.companyId, companyId))) .orderBy(companySecretProposals.expiresAt) .limit(limit); let expiredCount = 0; @@ -887,5 +893,25 @@ export function createSecretProposalsService(db: Db) { return expiredCount; } - return { getById, view: enrich, createSecret, createBinding, listForAgent, listForBoard, assertBindingSnapshotCurrent, approve, transition, sweepExpired }; + async function sweepExpiredForCompany(companyId: string) { + return sweepExpired(new Date(), DEFAULT_EXPIRY_SWEEP_LIMIT, undefined, companyId); + } + + async function expiryDeadlines(companyId?: string) { + const rows = await db.select({ companyId: companySecretProposals.companyId, + at: sql`min(${companySecretProposals.expiresAt})` }) + .from(companySecretProposals) + .where(and(eq(companySecretProposals.status, "pending"), + companyId === undefined ? undefined : eq(companySecretProposals.companyId, companyId))) + .groupBy(companySecretProposals.companyId); + return rows.map(row => ({ companyId: row.companyId, dueAt: new Date(row.at).getTime() })); + } + + async function nextExpiryAt(): Promise { + const [row] = await db.select({ at: sql`min(${companySecretProposals.expiresAt})` }) + .from(companySecretProposals).where(eq(companySecretProposals.status, "pending")); + return row?.at == null ? null : new Date(row.at).getTime(); + } + + return { getById, view: enrich, createSecret, createBinding, listForAgent, listForBoard, assertBindingSnapshotCurrent, approve, transition, sweepExpired, sweepExpiredForCompany, expiryDeadlines, nextExpiryAt }; } From 3cd3af5d75f200185ed4d9703a3145594a489b13 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:53:32 -0500 Subject: [PATCH 03/41] Trace native preparation and OpenCode execution phases (#17918) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Each native run prepares its workspace and skills, then starts its provider. > - A long preparation span does not show whether inventory refresh, a cache lock, repository reads, or file checks caused the delay. > - One hosted run spent 81 seconds in this phase. The same deployed service and pinned files took 2.55 seconds in a separate diagnostic process. > - Another hosted turn spent 69 seconds before its first output. A matched public-workspace local model replay took 7.64 seconds. > - We need skill preparation and provider-internal measurements before choosing another fix. > - This pull request adds skill spans, a real-service benchmark, and opt-in filtered provider timing. > - The benefit is a faster and more precise diagnosis loop. ## Linked Issues or Issue Description Refs #17620. That PR owns the batched read optimization. This PR adds diagnostics and does not change its read policy. Refs #17907. That PR protects long cache lock transactions from the database idle timeout. **What happened?** Native skill configuration can take much longer than matched standalone resolution. Provider execution also has long gaps with no internal timings. Existing spans cannot separate inventory refresh, cache acquisition, repository reads, context loading, Git snapshots, or model streaming. **Expected behavior** An operator-configured trace shows these phases under the existing run context. Without an OTLP endpoint, the new spans stay off. A local benchmark can reproduce the service path with real PostgreSQL, files, an agent actor, and a responsible user. **Steps to reproduce** 1. Run `pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts --actor=agent --staging-shape=true`. 2. Inspect cold and warm phase timings, SQL counts, and file verification results. 3. Set the existing operator OTLP configuration and add `--export-otlp=true` to inspect the same hierarchy in the collector. ## What Changed - Add bounded skill inventory, refresh wait, cache acquisition, row, version, repository, file check, and consumer spans. - Keep the existing endpoint gate, attribute allowlist, cache locks, authorization, file validation, and durable pin boundary. - Add a standalone benchmark with a full repository fixture, immutable snapshot input, calibrated local transport delay, and optional OTLP export. - Verify every delivered file and executable bit outside the timed section. Create its fixture agent through the lifecycle API. - Add an opt-in authenticated loopback sink for OpenCode. Discard raw attributes, logs and content. Retain only closed phase names and timestamps. - Carry one bounded diagnostic batch through the real proxy, Rust normalization, durable journal and server trace projection. Group provider phases under their execution parent. - Freeze the Date clock in an existing delegated Slack rate-limit test. This fixes a CI failure when consecutive requests cross a rate-window boundary. It does not change rate-limit behavior. - Document phase boundaries and the difference between SDK startup and confirmed ingestion. ## Verification - Skill service, runtime selection, concurrency, cache and timing suites: 122 tests passed with real PostgreSQL and migrations. Enabled-span integration tests also passed. They verify parent relationships and a real advisory-lock timeout. - A captured 208-file repository produced 209 verified runtime files. The benchmark uses real agent and responsible-user authorization. The lifecycle fixture correction passed with the staging-shaped inventory. - Affected OpenCode driver, collector and transport tests: 72 passed. The tests cover authentication, content removal, size/count bounds, disabled behavior and the rebuilt native Runner boundary. One Rust projection test and 18 server trace tests also passed. A held run-log sink proves that a full 128-span diagnostic batch adds no duplicate database writes before terminal consumption. - The two CI-failing delegated Slack rate-limit cases passed against real PostgreSQL with only the Date clock frozen. - Runner TypeScript and server TypeScript checks passed. Rust formatting and module-boundary checks passed. - Real Honeycomb readback confirmed the 14-span skill waterfall and a 40-span provider waterfall. Provider phases have the correct execution parent. The provider export uses synthetic model replies and is instrumentation proof only. - Earlier full local checks failed on missing package-local executables, then on the default Node heap and a missing generated service worker. The dependency links were repaired. The server check passed with a larger heap. The full local test attempt could not start embedded PostgreSQL on this host and was stopped after retaining its failures; real database coverage uses an isolated database on an existing local server. These attempts do not count as a full local pass. - CI on the prior revision passed typecheck, build, Runner and all E2E shards. It failed the benchmark lifecycle boundary and one existing rate-window test. Those fixes passed CI. A later Runner assertion exposed the Copilot source identity pin on the shared transport file. The diagnostic flag now uses the existing configured-environment projection, preserving that file byte-for-byte. All 66 selected installation and native transport tests passed. New-head CI and review remain required. - The under-15-second hosted response target remains unmet. This PR adds diagnostics and does not claim a hosted latency fix. ## Risks - Skill spans and Honeycomb export retain the operator endpoint gate. OpenCode phase collection requires a separate explicit diagnostic flag. Without an endpoint, its closed measurements stay in the instance run log. - Raw OpenCode OTLP can contain prompts and authorization attributes. The new sink rejects unauthenticated requests and discards all attributes before any measurement reaches PRP or the operator collector. Request, event and per-phase limits bound memory and journal growth. - Collection does not wait for a provider flush. Late or oversized exports can be absent; these measurements are not a complete profiling guarantee. - The benchmark's inventory contents and transport delay are explicit fixtures. They do not reproduce server contention or prove hosted response time. - The one durable diagnostic batch remains the local evidence. Its child span export does not add per-phase run-log rows. - No database migration, authorization policy, model selection or inference transport changes. The Rust Runner and OpenCode proxy must be shipped together for the new opt-in diagnostic. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell execution, browser tools, and Honeycomb queries. The session does not expose a verifiable backend deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip Co-authored-by: Paperclip --- doc/DEVELOPING.md | 35 +++ doc/observability.md | 43 +++ .../crates/runner-core/src/codex_provider.rs | 1 + .../crates/runner-core/src/provider_events.rs | 63 +++++ .../src/cli/opencode-app-server-proxy.ts | 8 + .../src/contracts/provider-performance.ts | 43 +++ .../durable-prp-control-plane.ts | 1 + .../opencode/opencode-server-driver.ts | 23 ++ .../opencode/performance-diagnostics.test.ts | 65 +++++ .../opencode/performance-diagnostics.ts | 103 +++++++ packages/paperclip-runner/src/index.ts | 1 + .../src/live/runnerd-codex-transport.test.ts | 19 +- .../test/fixtures/fake-opencode-server.mjs | 18 ++ .../benchmark-native-skill-preparation.ts | 260 ++++++++++++++++++ .../company-skills-observability.test.ts | 151 ++++++++++ .../connection-management.integration.test.ts | 4 +- server/src/services/company-skills.ts | 31 ++- .../execution-preparation-timing.test.ts | 3 +- .../services/execution-preparation-timing.ts | 4 + .../native-runtime/native-run-trace.test.ts | 34 +++ .../native-runtime/native-run-trace.ts | 25 +- .../native-runtime/native-session-executor.ts | 5 + server/src/services/skill-version-cache.ts | 8 +- server/src/vendor/paperclip-runner/index.ts | 1 + 24 files changed, 930 insertions(+), 19 deletions(-) create mode 100644 packages/paperclip-runner/src/contracts/provider-performance.ts create mode 100644 packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.test.ts create mode 100644 packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.ts create mode 100644 server/scripts/benchmark-native-skill-preparation.ts create mode 100644 server/src/__tests__/company-skills-observability.test.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 0afd307559a..05071cb68ff 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -838,6 +838,41 @@ Set `SEARCH_EVAL_SCALE=1` to additionally measure a disposable 10,000-task, 30,000-comment dataset. `SEARCH_EVAL_REPORT=/tmp/search-quality.json` saves per-query results and latency measurements; scale measurements are opt-in. +### Native skill preparation benchmark + +Measure native skill preparation through the production service, real PostgreSQL, +and an in-memory OpenTelemetry exporter: + +```sh +pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts +``` + +The default fixture has a 208-file repository of about 10.8 MB. The report includes +cold and warm resolution, a fresh service instance, detailed phase timings, and +regular-pool SQL statement counts (excluding the separate cache-lock connection). +Every delivered file is checked outside the timed section. No provider is started. +`--snapshot-file=/absolute/path/snapshot.json --entrypoint=skills/example/SKILL.md` +uses a captured repository file inventory instead. `--network-delay-ms=25` adds +25 ms in each direction through a local TCP relay; this is simulated transport +latency, not a measurement of a hosted database. + +`--actor=agent` adds a real agent, run, and responsible user, then records version +pins through the same callback boundary as execution. `--staging-shape=true` +seeds 14 projects, 10 sources, 8 bindings, and 156 other skills. These match one +observed staging inventory's counts; their content is synthetic. Combine these +options to include authorization and unselected library rows in the measurement. + +By default, spans stay in memory. `--export-otlp=true` also exports them using +the operator's `OTEL_EXPORTER_OTLP_ENDPOINT`, protocol, and headers. Keep header +values out of command arguments and reports. The report records source hashes, +the export mode, and trace IDs. Exported traces identify this local diagnostic +in their service version; they are not evidence of staging performance. + +If creating another embedded cluster is impractical, `--postgres-port=55448 +--postgres-user=paperclip` uses an existing **local** server. It creates and migrates +a uniquely named disposable database and drops that database afterward; it does +not reuse an application database. + ### Repository workspace admission benchmark Measure a fresh checkout with and without the repository object cache: diff --git a/doc/observability.md b/doc/observability.md index 55328507d0c..324273807ec 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -35,6 +35,13 @@ maintains as a dependency; once you install the packages below, they become normal dependencies of **your own** project, and your own Dependabot updates them. +An initialized SDK does not confirm ingestion. Verify a fresh trace from the +deployed revision at the configured destination. A rejected export can leave +the application healthy. For Honeycomb, configure the ingest key in the +`x-honeycomb-team` entry of `OTEL_EXPORTER_OTLP_HEADERS`; see +[Honeycomb authentication](https://docs.honeycomb.io/api/query-annotations/authorization). +Keep credential values out of logs and diagnostic reports. + ## Budget admission, accounting, and enforcement The operator-gated `paperclip.budgets` tracer records @@ -313,6 +320,9 @@ attempt or owning phase: | `runtime_selection.*` | Completion contracts, session/recovery reads, profiles, interaction materialization, backend discovery, bootstrap, run/coordinator writes, transaction entry/settlement, and workspace synchronization. | | `runtime_context.instructions`, `.skills`, `.mcp` | The three concurrent runtime-context branches. Their durations overlap. | | `runtime_context.asset_*` | Source reads, manifest hashing, cache lookup, cold materialization, and exact file verification on cold and warm paths. | +| `skills.inventory`, `.wait`, `.bundled`, `.releases`, `.folders`, `.local_sources` | Company skill reconciliation, waiting for an existing company refresh, and its bundled imports, release verification, folder, and local-source work beneath runtime skill preparation. Inventory children overlap their parent. | +| `skills.cache.acquire`, `skills.rows`, `skills.version.read`, `skills.repository.read`, `skills.version.materialize`, `skills.version.verify` | Dedicated connection/transaction startup plus advisory-lock acquisition, authorized skill rows, frozen version and repository reads, and exact runtime materialization/validation. Lock acquisition ends before the protected work; version verification is inside materialization. Only bounded file counts are added. | +| `skills.consume` | The caller's delivery and durable version-pinning callback while the cache lock remains held. | | `workspace_sync.prepare_host_current`, `.prepare_durable_seed`, `.prepare_adopt_remote` | Runtime preparation under the selected inbound mode, including snapshot and staging work. A warm mismatch may produce a second preparation; it is real repeated work. | | `workspace_sync.source_root`, `.repositories`, `.ignore_rules`, `.snapshot_git`, `.snapshot_baseline` | Physical source capture, managed repositories, ignore rules, Git metadata and baseline content snapshot. | | `workspace_sync.pack`, `.git_clone`, `.git_archive`, `.overlay_copy`, `.overlay_archive`, `.deleted_manifest` | Temporary shallow Git clone, its archive callback, selected working-tree overlay copy/archive, and the deleted-file manifest. The archive is inside the clone's lifetime; do not sum parent and child. | @@ -334,6 +344,33 @@ attempt or owning phase: | `response.resolve`, `.authorize`, `.add_comment`, `.transaction*`, `.row_lock`, `.write`, `.publish` | Response selection, existing authorization, comment transaction including commit or rollback, row wait, insert, and external publication enqueue. | | `execution.realtime` | Synchronous live-event and plugin publication dispatch. It does not measure browser receipt or paint. | +For bounded native OpenCode diagnosis, set +`PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` as an agent/environment variable. +The existing configured-environment projection carries it into the native Runner. +It defaults off. The Runner creates an authenticated loopback OTLP sink and enables +the pinned OpenCode runtime's internal tracing. The sink discards logs, resource +attributes, span attributes, events, and unknown span names. It retains only closed +phase names, wall timestamps, and success/failure, with a 1 MiB request limit, +128 retained measurements, and 16 measurements per phase. No operator OTLP endpoint +or ingestion credential is passed to OpenCode. + +Available phases cover context/instruction loading, skill discovery, tool and model +resolution, model streaming (`ai.streamText`, including stream consumption), Git +discovery/default branch, snapshots, dependency installation, and plugin startup. +These intervals overlap; do not sum them as elapsed time. At each terminal turn, +already-received measurements travel in one `harness.diagnostic` event through the +existing durable PRP channel. Late exports may be absent; collection never waits +for a provider flush or delays terminal publication to complete the diagnostic. +Late spans from a prior completed turn are discarded at the next boundary. + +The server validates the closed fields again and records them under a foldable +`provider.runtime` group inside `native.session.execute`. The normal operator +endpoint gate controls Honeycomb export. The single diagnostic batch stays in the instance run log. Its OTel projection +does not write duplicate phase rows or await additional database work before +terminal consumption. Without an endpoint, no provider child spans are exported. Raw OpenCode OTLP must +never be sent directly to the operator endpoint: it can contain prompts and +authorization attributes before the loopback sink removes them. + Detail phases emit at most 16 spans of each name per attempt. Larger receipt and asset inventories continue under the containing group without unbounded per-row or per-file spans. Adding detail spans does not itself change work, query order, @@ -1954,6 +1991,12 @@ It carries no skill contents, identifiers, locations, or credentials. It uses th existing run performance events and operator-configured OpenTelemetry endpoint; no first-party Telemetry event is added. +The `skills.*` detail spans above distinguish inventory reconciliation and lock +acquisition from repository transfer and file validation. A repository-backed +skill's authored `SKILL.md` is not its complete runtime payload: the generated +discovery entrypoint includes the pinned repository under `.paperclip-repository`. +Measure the delivered bundle's file count and bytes when reproducing its cost. + Runtime preparation refreshes the company inventory once per listing. Local and catalog directories remain direct sources, so edits are visible on the next preparation. Explicit version selections still use their stored snapshots. diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs index 70aa6735095..3ec890aa246 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/codex_provider.rs @@ -43,6 +43,7 @@ const OPENCODE_PROVIDER_ENVIRONMENT_KEYS: &[&str] = &[ "PAPERCLIP_NATIVE_MCP_TOKEN", "PAPERCLIP_OPENCODE_PERMISSION_MODE", "PAPERCLIP_OPENCODE_RUNTIME_DIR", + "PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS", "PAPERCLIP_RUNNER_INSTANCE_ID", "PAPERCLIP_RUN_ID", "PAPERCLIP_NORMALIZED_SESSION_ID", diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs b/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs index 5e1bc498751..755ad5402d5 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/provider_events.rs @@ -671,6 +671,27 @@ fn measurement(value: &Value) -> Value { /// Converts Codex app-server notifications into provider-neutral PRP events. /// Provider-native envelopes are consumed here and never cross the PRP boundary. +fn provider_performance_spans(params: &Value) -> Vec { + let mut counts = std::collections::BTreeMap::<&str, usize>::new(); + params.get("spans").and_then(Value::as_array).into_iter().flatten().take(128).filter_map(|span| { + let name = span.get("name")?.as_str()?; + if !matches!(name, + "provider.context.environment" | "provider.context.instructions" | "provider.skills.discovery" + | "provider.tools.resolve" | "provider.model.resolve" | "provider.model.stream" + | "provider.workspace.snapshot.track" | "provider.workspace.snapshot.patch" | "provider.workspace.snapshot.diff" + | "provider.workspace.git.discover" | "provider.workspace.git.default_branch" + | "provider.dependencies.install" | "provider.plugins.initialize") { return None; } + let start = span.get("startedAtMs")?.as_f64()?; + let end = span.get("endedAtMs")?.as_f64()?; + if !start.is_finite() || !end.is_finite() || start < 0.0 || end < start || end - start > 86_400_000.0 { return None; } + let count = counts.entry(name).or_default(); + if *count >= 16 { return None; } + *count += 1; + Some(json!({"name":name,"startedAtMs":start,"endedAtMs":end, + "outcome":if span.get("outcome").and_then(Value::as_str)==Some("failed") {"failed"} else {"ok"}})) + }).collect() +} + pub fn normalize_codex_notification(method: &str, params: &Value) -> Vec { let mut events = Vec::new(); let push = |events: &mut Vec, @@ -685,6 +706,17 @@ pub fn normalize_codex_notification(method: &str, params: &Value) -> Vec { + let spans = provider_performance_spans(params); + if !spans.is_empty() { + push( + &mut events, + "harness.diagnostic", + EventPriority::P1, + json!({"code":"provider_performance_spans","spans":spans}), + ); + } + } "thread/compacted" => push( &mut events, "context.compacted", @@ -1464,6 +1496,37 @@ fn has_rfc_uri_scheme_prefix(value: &str) -> bool { mod tests { use super::*; + #[test] + fn performance_diagnostics_keep_only_bounded_content_free_measurements() { + let mut spans = vec![json!({"name":"private prompt","startedAtMs":1,"endedAtMs":2})]; + for _ in 0..200 { + spans.push( + json!({"name":"provider.model.stream","startedAtMs":1000,"endedAtMs":1010, + "outcome":"failed","authorization":"private","attributes":{"prompt":"private"}}), + ); + } + let events = + normalize_codex_notification("paperclip/providerPerformance", &json!({"spans":spans})); + assert_eq!(events.len(), 1); + assert_eq!(events[0].event_type, "harness.diagnostic"); + assert_eq!(events[0].priority, EventPriority::P1); + let rows = events[0].payload["spans"].as_array().unwrap(); + assert_eq!(rows.len(), 16); + assert_eq!( + rows[0], + json!({"name":"provider.model.stream","startedAtMs":1000.0,"endedAtMs":1010.0,"outcome":"failed"}) + ); + assert!(!events[0].payload.to_string().contains("private")); + assert!(normalize_codex_notification( + "paperclip/providerPerformance", + &json!({"spans":[ + {"name":"provider.model.stream","startedAtMs":10,"endedAtMs":1}, + {"name":"provider.model.stream","startedAtMs":0,"endedAtMs":86400001} + ]}) + ) + .is_empty()); + } + #[test] fn unrelated_information_retains_only_bounded_run_log_diagnostics() { let events = normalize_codex_notification( diff --git a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts index 971c15f0e59..7737ef2ee20 100644 --- a/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts +++ b/packages/paperclip-runner/src/cli/opencode-app-server-proxy.ts @@ -30,6 +30,7 @@ import { withoutAmbientOpenCodeCommand, } from "./opencode-proxy-command.js"; import { OpenCodeProxyUsageLedger } from "./opencode-proxy-usage.js"; +import { parseProviderPerformanceSpans, PROVIDER_PERFORMANCE_DIAGNOSTIC } from "../contracts/provider-performance.js"; type RpcMessage = { id?: string | number; @@ -307,6 +308,13 @@ async function pumpEvents(opened: HarnessSession): Promise { ); activeTurnId = null; } else if (event.eventType === "harness.diagnostic") { + if (payload.code === PROVIDER_PERFORMANCE_DIAGNOSTIC) { + send({method: "paperclip/providerPerformance", params: { + threadId: opened.ids().driverSessionId, turnId: event.turnId, + spans: parseProviderPerformanceSpans(payload, Date.now()), + }}); + continue; + } send({ method: "warning", params: { diff --git a/packages/paperclip-runner/src/contracts/provider-performance.ts b/packages/paperclip-runner/src/contracts/provider-performance.ts new file mode 100644 index 00000000000..bc5c6ef279b --- /dev/null +++ b/packages/paperclip-runner/src/contracts/provider-performance.ts @@ -0,0 +1,43 @@ +/** Content-free provider diagnostics carried by the existing PRP diagnostic event. */ +export const PROVIDER_PERFORMANCE_DIAGNOSTIC = "provider_performance_spans"; +export const PROVIDER_PERFORMANCE_ENV = "PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS"; +export const MAX_PROVIDER_PERFORMANCE_SPANS = 128; +export const PROVIDER_PERFORMANCE_PHASES = [ + "provider.context.environment", "provider.context.instructions", "provider.skills.discovery", + "provider.tools.resolve", "provider.model.resolve", "provider.model.stream", + "provider.workspace.snapshot.track", "provider.workspace.snapshot.patch", "provider.workspace.snapshot.diff", + "provider.workspace.git.discover", "provider.workspace.git.default_branch", + "provider.dependencies.install", "provider.plugins.initialize", +] as const; +export type ProviderPerformancePhase = typeof PROVIDER_PERFORMANCE_PHASES[number]; +export interface ProviderPerformanceSpan { + name: ProviderPerformancePhase; + startedAtMs: number; + endedAtMs: number; + outcome: "ok" | "failed"; +} + +const names = new Set(PROVIDER_PERFORMANCE_PHASES); +const maxDurationMs = 24 * 60 * 60 * 1000; + +/** Project fields rather than forwarding provider attributes or untrusted diagnostic data. */ +export function parseProviderPerformanceSpans(payload: unknown, observedAtMs: number): ProviderPerformanceSpan[] { + if (!payload || typeof payload !== "object" || Array.isArray(payload)) return []; + const diagnostic = payload as Record; + if (diagnostic.code !== PROVIDER_PERFORMANCE_DIAGNOSTIC || !Array.isArray(diagnostic.spans)) return []; + const counts = new Map(); + return diagnostic.spans.slice(0, MAX_PROVIDER_PERFORMANCE_SPANS).flatMap((value): ProviderPerformanceSpan[] => { + if (!value || typeof value !== "object" || Array.isArray(value)) return []; + const row = value as Record; + const { name, startedAtMs, endedAtMs } = row; + if (typeof name !== "string" || !names.has(name) || typeof startedAtMs !== "number" || typeof endedAtMs !== "number" + || !Number.isFinite(startedAtMs) || !Number.isFinite(endedAtMs) + || startedAtMs < observedAtMs - maxDurationMs || endedAtMs > observedAtMs + 1000 + || endedAtMs < startedAtMs || endedAtMs - startedAtMs > maxDurationMs) return []; + const count = counts.get(name) ?? 0; + if (count >= 16) return []; + counts.set(name, count + 1); + return [{ name: name as ProviderPerformancePhase, startedAtMs, endedAtMs, + outcome: row.outcome === "failed" ? "failed" : "ok" }]; + }); +} diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index c209a37bf95..4f09db52dd9 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -3503,6 +3503,7 @@ const runnerExplicitProviderEnvironmentKeys = [ "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE", "PAPERCLIP_OPENCODE_PERMISSION_MODE", "PAPERCLIP_OPENCODE_RUNTIME_DIR", + "PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS", "PAPERCLIP_RUNNER_INSTANCE_ID", "PAPERCLIP_RUN_ID", "PAPERCLIP_NORMALIZED_SESSION_ID", diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index bf2a8bed9d5..4505b29d687 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -24,6 +24,8 @@ import { pipeline } from "node:stream/promises"; import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; import { parseOpenCodeReasoningMode, type OpenCodeReasoningMode } from "./reasoning-mode.js"; +import { startOpenCodePerformanceDiagnostics } from "./performance-diagnostics.js"; +import { PROVIDER_PERFORMANCE_ENV } from "../../contracts/provider-performance.js"; import { annotateOmittedProviderText, parseOpenCodeSseFrames } from "./sse-frames.js"; import { githubCredentialEnvironment } from "../../github-credential-environment.js"; @@ -217,6 +219,7 @@ interface OpenCodeRuntime { bridge: OpenCodeMcpBridge; trace: ProviderTraceFileSink | null; sensitiveValues: readonly string[]; + performanceDiagnostics: Awaited> | null; close(input?: { finalizeTrace?: boolean; reason?: string | null; @@ -520,6 +523,7 @@ class OpenCodeHarnessSession implements HarnessSession { #activeTraceEmittedEventIds: string[] = []; #sourceSequence: number; #activeTurnId: string | null; + #performanceBoundaryMs = 0; #result: PrpStructuredRunResult | null; #resultFingerprint: string | null; #resultCallId: string | null; @@ -2133,6 +2137,11 @@ class OpenCodeHarnessSession implements HarnessSession { }); return; } + if (["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"].includes(eventType)) { + const performance = this.#runtime.performanceDiagnostics?.take(this.#performanceBoundaryMs); + this.#performanceBoundaryMs = this.#now().getTime(); + if (performance) this.#emit("harness.diagnostic", performance, refs); + } const sourceSeq = ++this.#sourceSequence; const event: PrpEvent = { schema: "paperclip.prp.event.v1", @@ -2349,6 +2358,15 @@ async function startRuntime(input: { input.trace?.addSensitiveValues([providerProxy.token]); } let child: ChildProcess | undefined; + // Failure to collect diagnostics must not prevent the requested work. + const performanceDiagnostics = input.options.environment?.[PROVIDER_PERFORMANCE_ENV] === "1" + ? await startOpenCodePerformanceDiagnostics().catch(() => null) : null; + if (performanceDiagnostics) { + const values = [performanceDiagnostics.environment.OTEL_EXPORTER_OTLP_HEADERS, + performanceDiagnostics.environment.OTEL_EXPORTER_OTLP_HEADERS.replace("Authorization=Bearer ", "")]; + sensitiveValues.push(...values); + input.trace?.addSensitiveValues(values); + } let launchPrepared = false; const releaseExecutable = () => { if (!launchPrepared) return; @@ -2366,6 +2384,7 @@ async function startRuntime(input: { small_model: input.options.model, share: "disabled", autoupdate: false, + ...(performanceDiagnostics ? { experimental: { openTelemetry: true } } : {}), // The configured entry is already composed exactly once into the session // system prompt; siblings remain available through the read-only root. instructions: [], @@ -2450,6 +2469,7 @@ async function startRuntime(input: { OPENCODE_DISABLE_DEFAULT_PLUGINS: "true", OPENCODE_SERVER_USERNAME: username, OPENCODE_SERVER_PASSWORD: password, + ...performanceDiagnostics?.environment, ...(providerProxy ? { NO_PROXY: [input.options.environment?.no_proxy ?? input.options.environment?.NO_PROXY, "127.0.0.1", "localhost"].filter(Boolean).join(",") } : {}), }, ); @@ -2547,6 +2567,7 @@ async function startRuntime(input: { bridge, trace: input.trace, sensitiveValues, + performanceDiagnostics, close: async (closeInput = {}) => { await providerProxy?.close(); await bridge.close().catch(() => {}); @@ -2570,6 +2591,7 @@ async function startRuntime(input: { } await waitForExit(providerChild, 2_000); } + await performanceDiagnostics?.close().catch(() => {}); await rm(join(configHome, "opencode", "opencode.json"), { force: true, }).catch(() => undefined); @@ -2580,6 +2602,7 @@ async function startRuntime(input: { }, }; } catch (error) { + await performanceDiagnostics?.close().catch(() => {}); await providerProxy?.close(); await bridge.close().catch(() => {}); child?.kill("SIGKILL"); diff --git a/packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.test.ts b/packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.test.ts new file mode 100644 index 00000000000..46da01dc64d --- /dev/null +++ b/packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.test.ts @@ -0,0 +1,65 @@ +import { describe, expect, it } from "vitest"; +import { parseProviderPerformanceSpans, PROVIDER_PERFORMANCE_DIAGNOSTIC } from "../../contracts/provider-performance.js"; +import { startOpenCodePerformanceDiagnostics } from "./performance-diagnostics.js"; + +function batch(spans: unknown[]) { return { resourceSpans: [{resource:{attributes:[{key:"secret",value:"credential"}]},scopeSpans:[{spans}]}]}; } +function span(name = "ai.streamText", extra = {}) { + const now=Date.now(); + return {name,startTimeUnixNano:String(BigInt(now-10)*1000000n),endTimeUnixNano:String(BigInt(now)*1000000n), + attributes:[{key:"Authorization",value:{stringValue:"Bearer private"}},{key:"prompt",value:{stringValue:"private task"}}], + events:[{name:"private content"}],...extra}; +} + +describe("OpenCode content-free performance sink",()=>{ + it("accepts authenticated loopback OTLP and projects only closed phase names and timings",async()=>{ + const sink=await startOpenCodePerformanceDiagnostics(); + const url=sink.environment.OTEL_EXPORTER_OTLP_ENDPOINT; + const headers={Authorization:sink.environment.OTEL_EXPORTER_OTLP_HEADERS.slice("Authorization=".length)}; + try { + expect((await fetch(url+"/v1/traces",{method:"POST",body:JSON.stringify(batch([span()]))})).status).toBe(401); + expect(sink.take()).toBeNull(); + expect((await fetch(url+"/v1/traces",{method:"POST",headers,body:JSON.stringify(batch([ + span(),span("Snapshot.track",{status:{code:2}}),span("private-provider-name"),span("__proto__"),span("toString"), + ]))})).status).toBe(200); + const diagnostic=sink.take()!; + expect(diagnostic.code).toBe(PROVIDER_PERFORMANCE_DIAGNOSTIC); + expect(diagnostic.spans.map(s=>[s.name,s.outcome])).toEqual([["provider.model.stream","ok"],["provider.workspace.snapshot.track","failed"]]); + expect(diagnostic.spans[0].endedAtMs-diagnostic.spans[0].startedAtMs).toBeCloseTo(10, 2); + expect(JSON.stringify(diagnostic)).not.toMatch(/private|Authorization|prompt|credential/); + expect(Object.keys(diagnostic.spans[0]).sort()).toEqual(["endedAtMs","name","outcome","startedAtMs"]); + expect(sink.take()).toBeNull(); + expect((await fetch(url+"/v1/logs",{method:"POST",headers,body:'{"private":"discard"}'})).status).toBe(200); + expect(sink.take()).toBeNull(); + } finally {await sink.close();} + await expect(fetch(url+"/v1/traces")).rejects.toThrow(); + }); + + it("bounds bodies, malformed timestamps and repeated spans and drops prior-turn exports",async()=>{ + const sink=await startOpenCodePerformanceDiagnostics(); + const url=sink.environment.OTEL_EXPORTER_OTLP_ENDPOINT; + const headers={Authorization:sink.environment.OTEL_EXPORTER_OTLP_HEADERS.slice("Authorization=".length)}; + const post=(body:string)=>fetch(url+"/v1/traces",{method:"POST",headers,body}); + try { + expect((await post("x".repeat(1024*1024+1))).status).toBe(413); + expect((await post("invalid json")).status).toBe(400); + expect((await post(JSON.stringify(batch([ + span("ai.streamText",{endTimeUnixNano:"NaN"}),span("ai.streamText",{endTimeUnixNano:"1"}), + ...Array.from({length:200},()=>span()), + ])))).status).toBe(200); + expect(sink.take()!.spans).toHaveLength(16); + await post(JSON.stringify(batch([span()]))); + expect(sink.take(Date.now()+100)).toBeNull(); + }finally{await sink.close();} + }); + + it("validates the shared diagnostic again before server export",()=>{ + const now=Date.now(); + const safe={name:"provider.model.stream",startedAtMs:now-20,endedAtMs:now,outcome:"ok",prompt:"private"}; + const parsed=parseProviderPerformanceSpans({code:PROVIDER_PERFORMANCE_DIAGNOSTIC,spans:[ + safe,{...safe,name:"private"},{...safe,startedAtMs:Infinity},{...safe,endedAtMs:now+2000}, + {...safe,startedAtMs:now-86400001},{...safe,endedAtMs:now-30}, + ]},now); + expect(parsed).toEqual([{name:safe.name,startedAtMs:now-20,endedAtMs:now,outcome:"ok"}]); + expect(parseProviderPerformanceSpans({code:"other",spans:[safe]},now)).toEqual([]); + }); +}); diff --git a/packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.ts b/packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.ts new file mode 100644 index 00000000000..5727e8a3f48 --- /dev/null +++ b/packages/paperclip-runner/src/drivers/opencode/performance-diagnostics.ts @@ -0,0 +1,103 @@ +import { randomBytes, timingSafeEqual } from "node:crypto"; +import { createServer } from "node:http"; +import { + MAX_PROVIDER_PERFORMANCE_SPANS, PROVIDER_PERFORMANCE_DIAGNOSTIC, + type ProviderPerformancePhase, type ProviderPerformanceSpan, +} from "../../contracts/provider-performance.js"; + +const phases: Readonly> = { + "SystemPrompt.environment": "provider.context.environment", + "Instruction.system": "provider.context.instructions", + "Skill.discovery": "provider.skills.discovery", + "SessionTools.resolve": "provider.tools.resolve", + "Provider.getModel": "provider.model.resolve", + "ai.streamText": "provider.model.stream", + "Snapshot.track": "provider.workspace.snapshot.track", + "Snapshot.patch": "provider.workspace.snapshot.patch", + "Snapshot.diffFull": "provider.workspace.snapshot.diff", + "Git.repo.discover": "provider.workspace.git.discover", + "Git.defaultBranch": "provider.workspace.git.default_branch", + "Npm.install": "provider.dependencies.install", + "PluginInternal.boot": "provider.plugins.initialize", +}; +const maxBodyBytes = 1024 * 1024; + +function object(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} + +/** OpenCode's raw OTLP can contain authorization and prompt attributes. None leave this sink. */ +export async function startOpenCodePerformanceDiagnostics() { + const token = randomBytes(32).toString("hex"); + const authorization = Buffer.from(`Bearer ${token}`); + const spans: ProviderPerformanceSpan[] = []; + const counts = new Map(); + const server = createServer((request, response) => { + const provided = Buffer.from(request.headers.authorization ?? ""); + if (provided.length !== authorization.length || !timingSafeEqual(provided, authorization)) { + request.resume(); response.writeHead(401).end(); return; + } + if (request.method !== "POST" || !["/v1/traces", "/v1/logs"].includes(request.url ?? "")) { + request.resume(); response.writeHead(404).end(); return; + } + let bytes = 0; + const chunks: Buffer[] = []; + request.on("data", (chunk: Buffer) => { + bytes += chunk.length; + if (bytes <= maxBodyBytes && request.url === "/v1/traces") chunks.push(chunk); + else chunks.length = 0; + }); + request.on("error", () => response.destroy()); + request.on("end", () => { + if (bytes > maxBodyBytes) { response.writeHead(413).end(); return; } + if (request.url === "/v1/logs") { response.writeHead(200).end("{}"); return; } + try { + const batch = object(JSON.parse(Buffer.concat(chunks).toString("utf8"))); + for (const resource of Array.isArray(batch.resourceSpans) ? batch.resourceSpans : []) { + for (const scope of Array.isArray(object(resource).scopeSpans) ? object(resource).scopeSpans as unknown[] : []) { + for (const value of Array.isArray(object(scope).spans) ? object(scope).spans as unknown[] : []) { + const row = object(value); + const name = typeof row.name === "string" && Object.hasOwn(phases, row.name) ? phases[row.name] : undefined; + if (!name || spans.length >= MAX_PROVIDER_PERFORMANCE_SPANS || (counts.get(name) ?? 0) >= 16) continue; + const startedAtMs = Number(row.startTimeUnixNano) / 1e6; + const endedAtMs = Number(row.endTimeUnixNano) / 1e6; + const now = Date.now(); + if (!Number.isFinite(startedAtMs) || !Number.isFinite(endedAtMs) || startedAtMs < now - 86400000 + || endedAtMs < startedAtMs || endedAtMs > now + 1000) continue; + spans.push({name, startedAtMs, endedAtMs, outcome: object(row.status).code === 2 ? "failed" : "ok"}); + counts.set(name, (counts.get(name) ?? 0) + 1); + } + } + } + response.writeHead(200, {"Content-Type":"application/json"}).end("{}"); + } catch { response.writeHead(400).end(); } + }); + }); + server.requestTimeout = 2000; + server.headersTimeout = 2000; + server.maxConnections = 2; + await new Promise((resolve, reject) => { + server.once("error", reject); + server.listen(0, "127.0.0.1", () => { server.off("error", reject); resolve(); }); + }); + const address = server.address(); + if (!address || typeof address === "string") { server.close(); throw new Error("OpenCode diagnostics did not bind loopback"); } + return { + environment: { + OTEL_EXPORTER_OTLP_ENDPOINT: `http://127.0.0.1:${address.port}`, + OTEL_EXPORTER_OTLP_PROTOCOL: "http/json", + // OpenCode's Effect logger reads literal values; its trace exporter receives the same map. + OTEL_EXPORTER_OTLP_HEADERS: `Authorization=Bearer ${token}`, + OTEL_BSP_SCHEDULE_DELAY: "100", + }, + take(sinceMs = 0): {code: string; spans: ProviderPerformanceSpan[]} | null { + const collected = spans.splice(0).filter(span => span.startedAtMs >= sinceMs); + counts.clear(); + return collected.length ? {code: PROVIDER_PERFORMANCE_DIAGNOSTIC, spans: collected} : null; + }, + async close() { + spans.length = 0; + await new Promise(resolve => { server.close(() => resolve()); server.closeAllConnections(); }); + }, + }; +} diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 8a35f307a32..38c34e26872 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -4,6 +4,7 @@ export { CURSOR_DISTRIBUTION_PINS } from "./drivers/acpx/generated-profiles.js"; export * from "./contracts/control-plane-port.js"; export * from "./contracts/completion-result.js"; export * from "./contracts/codex.js"; +export * from "./contracts/provider-performance.js"; export * from "./contracts/durable-recovery.js"; export * from "./contracts/harness-driver.js"; export * from "./contracts/local-runner.js"; diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index d2b054ef787..26a3e0b20b4 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -1,5 +1,6 @@ import { bindRunnerLaunchResources, createRunnerProviderLaunch, runnerLaunchRequirementsDigest, type PreparedRunnerProviderLaunch, type RunnerProviderLaunchRequirements } from "./runner-provider-launch.js"; import { coldAdmissionTimeoutMs, waitForRunnerCommand } from "./runnerd-codex-transport.js"; +import { configuredEnvironmentProjection } from "../configured-environment.js"; import { chmod, cp, @@ -2086,6 +2087,9 @@ it("preserves OpenCode runtime bindings when a durable runner is respawned", () environment: { PATH: "/bin", OPENROUTER_API_KEY: "test-provider-key", + ...configuredEnvironmentProjection({ PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" }), + OTEL_EXPORTER_OTLP_ENDPOINT: "https://must-not-reach-opencode.invalid", + OTEL_EXPORTER_OTLP_HEADERS: "Authorization=private", HOME: "/host/home", CODEX_HOME: "/host/codex-home", DATABASE_URL: "must-not-reach-runnerd", @@ -2115,7 +2119,10 @@ it("preserves OpenCode runtime bindings when a durable runner is respawned", () PAPERCLIP_NORMALIZED_SESSION_ID: "session-1", PAPERCLIP_NATIVE_RUNTIME_CONTEXT_PATH: "/isolated/runtime-context.json", OPENROUTER_API_KEY: "test-provider-key", + PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1", }); + expect(environment.OTEL_EXPORTER_OTLP_ENDPOINT).toBeUndefined(); + expect(environment.OTEL_EXPORTER_OTLP_HEADERS).toBeUndefined(); expect(environment.HOME).toBeUndefined(); expect(environment.CODEX_HOME).toBeUndefined(); expect(environment.DATABASE_URL).toBeUndefined(); @@ -9066,7 +9073,9 @@ it.each([ opencodeProxySha256: digest(proxy), providerNodeCommand: providerNode, providerNodeCommandSha256: digest(providerNode), - environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "fixture-key" }, + environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: "fixture-key", + ...(supportsReasoning ? configuredEnvironmentProjection({ PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" }) : {}), + }, }); const task = createCodexTaskEnvelope({ objective: "Preserve the prepared task.", contractRevision: "prepared-v1", @@ -9113,6 +9122,14 @@ it.each([ expect(events.find(event => event.eventType === "session.started")?.payload.context).toMatchObject({ model }); expect(completionCalls).toBe(2); expect(events.filter(event => event.eventType === "run.result.proposed")).toHaveLength(1); + const performance = events.filter(event => event.eventType === "harness.diagnostic" && event.payload.code === "provider_performance_spans"); + expect(performance).toHaveLength(supportsReasoning ? 1 : 0); + if (supportsReasoning) { + expect(performance[0].payload.spans).toEqual([{ + name: "provider.model.stream", startedAtMs: expect.any(Number), endedAtMs: expect.any(Number), outcome: "ok", + }]); + expect(JSON.stringify(performance)).not.toContain("private-performance-sentinel"); + } const sessionRoots = (await readdir(runtime, { withFileTypes: true })).filter((entry) => entry.isDirectory()); expect(sessionRoots).toHaveLength(1); const requests = (await readFile(join(runtime, sessionRoots[0]!.name, "data/fake-prompt-requests.ndjson"), "utf8")).trim().split("\n").map((line) => JSON.parse(line)); diff --git a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs index dee873d0da3..412d602171a 100644 --- a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs +++ b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs @@ -128,6 +128,24 @@ let promptTurnSeq = 0; // sealed. let lateStragglerRemaining = 0; function emit(value) { + if (value.type === "session.idle" && runtimeConfig.experimental?.openTelemetry) { + const now=Date.now(); + const headers=Object.fromEntries((process.env.OTEL_EXPORTER_OTLP_HEADERS??"").split(",").map(entry=>{ + const i=entry.indexOf("=");return [entry.slice(0,i),entry.slice(i+1)]; + })); + void fetch(`${process.env.OTEL_EXPORTER_OTLP_ENDPOINT}/v1/traces`,{method:"POST",headers, + body:JSON.stringify({resourceSpans:[{scopeSpans:[{spans:[{name:"ai.streamText", + startTimeUnixNano:String(BigInt(now-22)*1000000n),endTimeUnixNano:String(BigInt(now)*1000000n), + attributes:[{key:"prompt",value:{stringValue:"private-performance-sentinel"}}], + }]}]}]})}).then(response=>{ + if(!response.ok)throw new Error(`Performance fixture rejected: ${response.status}`); + emitFrame(value); + }).catch(error=>{process.stderr.write(String(error));process.exit(1)}); + return; + } + emitFrame(value); +} +function emitFrame(value) { const scoped = promptTurnSeq > 0 && typeof value.id === "string" ? { ...value, id: `${value.id}#${promptTurnSeq}` } diff --git a/server/scripts/benchmark-native-skill-preparation.ts b/server/scripts/benchmark-native-skill-preparation.ts new file mode 100644 index 00000000000..da1310afdb0 --- /dev/null +++ b/server/scripts/benchmark-native-skill-preparation.ts @@ -0,0 +1,260 @@ +/** Real skill resolution, PostgreSQL, and filesystem work; no provider is started. */ +import assert from "node:assert/strict"; +import { createHash, randomUUID } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { connect, createServer, type Socket } from "node:net"; +import { context, trace } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; +import { resourceFromAttributes } from "@opentelemetry/resources"; +import { eq, sql } from "drizzle-orm"; +import { authUsers, companyMemberships, companySkillSources, heartbeatRuns, projects, projectSkillSources, + companies, companySkills, companySkillVersions, companySkillRepositorySnapshots, + applyPendingMigrations, closeRegisteredClients, createDb, startEmbeddedPostgresTestDatabase } from "../../packages/db/src/index.js"; +import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; +import { createRunPreparationTiming, withRunExecutionTiming } from "../src/services/execution-preparation-timing.js"; +import { repositorySkillProjection } from "../src/services/skill-repository-package.js"; +import { skillFileBytes, skillSnapshotHash } from "../src/services/skill-snapshot.js"; + +const argument = (name: string) => process.argv.find(value => value.startsWith(`${name}=`))?.slice(name.length + 1); +const exportOtlp = argument("--export-otlp") === "true"; +const authenticated = argument("--actor") === "agent"; +const stagingShape = argument("--staging-shape") === "true"; +// This standalone benchmark owns its tracer. Load the service graph with the +// app-wide SDK disabled, then restore the operator's export configuration. +const operatorEndpoint = process.env.OTEL_EXPORTER_OTLP_ENDPOINT; +delete process.env.OTEL_EXPORTER_OTLP_ENDPOINT; +let companySkillService: typeof import("../src/services/company-skills.js").companySkillService; +let withSkillActor: typeof import("../src/services/skill-access.js").withSkillActor; +let createAgentLifecycle: typeof import("../src/services/agent-lifecycle.js").createAgentLifecycle; +try { + ({ companySkillService } = await import("../src/services/company-skills.js")); + ({ withSkillActor } = await import("../src/services/skill-access.js")); + ({ createAgentLifecycle } = await import("../src/services/agent-lifecycle.js")); +} +finally { + if (operatorEndpoint !== undefined) process.env.OTEL_EXPORTER_OTLP_ENDPOINT = operatorEndpoint; +} +const sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { + cwd: fileURLToPath(new URL("../../", import.meta.url)), encoding: "utf8", +}).trim(); +const sourceHash = createHash("sha256"); +for (const file of [import.meta.url, new URL("../src/services/company-skills.ts", import.meta.url), + new URL("../src/services/skill-version-cache.ts", import.meta.url), new URL("../src/services/execution-preparation-timing.ts", import.meta.url)]) { + sourceHash.update(await fs.readFile(new URL(file))); +} +const sourceChangesHash = sourceHash.digest("hex"); +if (exportOtlp) assert.ok(process.env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim(), "OTLP export requires an operator-configured endpoint"); +const snapshotFile = argument("--snapshot-file"); +const entrypoint = argument("--entrypoint") ?? "skills/diagnostic/SKILL.md"; +const files: CompanySkillVersionFileInventoryEntry[] = snapshotFile + ? JSON.parse(await fs.readFile(snapshotFile, "utf8")) + : [{ path: entrypoint, kind: "skill", content: "---\nname: diagnostic\ndescription: Native preparation benchmark\n---\nReport the model ID.\n" }, + ...Array.from({ length: 207 }, (_, index) => ({ path: `assets/${index}.txt`, kind: "reference" as const, content: "x".repeat(52_000) }))]; +const projected = repositorySkillProjection(files, entrypoint); +const previous = { home: process.env.PAPERCLIP_HOME, instance: process.env.PAPERCLIP_INSTANCE_ID, endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT }; +const home = await fs.mkdtemp(path.join(os.tmpdir(), "native-skill-benchmark-")); +let provider: TracerProvider | undefined; +let manager: AsyncLocalStorageContextManager | undefined; +let database: Awaited> | undefined; +let db: ReturnType | undefined; +let proxy: ReturnType | null = null; +const connections = new Set(); +try { + process.env.PAPERCLIP_HOME = home; + process.env.PAPERCLIP_INSTANCE_ID = "default"; + process.env.OTEL_EXPORTER_OTLP_ENDPOINT = exportOtlp ? previous.endpoint : "http://127.0.0.1:4318"; + const exporter = new InMemorySpanExporter(); + const processors = [new SimpleSpanProcessor({ exporter })]; + const exportResults: number[] = []; + if (exportOtlp) { + const packages: Record = { grpc: "@opentelemetry/exporter-trace-otlp-grpc", + "http/protobuf": "@opentelemetry/exporter-trace-otlp-proto", "http/json": "@opentelemetry/exporter-trace-otlp-http" }; + const packageName = packages[process.env.OTEL_EXPORTER_OTLP_PROTOCOL ?? "grpc"]; + assert.ok(packageName, "Unsupported OTLP protocol"); + const { OTLPTraceExporter } = await import(packageName); + const live = new OTLPTraceExporter(); + processors.push(new SimpleSpanProcessor({ exporter: { + export: (spans, callback) => live.export(spans, (result: { code: number }) => { exportResults.push(result.code); callback(result); }), + shutdown: () => live.shutdown(), + } })); + } + provider = new TracerProvider({ spanProcessors: processors, resource: resourceFromAttributes({ + "service.name": process.env.OTEL_SERVICE_NAME || "paperclip", + "service.version": `local-skill-bench-${sourceChangesHash.slice(0, 12)}`, + "deployment.environment.name": "local-diagnostic", + }) }); + manager = new AsyncLocalStorageContextManager().enable(); + assert.ok(context.setGlobalContextManager(manager)); + assert.ok(trace.setGlobalTracerProvider(provider)); + async function startDatabase() { + const port = argument("--postgres-port"); + if (!port) return startEmbeddedPostgresTestDatabase("native-skill-benchmark-"); + assert.match(port, /^\d{1,5}$/); + assert.ok(Number(port) > 0 && Number(port) <= 65535); + const username = argument("--postgres-user") ?? os.userInfo().username; + const adminUrl = `postgres://${encodeURIComponent(username)}@127.0.0.1:${port}/postgres`; + const name = `native_skill_bench_${randomUUID().replaceAll("-", "")}`; + const connectionString = `postgres://${encodeURIComponent(username)}@127.0.0.1:${port}/${name}`; + const admin = createDb(adminUrl, { maxConnections: 1 }).$client; + try { await admin.unsafe(`CREATE DATABASE "${name}"`); } + finally { await admin.end({ timeout: 1 }); } + const cleanup = async () => { + await closeRegisteredClients(connectionString); + const cleanupAdmin = createDb(adminUrl, { maxConnections: 1 }).$client; + try { await cleanupAdmin.unsafe(`DROP DATABASE "${name}"`); } + finally { await cleanupAdmin.end({ timeout: 1 }); } + }; + try { await applyPendingMigrations(connectionString); } + catch (error) { await cleanup(); throw error; } + return { connectionString, cleanup }; + } + database = await startDatabase(); + const networkDelayMs = Number(argument("--network-delay-ms") ?? 0); + assert.ok(Number.isFinite(networkDelayMs) && networkDelayMs >= 0 && networkDelayMs <= 100); + const target = new URL(database.connectionString); + const upstreamPort = Number(target.port); + proxy = networkDelayMs ? createServer(client => { + const upstream = connect({ host: "127.0.0.1", port: upstreamPort }); + for (const socket of [client, upstream]) { + socket.setNoDelay(true); + connections.add(socket); + socket.on("error", () => { client.destroy(); upstream.destroy(); }); + socket.on("close", () => { connections.delete(socket); }); + } + const relay = (source: Socket, destination: Socket) => { + let pendingBytes = 0; + source.on("data", chunk => { + pendingBytes += chunk.length; + if (pendingBytes >= 2 * 1024 * 1024) source.pause(); + setTimeout(() => { + pendingBytes -= chunk.length; + if (destination.destroyed) return; + const writable = destination.write(chunk); + if (writable && pendingBytes < 2 * 1024 * 1024) source.resume(); + }, networkDelayMs); + }); + destination.on("drain", () => { if (pendingBytes < 2 * 1024 * 1024) source.resume(); }); + source.on("end", () => setTimeout(() => destination.end(), networkDelayMs)); + }; + relay(client, upstream); relay(upstream, client); + }) : null; + if (proxy) { + const server = proxy; + await new Promise((resolve, reject) => { server.once("error", reject); server.listen(0, "127.0.0.1", resolve); }); + const address = server.address(); + assert.ok(address && typeof address !== "string"); + target.port = String(address.port); + } + db = createDb(target.href); + const companyId = randomUUID(); + const agentId = randomUUID(); + const userId = randomUUID(); + const skillId = randomUUID(); + const key = `company/${companyId}/diagnostic`; + const samples: unknown[] = []; + let rootPoolQueries = 0; + db.$client.options.debug = () => { rootPoolQueries++; }; + await db.insert(companies).values({ id: companyId, name: "Native preparation benchmark", issuePrefix: "BENCH" }); + if (authenticated) { + await createAgentLifecycle(db).requestHire(companyId, { + id: agentId, name: "Diagnostic reader", adapterType: "paperclip_runner", status: "paused", + }, { runnerResolved: true }); + await db.insert(authUsers).values({ id: userId, name: "Diagnostic owner", email: `${userId}@example.invalid`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values([ + { companyId, principalType: "agent", principalId: agentId, membershipRole: "member" }, + { companyId, principalType: "user", principalId: userId, membershipRole: "owner" }, + ]); + } + if (stagingShape) { + // Match observed staging cardinalities without copying unrelated company content. + const projectRows = await db.insert(projects).values(Array.from({ length: 14 }, (_, i) => ({ companyId, name: `Fixture ${i}` }))).returning(); + const sourceRows = await db.insert(companySkillSources).values(Array.from({ length: 10 }, (_, i) => ({ + companyId, repositoryUrl: `https://example.invalid/fixture-${i}`, fullName: `fixture/${i}`, trackingRef: "main", + accessScope: i < 4 ? "projects" as const : "company" as const, + }))).returning(); + await db.insert(projectSkillSources).values(Array.from({ length: 8 }, (_, i) => ({ companyId, + projectId: projectRows[i % 5]!.id, sourceId: sourceRows[i]!.id, + repositoryUrl: sourceRows[i]!.repositoryUrl, trackingRef: "main", status: "ready" as const, + }))); + await db.insert(companySkills).values(Array.from({ length: 155 }, (_, i) => ({ companyId, + key: `fixture/unselected-${i}`, slug: `unselected-${i}`, name: `Unselected ${i}`, sourceType: "github", + sourceLocator: "https://example.invalid/immutable-fixture", markdown: "x".repeat(7_330), + accessSourceId: sourceRows[i % sourceRows.length]!.id, + }))); + await db.insert(companySkills).values({ companyId, key: "fixture/catalog", slug: "catalog", name: "Unselected catalog", sourceType: "catalog", markdown: "x".repeat(1_578) }); + } + await db.insert(companySkills).values({ id: skillId, companyId, key, slug: "diagnostic", name: "Diagnostic", + markdown: files.find(file => file.path === entrypoint)!.content, sourceType: "github", + sourceLocator: "https://example.invalid/immutable-fixture", metadata: { packageMode: "repository", snapshotHash: "fixture" } }); + const [snapshot] = await db.insert(companySkillRepositorySnapshots).values({ companyId, contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, revisionNumber: 1, + repositorySnapshotId: snapshot!.id, repositorySkillPath: entrypoint }).returning(); + await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skillId)); + const benchmarkDb = db; + const service = companySkillService(db); + for (const label of ["cold", "warm-1", "warm-2", "warm-new-service"]) { + exporter.reset(); + rootPoolQueries = 0; + const selected: ReturnType = label === "warm-new-service" ? companySkillService(db) : service; + const runId = randomUUID(); + if (authenticated) await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", responsibleUserId: userId, contextSnapshot: { executionPolicy: {} } }); + rootPoolQueries = 0; + const measuredStarted = performance.now(); + const resolve = () => selected.withRuntimeSkillEntries(companyId, { + onlyKeys: new Set([key]), versionSelections: new Map([[key, version!.id]]), snapshotLatest: true, + }, async entries => { + if (authenticated) await benchmarkDb.update(heartbeatRuns).set({ contextSnapshot: sql`coalesce(${heartbeatRuns.contextSnapshot}, '{}'::jsonb) || ${JSON.stringify({ skillVersions: { [key]: version!.id }, skillSelectionSnapshot: true })}::jsonb` }).where(eq(heartbeatRuns.id, runId)); + return entries; + }); + const entries: Awaited> = await withRunExecutionTiming(runId, () => createRunPreparationTiming(runId)("configuration_skills", () => authenticated + ? withSkillActor({ type: "agent", agentId, companyId, runId, source: "agent_jwt", onBehalfOfUserId: userId }, resolve) + : resolve())); + const elapsedMs = performance.now() - measuredStarted; + const queryCount = rootPoolQueries; + assert.equal(entries.length, 1); + assert.equal(entries[0]!.sourceStatus, "available", entries[0]!.missingDetail ?? "Skill unavailable"); + // Verify the whole delivered repository, not just its authored SKILL.md. + for (const file of projected) { + const location = path.join(entries[0]!.source, file.path); + assert.deepEqual(await fs.readFile(location), skillFileBytes(file)); + if (process.platform !== "win32") assert.equal(Boolean((await fs.stat(location)).mode & 0o111), Boolean(file.executable)); + } + await provider.forceFlush(); + if (exportOtlp) assert.ok(exportResults.length > 0 && exportResults.every(code => code === 0), "OTLP export failed"); + const spans = exporter.getFinishedSpans(); + assert.ok(spans.some(span => span.name === "skills.version.verify" && span.attributes["paperclip.execution.fileCount"] === projected.length)); + assert.equal(new Set(spans.map(span => span.spanContext().traceId)).size, 1); + assert.ok(spans.every(span => Object.keys(span.attributes).every(key => ["paperclip.task.run.run_id", "paperclip.execution.fileCount", "paperclip.execution.bytes"].includes(key)))); + samples.push({ label, elapsedMs, rootPoolQueries: queryCount, traceId: spans[0]!.spanContext().traceId, + phases: spans.filter(span => span.name.startsWith("skills.")).map(span => ({ name: span.name, + durationMs: span.duration[0] * 1000 + span.duration[1] / 1e6, + spanId: span.spanContext().spanId, parentSpanId: span.parentSpanContext?.spanId, + attributes: span.attributes })) }); + console.error(JSON.stringify({ label, elapsedMs, rootPoolQueries: queryCount })); + } + console.log(JSON.stringify({ schema: "paperclip.native-skill-preparation-benchmark.v1", + sourceRevision, sourceChangesHash, exportedOtlp: exportOtlp, + source: snapshotFile ? "supplied immutable snapshot" : "synthetic repository", + networkDelayMsPerDirection: networkDelayMs, authenticated, stagingShape, + repositoryHash: skillSnapshotHash(files), repositoryFiles: files.length, runtimeFiles: projected.length, + runtimeBytes: projected.reduce((sum, file) => sum + skillFileBytes(file).length, 0), samples }, null, 2)); +} finally { + await db?.$client.end({ timeout: 1 }); + if (proxy) { + for (const socket of connections) socket.destroy(); + const server = proxy; + await new Promise(resolve => server.close(() => resolve())); + } + await database?.cleanup(); + await provider?.shutdown(); + trace.disable(); context.disable(); manager?.disable(); + for (const [name, value] of Object.entries({ PAPERCLIP_HOME: previous.home, PAPERCLIP_INSTANCE_ID: previous.instance, OTEL_EXPORTER_OTLP_ENDPOINT: previous.endpoint })) { + if (value === undefined) delete process.env[name]; else process.env[name] = value; + } + await fs.rm(home, { recursive: true, force: true }); +} diff --git a/server/src/__tests__/company-skills-observability.test.ts b/server/src/__tests__/company-skills-observability.test.ts new file mode 100644 index 00000000000..45d50c2da59 --- /dev/null +++ b/server/src/__tests__/company-skills-observability.test.ts @@ -0,0 +1,151 @@ +import assert from "node:assert/strict"; +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { context, SpanStatusCode, trace } from "@opentelemetry/api"; +import { AsyncLocalStorageContextManager } from "@opentelemetry/context-async-hooks"; +import { InMemorySpanExporter, SimpleSpanProcessor, TracerProvider } from "@opentelemetry/sdk-trace"; +import { companies, companySkills, companySkillVersions, companySkillRepositorySnapshots, createDb, + EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; +import { eq } from "drizzle-orm"; +import { createRunPreparationTiming, withRunExecutionTiming } from "../services/execution-preparation-timing.js"; +import { skillSnapshotHash } from "../services/skill-snapshot.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +// This test owns its in-memory tracer; do not start the application exporter. +vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", ""); +const { companySkillService } = await import("../services/company-skills.js"); +const { withSkillVersionCacheLock } = await import("../services/skill-version-cache.js"); +const support = await getEmbeddedPostgresTestSupport(); +const describeDatabase = support.supported ? describe : describe.skip; + +describeDatabase("runtime skill observability", () => { + let database: Awaited>; + let db: ReturnType; + let home: string; + const exporter = new InMemorySpanExporter(); + const lifecycle: string[] = []; + const provider = new TracerProvider({ spanProcessors: [new SimpleSpanProcessor({ exporter }), { + onStart: span => { lifecycle.push(`start:${span.name}`); }, + onEnd: span => { lifecycle.push(`end:${span.name}`); }, + forceFlush: async () => {}, shutdown: async () => {}, + }] }); + const manager = new AsyncLocalStorageContextManager(); + + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("skill-observability-"); + db = createDb(database.connectionString); + home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-observability-")); + vi.stubEnv("PAPERCLIP_HOME", home); + vi.stubEnv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://collector.test"); + expect(context.setGlobalContextManager(manager.enable())).toBe(true); + expect(trace.setGlobalTracerProvider(provider)).toBe(true); + }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); + + afterAll(async () => { + await provider.shutdown(); + trace.disable(); context.disable(); manager.disable(); + vi.unstubAllEnvs(); + await db?.$client.end({ timeout: 1 }); + await database?.cleanup(); + if (home) await fs.rm(home, { recursive: true, force: true }); + }); + + function deferred() { + let resolve!: () => void; + const promise = new Promise(done => { resolve = done; }); + return { promise, resolve }; + } + + async function fixture() { + const companyId = randomUUID(); + const skillId = randomUUID(); + const key = `company/${companyId}/traced`; + const files = [{ path: "skills/traced/SKILL.md", kind: "skill" as const, + content: "---\nname: traced\ndescription: Trace fixture\n---\nReport the model ID.\n" }]; + await db.insert(companies).values({ id: companyId, name: "Trace fixture", issuePrefix: `TR${companyId.slice(0, 6)}` }); + await db.insert(companySkills).values({ id: skillId, companyId, key, slug: "traced", name: "Traced", + sourceType: "github", sourceLocator: "https://example.invalid/immutable-fixture", markdown: files[0]!.content }); + const [snapshot] = await db.insert(companySkillRepositorySnapshots).values({ companyId, + contentHash: skillSnapshotHash(files), fileInventory: files }).returning(); + const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skillId, + revisionNumber: 1, repositorySnapshotId: snapshot!.id, repositorySkillPath: files[0]!.path }).returning(); + await db.update(companySkills).set({ currentVersionId: version!.id }).where(eq(companySkills.id, skillId)); + const service = companySkillService(db); + const options = { onlyKeys: new Set([key]), versionSelections: new Map([[key, version!.id]]), snapshotLatest: true }; + return { companyId, service, options }; + } + + it("parents real preparation work and ends acquisition before the consumer settles", async () => { + const f = await fixture(); + exporter.reset(); + lifecycle.length = 0; + const entered = deferred(), release = deferred(); + const runId = randomUUID(); + const preparation = withRunExecutionTiming(runId, () => createRunPreparationTiming(runId)("configuration_skills", + () => f.service.withRuntimeSkillEntries(f.companyId, f.options, async entries => { + entered.resolve(); await release.promise; return entries; + }))); + try { + await Promise.race([entered.promise, preparation]); + expect(exporter.getFinishedSpans().filter(span => span.name === "skills.cache.acquire")).toHaveLength(1); + expect(exporter.getFinishedSpans().some(span => span.name === "skills.consume")).toBe(false); + } finally { release.resolve(); await preparation; } + expect(await preparation).toMatchObject([{ sourceStatus: "available" }]); + await provider.forceFlush(); + const spans = exporter.getFinishedSpans(); + const span = (name: string) => { const found = spans.find(candidate => candidate.name === name); assert.ok(found, name); return found; }; + for (const [child, parent] of [ + ["heartbeat.prepare.configuration_skills", "task.execution_attempt"], + ["skills.inventory", "heartbeat.prepare.configuration_skills"], + ["skills.inventory.bundled", "skills.inventory"], + ["skills.inventory.releases", "skills.inventory"], + ["skills.inventory.folders", "skills.inventory"], + ["skills.inventory.local_sources", "skills.inventory"], + ["skills.cache.acquire", "heartbeat.prepare.configuration_skills"], + ["skills.rows", "heartbeat.prepare.configuration_skills"], + ["skills.version.read", "heartbeat.prepare.configuration_skills"], + ["skills.version.materialize", "heartbeat.prepare.configuration_skills"], + ["skills.repository.read", "skills.version.materialize"], + ["skills.version.verify", "skills.version.materialize"], + ["skills.consume", "heartbeat.prepare.configuration_skills"], + ]) expect(span(child!).parentSpanContext?.spanId, child).toBe(span(parent!).spanContext().spanId); + const acquisition = span("skills.cache.acquire"); + expect(acquisition.status.code).not.toBe(SpanStatusCode.ERROR); + // SDK timestamps can differ by less than a millisecond at this boundary. + // Observe actual lifecycle order instead of comparing rounded wall clocks. + expect(lifecycle.indexOf("end:skills.cache.acquire")).toBeLessThan(lifecycle.indexOf("start:skills.rows")); + expect(new Set(spans.map(value => value.spanContext().traceId)).size).toBe(1); + expect(spans.every(value => Object.keys(value.attributes).every(key => [ + "paperclip.task.run.run_id", "paperclip.execution.fileCount", "paperclip.execution.bytes", + ].includes(key)))).toBe(true); + }, 30_000); + + it("ends a failed real lock acquisition and preserves the database error", async () => { + const f = await fixture(); + await f.service.listRuntimeSkillEntries(f.companyId, f.options); + const name = new URL(database.connectionString).pathname.slice(1); + assert.match(name, /^[a-zA-Z0-9_-]+$/); + await db.$client.unsafe(`ALTER DATABASE "${name}" SET lock_timeout TO '100ms'`); + const entered = deferred(), release = deferred(); + const holder = withSkillVersionCacheLock(db, f.companyId, async () => { entered.resolve(); await release.promise; }); + try { + await Promise.race([entered.promise, holder]); + exporter.reset(); + const runId = randomUUID(); + await expect(withRunExecutionTiming(runId, () => createRunPreparationTiming(runId)("configuration_skills", + () => f.service.listRuntimeSkillEntries(f.companyId, f.options)))).rejects.toMatchObject({ cause: { code: "55P03" } }); + await provider.forceFlush(); + const spans = exporter.getFinishedSpans(); + const acquisitions = spans.filter(span => span.name === "skills.cache.acquire"); + expect(acquisitions).toHaveLength(1); + expect(acquisitions[0]!.status.code).toBe(SpanStatusCode.ERROR); + const configuration = spans.find(span => span.name === "heartbeat.prepare.configuration_skills"); + expect(acquisitions[0]!.parentSpanContext?.spanId).toBe(configuration?.spanContext().spanId); + expect(spans.some(span => span.name === "skills.rows")).toBe(false); + } finally { release.resolve(); await holder; } + expect(await f.service.listRuntimeSkillEntries(f.companyId, f.options)).toMatchObject([{ sourceStatus: "available" }]); + }, 30_000); +}); diff --git a/server/src/__tests__/connection-management.integration.test.ts b/server/src/__tests__/connection-management.integration.test.ts index 4aec44db9c1..8f091fa49d6 100644 --- a/server/src/__tests__/connection-management.integration.test.ts +++ b/server/src/__tests__/connection-management.integration.test.ts @@ -842,6 +842,8 @@ describe("connection management with production default agents", () => { }); const path = `/chat-endpoints/${created.body.id}/slack/test-message`; if (rate) { + // Keep both calls in one rate window even when CI crosses a minute boundary. + vi.useFakeTimers({ toFake: ["Date"] }); // Shared buckets admit one actual send, while each principal's own // limit also prevents the next send. Neither may consume twice. expect(await f.request("POST", path, { requestId: randomUUID(), slackUserId: "UHUMAN" }, f.jwt), mode) @@ -856,7 +858,7 @@ describe("connection management with production default agents", () => { expect(sends, mode).toBe(rate ? 1 : 0); const audit = await server.db.select().from(toolAccessAuditEvents).where(eq(toolAccessAuditEvents.companyId, f.companyId)); expect(audit.some(event => event.details.agentId === f.agentId && event.actorId === f.agentId), mode).toBe(true); - } finally { provider = null; } + } finally { vi.useRealTimers(); provider = null; } } }, 90_000); diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index af1b4e063af..d2819de5714 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -1,4 +1,5 @@ import { ensureSkillCacheDirectory, touchSkillVersionCache, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { measureRunPhase } from "./execution-preparation-timing.js"; import { verifyExistingBundledSkillRelease } from "./bundled-skill-release-validation.js"; import { retainProjectSkillAccess, skillAccessService } from "./skill-access.js"; import type { AuthorizationActor } from "./authorization.js"; @@ -3237,7 +3238,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!visibility().canReadCompany(companyId)) return; const existingRefresh = skillInventoryRefreshPromises.get(companyId); if (existingRefresh) { - await existingRefresh; + await measureRunPhase("skills.inventory.wait", () => existingRefresh); return; } @@ -3250,10 +3251,10 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo if (!companyExists) { throw notFound("Company not found"); } - const bundledSkills = await ensureBundledSkills(companyId); - await ensureBundledSkillReleases(companyId, bundledSkills); - await reconcilePaperclipSkillFolders(companyId); - await reconcileLocalPathSkillSources(companyId); + const bundledSkills = await measureRunPhase("skills.inventory.bundled", () => ensureBundledSkills(companyId)); + await measureRunPhase("skills.inventory.releases", () => ensureBundledSkillReleases(companyId, bundledSkills)); + await measureRunPhase("skills.inventory.folders", () => reconcilePaperclipSkillFolders(companyId)); + await measureRunPhase("skills.inventory.local_sources", () => reconcileLocalPathSkillSources(companyId)); })(); skillInventoryRefreshPromises.set(companyId, refreshPromise); @@ -6009,8 +6010,9 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo async function runtimeVersionInventory(companyId: string, version: CompanySkillVersion) { if (!version.repositorySnapshotId) return version.fileInventory; - const [snapshot] = await db.select().from(companySkillRepositorySnapshots).where(and( - eq(companySkillRepositorySnapshots.companyId, companyId), eq(companySkillRepositorySnapshots.id, version.repositorySnapshotId))); + const snapshotId = version.repositorySnapshotId; + const [snapshot] = await measureRunPhase("skills.repository.read", () => db.select().from(companySkillRepositorySnapshots).where(and( + eq(companySkillRepositorySnapshots.companyId, companyId), eq(companySkillRepositorySnapshots.id, snapshotId)))); if (!snapshot || !version.repositorySkillPath) throw unprocessable("Installed repository snapshot is unavailable."); return repositorySkillProjection(snapshot.fileInventory, version.repositorySkillPath); } @@ -6029,7 +6031,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const skillDir = path.join(runtimeRoot, version.id); return runVersionSnapshotFlight(skillDir, async () => { await ensureSkillCacheDirectory(runtimeRoot, true); - if (await materializedVersionSnapshotMatches(skillDir, version)) return skillDir; + if (await measureRunPhase("skills.version.verify", () => materializedVersionSnapshotMatches(skillDir, version), { fileCount: version.fileInventory.length })) return skillDir; const staging = await fs.mkdtemp(path.join(runtimeRoot, `.${version.id}-`)); try { let wroteSkillFile = false; @@ -6070,7 +6072,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ?? (options.snapshotLatest || (skill.metadata?.skillSourceId && skill.metadata?.snapshotHash) || skill.accessSourceId ? skill.currentVersionId : null); if (selectedVersionId) { const versionPath = path.resolve(resolveManagedSkillsRoot(companyId), "__versions__", skill.id, selectedVersionId); - const version = await getVersion(companyId, skill.id, selectedVersionId); + const version = await measureRunPhase("skills.version.read", () => getVersion(companyId, skill.id, selectedVersionId)); if (!version) { return { status: "missing", @@ -6096,7 +6098,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo detail: "The selected skill version will be materialized on the next run." }; } } - const versionSource = await materializeVersionSnapshot(companyId, skill, version); + const versionSource = await measureRunPhase("skills.version.materialize", () => materializeVersionSnapshot(companyId, skill, version)); if (versionSource) return { status: "available", source: versionSource }; return { status: "missing", source: versionPath, detail: "The selected skill version produced no files." }; } catch (error) { @@ -6174,15 +6176,18 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo ): Promise { if (options.onlyKeys?.size === 0 || (!options.onlyKeys && options.selectionConfig && !options.includeLegacy && readPaperclipSkillSyncPreference(options.selectionConfig).desiredSkills.length === 0)) return consume([]); - await ensureSkillInventoryCurrent(companyId); - return withSkillVersionCacheLock(db, companyId, async () => consume(await listRuntimeSkillEntriesLocked(companyId, options))); + await measureRunPhase("skills.inventory", () => ensureSkillInventoryCurrent(companyId)); + return withSkillVersionCacheLock(db, companyId, async () => { + const entries = await listRuntimeSkillEntriesLocked(companyId, options); + return measureRunPhase("skills.consume", () => consume(entries)); + }); } async function listRuntimeSkillEntriesLocked( companyId: string, options: RuntimeSkillEntryOptions, ): Promise { - const skills = await readFullSkillRows(companyId); + const skills = await measureRunPhase("skills.rows", () => readFullSkillRows(companyId)); const selectedKeys = options.onlyKeys ?? (options.selectionConfig ? new Set( (options.includeLegacy ? resolveLegacyPaperclipDesiredSkillNames : resolvePaperclipDesiredSkillNames)( options.selectionConfig, skills.map(skill => ({ key: skill.key, runtimeName: buildSkillRuntimeName(skill.key, skill.slug) }))), diff --git a/server/src/services/execution-preparation-timing.test.ts b/server/src/services/execution-preparation-timing.test.ts index 05798901709..21eac0a05f4 100644 --- a/server/src/services/execution-preparation-timing.test.ts +++ b/server/src/services/execution-preparation-timing.test.ts @@ -41,7 +41,8 @@ it.each([undefined, "", " "])("does not consult tracing for endpoint %s", async const error = vi.spyOn(console, "error"); try { await expect(createRunPreparationTiming("run", "native.controller")("backend_setup", () => 42)).resolves.toBe(42); - for (const phase of ["execution.usage_startup", "execution.mcp_handoff", "execution.dispatch_admission", "execution.budget_reservation"] as const) { + for (const phase of ["execution.usage_startup", "execution.mcp_handoff", "execution.dispatch_admission", "execution.budget_reservation", + "skills.inventory", "skills.inventory.wait", "skills.cache.acquire", "skills.version.materialize", "skills.consume"] as const) { const result = {}; const work = vi.fn(() => { recordRepositoryObjectCacheDetail({ outcome: "hit", referenceSupplied: true }); diff --git a/server/src/services/execution-preparation-timing.ts b/server/src/services/execution-preparation-timing.ts index 863ae7b6e5b..a4ff6e82103 100644 --- a/server/src/services/execution-preparation-timing.ts +++ b/server/src/services/execution-preparation-timing.ts @@ -34,6 +34,10 @@ type PreparationPhase = "environment_selection" | "execution_configuration" | "s | "instruction_changes" | "warm_release" | "instruction_collection"; export type ExecutionPhase = + | "skills.inventory" | "skills.inventory.wait" | "skills.inventory.bundled" | "skills.inventory.releases" + | "skills.inventory.folders" | "skills.inventory.local_sources" | "skills.rows" + | "skills.cache.acquire" | "skills.version.read" | "skills.repository.read" + | "skills.version.verify" | "skills.version.materialize" | "skills.consume" | "workspace.transaction" | "workspace.transaction.entry" | "workspace.transaction.commit" | "workspace.transaction.rollback" | "workspace.identity_read" | "workspace.row_lock" | "workspace.provenance" | "workspace.notification" | "workspace.write" | "workspace.sources" | "workspace.sources.canonicalize" | "workspace.sources.inventory" | "workspace.sources.alias_scan" diff --git a/server/src/services/native-runtime/native-run-trace.test.ts b/server/src/services/native-runtime/native-run-trace.test.ts index 2077e64a2ce..8b3a076850c 100644 --- a/server/src/services/native-runtime/native-run-trace.test.ts +++ b/server/src/services/native-runtime/native-run-trace.test.ts @@ -9,6 +9,7 @@ import { createNativeRunTrace, nativeRunPreparationStarts, recordFailedSkillPreparation, + recordNativeProviderPerformanceSpans, NATIVE_RUN_SPAN_EVENT_TYPE, NATIVE_RUN_TRACE_SCHEMA_VERSION, } from "./native-run-trace.js"; @@ -66,6 +67,39 @@ function createRecordingTraceContext(): { } describe("native runner performance trace", () => { + it("projects a full durable diagnostic batch without waiting on duplicate run-log writes", async () => { + const {traceContext, spans}=createRecordingTraceContext(); + let release!: () => void; + const barrier = new Promise(resolve => { release=resolve; }); + const write=vi.fn(async()=>barrier); + const trace=createNativeRunTrace({runId:"provider-perf-blocked-log",startedAtMs:1000,traceContext,onEvent:write}); + const names=["provider.model.stream","provider.context.environment","provider.tools.resolve","provider.model.resolve", + "provider.workspace.snapshot.track","provider.workspace.snapshot.patch","provider.workspace.snapshot.diff","provider.skills.discovery"]; + let completed=false; + const work=recordNativeProviderPerformanceSpans(trace,Array.from({length:128},(_,index)=>({ + name:names[index%names.length],startedAtMs:1100+index,endedAtMs:1200+index, + }))).then(()=>{completed=true;}); + try { + await new Promise(resolve=>setImmediate(resolve)); + expect(completed).toBe(true); + expect(write).not.toHaveBeenCalled(); + expect(spans.filter(span=>span.parentName==="provider.runtime")).toHaveLength(128); + } finally {release();await work;await trace.finish("ok");} + }); + it("groups normalized provider phases under their real native execution parent", async () => { + const {traceContext, spans}=createRecordingTraceContext(); + const trace=createNativeRunTrace({runId:"provider-perf",startedAtMs:1000,traceContext}); + const execute=trace.start("native.session.execute",{startedAtMs:1000}); + await recordNativeProviderPerformanceSpans(trace,[ + {name:"provider.context.environment",startedAtMs:1100,endedAtMs:1200}, + {name:"provider.model.stream",startedAtMs:1200,endedAtMs:1700}, + ]); + await trace.end(execute,{endedAtMs:1800}); + await trace.finish("ok"); + expect(spans.find(s=>s.name==="provider.runtime")).toMatchObject({parentName:"native.session.execute",endedAtMs:1700}); + expect(spans.filter(s=>s.name==="provider.model.stream"||s.name==="provider.context.environment").map(s=>s.parentName)) + .toEqual(["provider.runtime","provider.runtime"]); + }); it.each([7_200_000, 10_800_000])( "times the current answer after %i ms without charging prior questions or human wait", async (answeredAtMs) => { diff --git a/server/src/services/native-runtime/native-run-trace.ts b/server/src/services/native-runtime/native-run-trace.ts index 8594de2d31b..7c38e99cda4 100644 --- a/server/src/services/native-runtime/native-run-trace.ts +++ b/server/src/services/native-runtime/native-run-trace.ts @@ -138,6 +138,7 @@ interface NativeRunSpanScopeState extends NativeRunSpanScope { context: unknown; attributes: Record; ended: boolean; + persistRunLog: boolean; } const NOOP_SPAN: SpanHandle = { @@ -323,6 +324,8 @@ export function createNativeRunTrace(input: { parentName?: string; startedAtMs?: number; attributes?: Record; + /** False only when an existing durable diagnostic batch already contains the measurements. */ + persistRunLog?: boolean; } = {}, ): NativeRunSpanScope => { const startedAtMs = finiteMs(options.startedAtMs ?? Date.now()); @@ -364,6 +367,7 @@ export function createNativeRunTrace(input: { context, attributes, ended: false, + persistRunLog: options.persistRunLog !== false, }; scopes.add(state); openScopes.add(state); @@ -413,7 +417,7 @@ export function createNativeRunTrace(input: { }); if (outcome === "failed") { try { state.span.setStatus({ code: 2 }); } catch {} } try { state.span.end(endedAtMs); } catch {} - await emitCompletedSpanEvent({ + if (state.persistRunLog) await emitCompletedSpanEvent({ name: state.name, parentName: state.parentName, startedAtMs: state.startedAtMs, @@ -458,11 +462,12 @@ export function createNativeRunTrace(input: { ); }; - const record = async (span: NativeRunHistoricalSpan): Promise => { + const record = async (span: NativeRunHistoricalSpan, options: { persistRunLog?: boolean } = {}): Promise => { const scope = start(span.name, { parentName: span.parentName, startedAtMs: span.startedAtMs, attributes: span.attributes, + persistRunLog: options.persistRunLog, }); await end(scope, { endedAtMs: span.endedAtMs, @@ -528,6 +533,22 @@ export function createNativeRunTrace(input: { export type NativeRunTrace = ReturnType; +/** Group already-normalized provider measurements without exporting provider attributes. */ +export async function recordNativeProviderPerformanceSpans(trace: NativeRunTrace, spans: NativeRunHistoricalSpan[]): Promise { + if (!spans.length) return; + try { + const scope = trace.start("provider.runtime", { parentName: "native.session.execute", + startedAtMs: Math.min(...spans.map(span => span.startedAtMs)), persistRunLog: false }); + try { + for (const span of spans) await trace.record({ name: span.name, parentName: "provider.runtime", + startedAtMs: span.startedAtMs, endedAtMs: span.endedAtMs, outcome: span.outcome }, { persistRunLog: false }); + } finally { + await trace.end(scope, { endedAtMs: Math.max(...spans.map(span => span.endedAtMs)), + outcome: spans.some(span => span.outcome === "failed") ? "failed" : "ok" }); + } + } catch { /* Diagnostics cannot replace the work's result. */ } +} + /** Emit preparation failure even when execution aborts before a native session exists. */ export async function recordFailedSkillPreparation(input: { runId: string; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 8466a74eef6..cea0ac147cb 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -17,6 +17,7 @@ import { assertExternalNativeRunAuthority } from "./external-native-authority.js import { stageRemoteLaunchAssets } from "./remote-launch-assets.js"; import { createRunPreparationTiming } from "../execution-preparation-timing.js"; import { configuredEnvironment } from "../../vendor/paperclip-runner/index.js"; +import { parseProviderPerformanceSpans } from "../../vendor/paperclip-runner/index.js"; import { apiKeyBillingType } from "@paperclipai/adapter-utils/api-billing-type"; import { agents } from "@paperclipai/db"; import { isCloudManagedInstance } from "../cloud-instance.js"; @@ -204,6 +205,7 @@ import { createNativeRunTrace, isNativeRunRootHistoricalSpan, nativeRunPreparationStarts, + recordNativeProviderPerformanceSpans, type NativeRunHistoricalSpan, type NativeRunSpanScope, type NativeRunTrace, @@ -8527,6 +8529,9 @@ async function executePaperclipNativeSessionWithinScope( assertControllerActive, assertRunAdmissionActive, onCommittedEvent: async (event) => { + if (event.eventType === "harness.diagnostic") { + await recordNativeProviderPerformanceSpans(trace, parseProviderPerformanceSpans(event.payload, Date.now())); + } const textKind = !firstTokenRecorded ? observeDeliverableText(event) : null; if (textKind) { firstTokenRecorded = true; diff --git a/server/src/services/skill-version-cache.ts b/server/src/services/skill-version-cache.ts index 2f3ec15c5ad..57787129e80 100644 --- a/server/src/services/skill-version-cache.ts +++ b/server/src/services/skill-version-cache.ts @@ -10,6 +10,7 @@ import { assertSkillSnapshotPath } from "./skill-snapshot.js"; import { repositorySkillFrontmatter } from "./skill-repository-package.js"; import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js"; +import { beginRunPhase } from "./execution-preparation-timing.js"; export const SKILL_VERSION_CACHE_IDLE_MS = 60 * 60 * 1000; const locks = new AsyncLocalStorage>(); @@ -26,14 +27,19 @@ export async function withSkillVersionCacheLock(db: Db, companyId: string, ac if (held.some(lock => lock.db === db && lock.companyId === companyId)) return action(); // Preparation uses the ordinary pool for version/pin writes. Borrowing a pool // connection for this lock would deadlock installations with a one-slot pool. + const acquired = beginRunPhase("skills.cache.acquire"); return withDedicatedDbConnection(db, connection => connection.transaction(async tx => { const key = `skill_version_cache:${companyId}`; if (tryOnly) { const rows = await tx.execute<{ acquired: boolean }>(sql`select pg_try_advisory_xact_lock(hashtextextended(${key}, 0)) as acquired`); if (!rows[0]?.acquired) throw new SkillCacheBusy(); } else await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${key}, 0))`); + acquired(); return locks.run([...held, { db, companyId }], action); - })); + })).catch(error => { + acquired(true); + throw error; + }); } export function skillVersionCachePaths(companyId: string, skillId: string, versionId: string) { diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 6790b801b36..9e6f34b2f1e 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -154,6 +154,7 @@ export const resolveSourceCodexHome = runner.resolveSourceCodexHome; export const resolveCodexCommand = runner.resolveCodexCommand; export const resolvePinnedCodexCommand = runner.resolvePinnedCodexCommand; export const validatePrpEvent = runner.validatePrpEvent; +export const parseProviderPerformanceSpans = runner.parseProviderPerformanceSpans; export const validatePrpStructuredRunResult = runner.validatePrpStructuredRunResult; From 5398275d2ae7ce11136f09a3f10529328d01dc37 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:58:11 -0500 Subject: [PATCH 04/41] fix(server): schedule gateway credential expiry from committed changes (#17450) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The tool gateway stores credentials for sessions and named gateways. > - Those credentials already contain expiry dates. > - The server repeatedly scans their tables even when no expiry work exists. > - Creation signals and saved deadlines can drive cleanup without idle scans. > - This PR registers that work with the existing coordinator. > - Authentication continues to reject expired credentials independently of physical cleanup. ## Linked Issues or Issue Description Refs #17446, #17221, #17274. This is the third sleep-migration slice, stacked on #17446. **What problem does this solve?** Gateway credential cleanup queries PostgreSQL on every maintenance tick, including empty tables and retained non-expiring tokens. **What is the proposed solution?** Signal expiring token/session creation from its transaction. Restore the earliest saved expiry at startup. Retry only while cleanup or reconciliation remains outstanding. **Why is this useful?** Configured idle gateways no longer create recurring expiry traffic. Existing credential data remains the durable source of truth. Related work: #17221 and #17274 change run-token lifetime and cleanup eligibility. They remain separate. This PR preserves current master authorization and deletion rules. When those PRs are combined, update the deadline query and signals to match their new operator-token expiry predicate; run-token idle markers must not keep this worker scanning. Retain the separate run-liveness reaper and run both regression suites. Text mergeability is insufficient evidence for that integration. ## What Changed - Make the credential test clock follow real scheduled work without overshooting future-expiry assertions; production behavior is unchanged by this fixture follow-up. - Coalesce sequential credential notifications within the former cleanup cadence, preserve loaded expiry deadlines and immediate authentication rejection, and bound scheduled deletion to 500 rows per pass. - Added transaction-aware intent before the two existing credential inserts. Non-expiring named tokens do not signal expiry work. - Added earliest-expiry selection in the gateway service and a narrow registration wrapper for the shared deadline helper. - Removed gateway credential cleanup from the aggregate maintenance tick. Startup reconstructs saved deadlines. Heartbeat-disabled instances retain their startup-only cleanup behavior. - Preserved expiry/revocation checks, run authorization, audit behavior, and the existing 500-token cleanup batch limit. - Added real PostgreSQL tests and a lifecycle/producer document. Tool-action review recovery remains a separate slice. - Drain all expired session/token pages in explicit and scheduler-disabled startup cleanup, while retaining one-page scheduled passes and authentication enforcement. ## Verification - At `907098cd3656bac59c60220c73b085d0349d4b5f`, all 290 Runner transport tests pass locally (including all seven prepared-environment variants). CI run `40570815360` attempt 1 had one provider-drain settlement failure in the `close-attachment` case; the identical source passed the preceding CI lane. A single failed-job rerun is in progress. The failure remains recorded; no shutdown assertion or production behavior was changed to suppress it. - Master landing reconciliation at `907098cd3656bac59c60220c73b085d0349d4b5f` after #17446 merged: all production and test files are byte-identical to `24318d9b99ee7f044a864ff3981fc1929bf3e509`, where the 150-case acceptance and server types passed. Only commit ancestry and the plan's merged status changed. Fresh CI/review is pending on this head. - Parent reconciliation at `24318d9b99ee7f044a864ff3981fc1929bf3e509`: 150 affected tests pass across gateway credential expiry, the complete tool gateway suite, custom-image worker and startup. The accepted PostgreSQL runs skip no cases. Server types pass. Existing direct tests cover sequential-write coalescing and saved-deadline precedence; the older Ponytail finding needs a fresh review. Remote CI is pending. Current head: `10c85a5eaa288867aa2997a27aee5f675c2a713d`. This remains a single-duty PR stacked on #17446. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md), [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Shared-owner signals cover supported local producers. Independent SQL writers or another replica need a separate ingress contract; startup recovery is not live discovery. - This is scheduling only. The open run-token lifetime PRs need semantic integration as described above before combining their behavior. - Expiry cleanup can wait during drain; authentication still enforces expiry on use. Existing Cloud safety gates are preserved. Local deadlines do not prove external wake coverage or deployed suspension. - Revocation does not move expiry. Non-expiring tokens remain retained by design; this PR does not invent a retention deadline. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip --- doc/idle-gateway-credential-expiry.md | 141 +++++++++ ...-instance-sleep-and-scheduler-inventory.md | 6 +- ...2026-10-24-recurring-server-sleep-stack.md | 2 +- packages/db/src/client.ts | 1 + .../db/src/migrations/0381_mean_scream.sql | 1 + ...{0375_snapshot.json => 0381_snapshot.json} | 50 +++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/tool_access.ts | 1 + .../gateway-credential-expiry.test.ts | 269 ++++++++++++++++++ .../server-startup-feedback-export.test.ts | 1 + server/src/__tests__/tool-gateway.test.ts | 10 +- server/src/index.ts | 7 +- server/src/services/deadline-work.ts | 8 + .../services/delivery-work-notifications.ts | 1 + .../gateway-credential-expiry-work.ts | 18 ++ server/src/services/tool-gateway.ts | 161 ++++++----- 16 files changed, 609 insertions(+), 75 deletions(-) create mode 100644 doc/idle-gateway-credential-expiry.md create mode 100644 packages/db/src/migrations/0381_mean_scream.sql rename packages/db/src/migrations/meta/{0375_snapshot.json => 0381_snapshot.json} (99%) create mode 100644 server/src/__tests__/gateway-credential-expiry.test.ts create mode 100644 server/src/services/gateway-credential-expiry-work.ts diff --git a/doc/idle-gateway-credential-expiry.md b/doc/idle-gateway-credential-expiry.md new file mode 100644 index 00000000000..e0486d39886 --- /dev/null +++ b/doc/idle-gateway-credential-expiry.md @@ -0,0 +1,141 @@ +# Gateway credential expiry and idle behavior + +Gateway sessions and named gateway tokens already persist their expiry times. +`cleanupExpiredSessions` remains the owner of physical deletion. Authentication +continues to check expiry, revocation, company scope, run ownership, and active +run state on use. No table, authorization rule, or external alarm is added. + +## Producer inventory + +- `createSession` registers intent before inserting a run-bound session in a + transaction. Its existing audit and returned credential remain unchanged. +- `createNamedGatewayToken` registers intent before inserting an expiring token + in a transaction. A token without an expiry owes no expiry cleanup and does + not notify this worker. +- Last-use updates do not change expiry. Token/session revocation, including + heartbeat completion and connection removal, also leaves the saved expiry + intact. Revoked rows retain the same physical deletion deadline. +- Gateway/company deletion removes dependent rows and can leave one harmless + stale deadline. It cannot create overdue work. No supported service changes + a saved credential expiry after creation. + +The database signal layer owns outer commit, rollback, savepoints, and uncertain +outcomes. The worker supports producers sharing that signal owner. Independent +SQL writers and another process need a separate live ingress contract. Startup +reconstructs existing rows but does not provide continuous replica discovery. + +## Worker contract + +`nextCredentialExpiryAt` reads one earliest non-null expiry from each table using +expiry-first indexes, then compares those at most two values. The migration adds +the session expiry index; the token expiry index already exists. +An empty credential store or retained non-expiring tokens returns no deadline. +`registerGatewayCredentialExpiryWork` supplies cleanup and deadline selection to +the shared `registerDeadlineWork` helper. Provider and authentication behavior +stay in the gateway service; startup only registers the worker. + +The worker starts where the old startup cleanup ran. With heartbeat scheduling +enabled, committed changes and actual expiry times replace the recurring cleanup +call. With heartbeat scheduling disabled, the existing startup-only cleanup is +preserved. Tool action-review recovery is a separate obligation and is unchanged. + +Token and session deletion are each bounded to 500 rows per pass. Session +selection orders by expiry and rechecks expiry when deleting the selected IDs. +A remaining overdue expiry schedules another pass after 30 seconds. Failed reconciliation also +retries at that bound. No unfinished reconciliation or future expiry means no +recurring SQL. A generation retains writes arriving during reconciliation. +Startup restores deadlines; drain retains first writes without passive SQL; +resume completes overdue cleanup. The coordinator owns shutdown and waits for +in-flight work. + +Sequential committed writes are coalesced for up to 30 seconds after a +successful cleanup pass, matching the former heartbeat cleanup cadence. A +previously loaded expiry can preempt that window. A newly reported credential +may physically remain until that bounded follow-up; authentication still rejects +it immediately at expiry. A successful pass with no future expiry and no new +notifications creates no timer. This uses the shared optional notification +coalescing capability, promoted from the later plugin/decision recovery slices; +workers that omit it retain their existing behavior. + +## Evidence and limits + +The concurrent session index runs through the migration executor's existing +per-file top-level path. Initial application and repeat application were tested. +A PostgreSQL plan with 20,000 sessions and 20,000 tokens, including 19,000 +non-expiring tokens, used both expiry indexes and read one row from each. +This establishes bounded lookup on that fixture, not production latency. + +The new PostgreSQL suite exercises real session/token creation, configured idle +state, non-expiring history, expiry ordering, revoked-token cleanup, outer commit +and rollback, startup reconstruction, a backlog larger than one cleanup batch, +and drain/resume. It tests expired authentication before deferred physical +cleanup, through both session and named-gateway entry points. Existing gateway +acceptance and server-startup tests cover the changed integration boundaries. + +Use only disposable migrated databases for the test URL overrides. Their names +must end in `_test`; the suites delete their fixtures. Ordinary CI uses embedded +PostgreSQL. Test credentials are synthetic and no external provider is contacted +by the expiry suite. + +The migration proves worker behavior, not host suspension. Existing Cloud idle +proof remains conservative. An in-process deadline cannot wake stopped compute; +external wake coverage must be verified before changing sleep eligibility. + +## Related unmerged token-lifetime work + +At implementation time, PRs #17221 and #17274 change run-scoped token lifetime +and cleanup eligibility. This slice preserves the current master policy and +changes scheduling only. When combining those changes, the deadline query and +producer signals must use the same token eligibility as cleanup: run-token idle +markers must not schedule the operator-token expiry worker. Keep the run-liveness +reaper with its own owner and retain both regression suites. A conflict-free +text merge alone does not prove this integration. + +## Notification amplification review repair (2026-10-27) + +Ten sequential real session/token creation transactions reproduced ten extra +instance-wide cleanup passes. With coalescing they cause one follow-up pass. +A separate 501-session fixture reproduced the unbounded delete, then passed +with 500 removed initially and the remaining overdue row removed at the saved +30-second retry. A still-valid session survives both passes. The tests also +preserve an already loaded expiry only 500 milliseconds away and verify that a +new one-second session is rejected while physical cleanup waits for coalescing. + +The initial full run exposed four outer-transaction tests whose fake-clock wait +expected the intent hold to release before the new coalescing follow-up. Their +settlement helper now advances the clock through that window while preserving +the original commit/rollback assertions. The complete gateway expiry, gateway +acceptance, custom-image deadline, and startup suites pass: 147 tests in four +files. Server typechecking also passes on the owning branch. This is local +behavioral evidence; combined-stack and fresh remote checks remain required. +The full local test:run remains unverified because of embedded PostgreSQL +startup/capacity limits; the two real gateway suites used separate migrated, +task-owned disposable PostgreSQL databases. + +A later combined run exposed a test-clock overshoot: repeated blind 30-second +advances could expire the future session before its intermediate assertion. +The settlement helper now lets real SQL settle and advances only to an actually +armed follow-up within the coalescing window. All 12 credential expiry tests pass +on the owning gateway branch with the existing exact-deadline and transaction +assertions intact. This follow-up changes the test fixture only. + + +## One-shot cleanup preserves complete backlog removal + +Review found that bounding worker passes also truncated startup cleanup when the +heartbeat scheduler was disabled. That path calls cleanup only once, so it could +leave every expired token or session after the first 500 rows indefinitely. + +`cleanupExpiredSessions` now drains the existing backlog in 500-row SQL batches +by default, using one fixed expiry cutoff for the invocation. The deadline worker +opts into `singleBatch: true` and retains its existing coalescing and next-page +deadline. Future and nonexpiring credentials survive both paths. The return value +still counts deleted sessions, as before. No periodic fallback was added. + +Three real-PostgreSQL regressions reproduced the truncation with 1,001 expired +sessions, tokens, or both, without registering a worker. They pass after the fix. +All 55 expiry/startup cases pass, including the existing worker's 501-row bounded +pass and commit/rollback tests. The original explicit-cleanup compatibility case +and server typecheck also pass. Evidence is in `/tmp/sleep-master-gateway-manual-` +logs: `red`, `green`, `compatibility`, and `types`. This fixes one-shot behavior; +it does not establish provider suspension or complete the stack's CI acceptance. diff --git a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md index 9141878caa8..38dec000a0e 100644 --- a/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md +++ b/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md @@ -649,8 +649,10 @@ Longer polling intervals alone do not complete an item. - [ ] **JOB-21 — Secret proposals and expired action tokens (partially migrated).** Secret-proposal expiry now uses transaction signals and saved expiry deadlines; - see [the lifecycle contract](../idle-secret-proposal-expiry.md). Tool review - recovery and gateway token cleanup still use **30s** queries. Change: enforce expiry synchronously on use and + see [the lifecycle contract](../idle-secret-proposal-expiry.md). Gateway token + and session expiry now also uses saved deadlines and producer signals; see + [its lifecycle contract](../idle-gateway-credential-expiry.md). Tool review + recovery still uses **30s** queries. Change: enforce expiry synchronously on use and register required scrub/cleanup deadlines. **Complete when:** idle cleanup does not extend authorization or secret-retention periods. Sources: [secret-proposals.ts](../../server/src/services/secret-proposals.ts), diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 20221274e70..009401a1d93 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -18,7 +18,7 @@ Do not mark an item done from a timer removal or a green check alone. | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | -| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret-proposal expiry in progress; tool reviews and gateway tokens pending | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged after CI and passing reviews; gateway credential expiry in progress; tool reviews pending | | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 7f067cbfc1c..77fb0c62fa1 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -501,6 +501,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0332_little_night_nurse.sql", "0345_luxuriant_carmella_unuscione.sql", "0369_late_logan.sql", + "0381_mean_scream.sql", ]); async function applyPendingMigrationsManually( diff --git a/packages/db/src/migrations/0381_mean_scream.sql b/packages/db/src/migrations/0381_mean_scream.sql new file mode 100644 index 00000000000..8827365ffdd --- /dev/null +++ b/packages/db/src/migrations/0381_mean_scream.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "tool_gateway_sessions_expiry_idx" ON "tool_gateway_sessions" USING btree ("expires_at"); diff --git a/packages/db/src/migrations/meta/0375_snapshot.json b/packages/db/src/migrations/meta/0381_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0375_snapshot.json rename to packages/db/src/migrations/meta/0381_snapshot.json index 7bd1861bd2c..23802aaeece 100644 --- a/packages/db/src/migrations/meta/0375_snapshot.json +++ b/packages/db/src/migrations/meta/0381_snapshot.json @@ -1,6 +1,6 @@ { - "id": "3e17203b-ec95-41e9-8363-685982450199", - "prevId": "945c31ed-f39f-4272-a91d-85de925f5361", + "id": "2e93deb7-f24c-42d1-8c8e-30b21cff90bc", + "prevId": "bfda87fa-662b-47bb-87c0-dabc814aebd7", "version": "7", "dialect": "postgresql", "tables": { @@ -47782,6 +47782,18 @@ "primaryKey": false, "notNull": false }, + "git_authorization_grant_id": { + "name": "git_authorization_grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "git_authorized_by_user_id": { + "name": "git_authorized_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "pause_reason": { "name": "pause_reason", "type": "text", @@ -47920,6 +47932,19 @@ ], "onDelete": "no action", "onUpdate": "no action" + }, + "projects_git_authorization_grant_id_connection_grants_id_fk": { + "name": "projects_git_authorization_grant_id_connection_grants_id_fk", + "tableFrom": "projects", + "tableTo": "connection_grants", + "columnsFrom": [ + "git_authorization_grant_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" } }, "compositePrimaryKeys": {}, @@ -55197,6 +55222,21 @@ "method": "btree", "with": {} }, + "tool_gateway_sessions_expiry_idx": { + "name": "tool_gateway_sessions_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_gateway_sessions_run_idx": { "name": "tool_gateway_sessions_run_idx", "columns": [ @@ -55942,6 +55982,12 @@ "notNull": true, "default": "''" }, + "responsible_user_id": { + "name": "responsible_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "owner_note": { "name": "owner_note", "type": "text", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 96c505941fd..176a963f7a4 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2654,6 +2654,13 @@ "when": 1793060791724, "tag": "0380_broken_aqueduct", "breakpoints": true + }, + { + "idx": 381, + "version": "7", + "when": 1793060791725, + "tag": "0381_mean_scream", + "breakpoints": true } ] } diff --git a/packages/db/src/schema/tool_access.ts b/packages/db/src/schema/tool_access.ts index f4aede1b4cd..825eb91589f 100644 --- a/packages/db/src/schema/tool_access.ts +++ b/packages/db/src/schema/tool_access.ts @@ -714,6 +714,7 @@ export const toolGatewaySessions = pgTable( uniqueIndex("tool_gateway_sessions_token_hash_uq").on(table.tokenHash), index("tool_gateway_sessions_company_agent_idx").on(table.companyId, table.agentId), index("tool_gateway_sessions_company_expires_idx").on(table.companyId, table.expiresAt), + index("tool_gateway_sessions_expiry_idx").on(table.expiresAt), index("tool_gateway_sessions_run_idx").on(table.companyId, table.runId), index("tool_gateway_sessions_issue_idx").on(table.companyId, table.issueId), index("tool_gateway_sessions_gateway_idx").on(table.companyId, table.gatewayId), diff --git a/server/src/__tests__/gateway-credential-expiry.test.ts b/server/src/__tests__/gateway-credential-expiry.test.ts new file mode 100644 index 00000000000..caa76674104 --- /dev/null +++ b/server/src/__tests__/gateway-credential-expiry.test.ts @@ -0,0 +1,269 @@ +import { randomUUID } from "node:crypto"; +import { sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, agents, companies, createDb, heartbeatRuns, toolAccessAuditEvents, toolGatewaySessions, + toolMcpGateways, toolMcpGatewayTokens, toolProfiles } from "@paperclipai/db"; +import { createToolGatewayService } from "../services/tool-gateway.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { registerGatewayCredentialExpiryWork } from "../services/gateway-credential-expiry-work.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_GATEWAY_EXPIRY_TEST_DATABASE_URL"); +const supported = external ? true : (await getEmbeddedPostgresTestSupport()).supported; + +describe.skipIf(!supported)("gateway credential expiry", () => { + let db: ReturnType; + let temporary: Awaited> | undefined; + let gateway: ReturnType; + let coordinator: ReturnType | undefined; + let companyId: string, agentId: string, runId: string, namedId: string; + let draining: boolean; + const errors: unknown[] = []; + + beforeAll(async () => { + if (!external) temporary = await startEmbeddedPostgresTestDatabase("gateway-expiry-"); + db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 90_000); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + beforeEach(async () => { + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + companyId = randomUUID(); agentId = randomUUID(); runId = randomUUID(); draining = false; errors.length = 0; + await db.insert(companies).values({ id: companyId, name: "Credential expiry", issuePrefix: "EXP" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Agent", role: "engineer", adapterType: "process" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, invocationSource: "assignment", status: "running" }); + const [profile] = await db.insert(toolProfiles).values({ companyId, profileKey: "expiry", name: "Expiry", defaultAction: "deny" }).returning(); + gateway = createToolGatewayService(db, { toolActionSigningSecret: "expiry-test-only" }); + namedId = (await gateway.createNamedGateway({ companyId, body: { name: "Expiry", profileId: profile.id } })).id; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.delete(activityLog); + await db.delete(toolAccessAuditEvents); + await db.delete(toolGatewaySessions); + await db.delete(toolMcpGatewayTokens); + await db.delete(toolMcpGateways); + await db.delete(toolProfiles); + await db.delete(heartbeatRuns); + await db.delete(agents); + await db.delete(companies); + expect(errors).toEqual([]); + }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerGatewayCredentialExpiryWork(coordinator, gateway, () => !draining).ready; + } + async function settle(ms = 60_000) { + await vi.advanceTimersByTimeAsync(ms); + if (!draining) await vi.waitFor(async () => { + // Let real SQL settle before advancing only to an actually armed follow-up. + // Repeated blind clock jumps can expire the future credential under test. + if (idleWorkSnapshot().active > 0) { + await db.execute(sql`select pg_sleep(0.01)`); + const next = coordinator!.nextWakeAt(); + if (next !== null && next <= Date.now() + 30_000) { + await vi.advanceTimersByTimeAsync(Math.max(0, next - Date.now())); + } + } + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + } + async function quiet() { + await settle(); expect(coordinator!.nextWakeAt()).toBeNull(); + const reads = vi.spyOn(db, "execute"), selects = vi.spyOn(db, "select"), deletes = vi.spyOn(db, "delete"); + await settle(60 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(selects).not.toHaveBeenCalled(); expect(deletes).not.toHaveBeenCalled(); + reads.mockRestore(); selects.mockRestore(); deletes.mockRestore(); + } + const token = (expiresAt?: Date) => gateway.createNamedGatewayToken({ companyId, gatewayId: namedId, body: { name: "Expiry", expiresAt } }); + const session = (service = gateway) => service.createSession({ companyId, agentId, runId, ttlMs: 5 * 60_000 }); + + it("stays quiet with configuration and a nonexpiring token, then expires real tokens and sessions by deadline", async () => { + await start(); await quiet(); + const retained = await token(); await quiet(); + const active = await session(); + const due = new Date(Date.now() + 3 * 60_000); + const expiring = await token(due); + await settle(); + expect(coordinator!.nextWakeAt()).toBe(due.getTime()); + const reads = vi.spyOn(db, "execute"), selects = vi.spyOn(db, "select"); + await settle(30_000); + expect(reads).not.toHaveBeenCalled(); expect(selects).not.toHaveBeenCalled(); + reads.mockRestore(); selects.mockRestore(); + await gateway.revokeNamedGatewayToken({ companyId, tokenId: expiring.id }); + await settle(2 * 60_000); + expect((await db.select().from(toolMcpGatewayTokens)).map(row => row.id)).toEqual([retained.id]); + expect(coordinator!.nextWakeAt()).toBe(active.expiresAt.getTime()); + await settle(2 * 60_000); + expect(await db.select().from(toolGatewaySessions)).toEqual([]); + await quiet(); + }); + + it.each([ + { kind: "session", rollback: false }, { kind: "session", rollback: true }, + { kind: "token", rollback: false }, { kind: "token", rollback: true }, + ])("respects $kind outer transaction settlement (rollback $rollback)", async ({ kind, rollback }) => { + await start(); await quiet(); + let release!: () => void, written!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const staged = new Promise(resolve => { written = resolve; }); + const rollbackError = new Error("rollback fixture"); + const writing = db.transaction(async tx => { + const scoped = createToolGatewayService(tx as unknown as typeof db, { toolActionSigningSecret: "expiry-test-only" }); + if (kind === "session") await session(scoped); + else await scoped.createNamedGatewayToken({ companyId, gatewayId: namedId, + body: { name: "Outer transaction", expiresAt: new Date(Date.now() + 5 * 60_000) } }); + written(); await gate; if (rollback) throw rollbackError; + }).catch(error => { if (error !== rollbackError) throw error; }); + await Promise.race([staged, writing]); + try { + await vi.advanceTimersByTimeAsync(60_000); + expect(await db.select().from(toolGatewaySessions)).toEqual([]); + expect(await db.select().from(toolMcpGatewayTokens)).toEqual([]); + } finally { release(); } + await writing; await settle(); + if (rollback) await quiet(); + else { + expect(coordinator!.nextWakeAt()).toBe(await gateway.nextCredentialExpiryAt()); + await settle(5 * 60_000); await quiet(); + } + }); + + it("coalesces sequential credential writes within the former cleanup cadence", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + const cleanup = vi.spyOn(gateway, "cleanupExpiredSessions"); + const next = vi.spyOn(gateway, "nextCredentialExpiryAt"); + await start(); + const firstPassAt = Date.now(); + for (let i = 0; i < 10; i++) { + if (i % 2 === 0) await session(); + else await token(new Date(Date.now() + 5 * 60_000)); + await vi.advanceTimersByTimeAsync(1); + // Let each old scan finish so this measures sequential writes, not overlapping calls. + await db.execute(sql`select pg_sleep(0.01)`); + } + expect(cleanup).toHaveBeenCalledTimes(1); + expect(next).toHaveBeenCalledTimes(1); + expect(coordinator!.nextWakeAt()).toBe(firstPassAt + 30_000); + await vi.advanceTimersByTimeAsync(30_000); + await vi.waitFor(() => expect(next).toHaveBeenCalledTimes(2)); + await db.execute(sql`select pg_sleep(0.01)`); + expect(cleanup).toHaveBeenCalledTimes(2); + expect(coordinator!.nextWakeAt()).toBeGreaterThan(Date.now()); + await settle(5 * 60_000); await quiet(); + }); + + it("keeps a loaded expiry deadline ahead of credential notification coalescing", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + const deadline = Date.now() + 500; + await token(new Date(deadline)); + await start(); + expect(coordinator!.nextWakeAt()).toBe(deadline); + await session(); + await vi.advanceTimersByTimeAsync(1); + await db.execute(sql`select pg_sleep(0.01)`); + expect(coordinator!.nextWakeAt()).toBe(deadline); + await vi.advanceTimersByTimeAsync(499); + await vi.waitFor(async () => expect(await db.select().from(toolMcpGatewayTokens)).toEqual([])); + expect(await db.select().from(toolGatewaySessions)).toHaveLength(1); + await settle(5 * 60_000); await quiet(); + }); + + it("rejects a newly expired credential while physical cleanup is coalesced", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + const cleanup = vi.spyOn(gateway, "cleanupExpiredSessions"); + await start(); + const at = Date.now(); + const active = await gateway.createSession({ companyId, agentId, runId, ttlMs: 1000 }); + await vi.advanceTimersByTimeAsync(1000); + await db.execute(sql`select pg_sleep(0.01)`); + expect(cleanup).toHaveBeenCalledTimes(1); + expect(await db.select().from(toolGatewaySessions)).toHaveLength(1); + await expect(gateway.listToolsForSession(active.token)).rejects.toMatchObject({ status: 401, reasonCode: "session_expired" }); + expect(coordinator!.nextWakeAt()).toBe(at + 30_000); + await vi.advanceTimersByTimeAsync(29_000); + await vi.waitFor(async () => expect(await db.select().from(toolGatewaySessions)).toEqual([])); + await quiet(); + }); + + it.each(["sessions", "tokens", "both"] as const)("drains the full %s backlog without a registered scheduler", async kind => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + const now = new Date(); + const futureSession = await session(); + const futureToken = await token(new Date(now.getTime() + 60_000)); + const nonexpiringToken = await token(); + if (kind !== "tokens") await db.insert(toolGatewaySessions).values(Array.from({ length: 1001 }, () => ({ + companyId, agentId, runId, tokenHash: randomUUID(), expiresAt: new Date(now.getTime() - 1), + }))); + if (kind !== "sessions") await db.insert(toolMcpGatewayTokens).values(Array.from({ length: 1001 }, () => ({ + companyId, gatewayId: namedId, name: "Expired", tokenHash: randomUUID(), expiresAt: new Date(now.getTime() - 1), + }))); + await expect(gateway.cleanupExpiredSessions({ now })).resolves.toEqual({ deletedCount: kind === "tokens" ? 0 : 1001 }); + expect((await db.select().from(toolGatewaySessions)).map(row => row.id)).toEqual([futureSession.id]); + expect((await db.select().from(toolMcpGatewayTokens)).map(row => row.id).sort()).toEqual([futureToken.id, nonexpiringToken.id].sort()); + expect(coordinator).toBeUndefined(); + expect(idleWorkSnapshot().active).toBe(0); + }); + + it("bounds session cleanup and schedules the remaining overdue batch", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + await db.insert(toolGatewaySessions).values(Array.from({ length: 501 }, () => ({ + companyId, agentId, runId, tokenHash: randomUUID(), expiresAt: new Date(Date.now() - 1), + }))); + const future = await session(); + const cleanup = vi.spyOn(gateway, "cleanupExpiredSessions"); + await start(); + expect(cleanup).toHaveResolvedWith({ deletedCount: 500 }); + expect(await db.select().from(toolGatewaySessions)).toHaveLength(2); + expect(coordinator!.nextWakeAt()).toBe(Date.now() + 30_000); + await vi.advanceTimersByTimeAsync(29_999); + expect(cleanup).toHaveBeenCalledTimes(1); + await vi.advanceTimersByTimeAsync(1); + await vi.waitFor(async () => expect((await db.select().from(toolGatewaySessions)).map(row => row.id)).toEqual([future.id])); + await settle(5 * 60_000); await quiet(); + }); + + it("reconstructs saved token expiry on startup", async () => { + const due = new Date(Date.now() + 5 * 60_000); + await token(due); + await start(); await settle(); + expect(coordinator!.nextWakeAt()).toBe(due.getTime()); + await settle(5 * 60_000); + expect(await db.select().from(toolMcpGatewayTokens)).toEqual([]); + await quiet(); + }); + + it("drains an overdue persisted backlog across bounded batches", async () => { + await db.insert(toolMcpGatewayTokens).values(Array.from({ length: 501 }, (_, index) => ({ + companyId, gatewayId: namedId, name: `Expired ${index}`, tokenHash: randomUUID(), + tokenPrefix: "expired", expiresAt: new Date(Date.now() - 1), + }))); + await start(); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + await quiet(); + expect(await db.select().from(toolMcpGatewayTokens)).toEqual([]); + }); + + it("retains cleanup through drain without permitting expired authentication", async () => { + await start(); await quiet(); draining = true; + const active = await session(); + const expiredToken = await token(new Date(Date.now() + 3 * 60_000)); + await settle(); + const reads = vi.spyOn(db, "execute"), selects = vi.spyOn(db, "select"); + await settle(10 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(selects).not.toHaveBeenCalled(); + reads.mockRestore(); selects.mockRestore(); + expect(await db.select().from(toolGatewaySessions)).toHaveLength(1); + await expect(gateway.listToolsForSession(active.token)).rejects.toMatchObject({ status: 401, reasonCode: "session_expired" }); + await expect(gateway.initializeNamedGatewayProtocol({ gatewayId: namedId, bearerToken: expiredToken.token })) + .rejects.toMatchObject({ status: 401 }); + draining = false; await settle(); + expect(await db.select().from(toolGatewaySessions)).toEqual([]); + expect(await db.select().from(toolMcpGatewayTokens)).toEqual([]); + await quiet(); + }); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index f3d7171c03a..9788b8599ff 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -71,6 +71,7 @@ const { toolGateway: { sweepActionReviews: vi.fn(async () => ({ scanned: 0 })), cleanupExpiredSessions: vi.fn(async () => ({ deletedCount: 0 })), + nextCredentialExpiryAt: vi.fn(async () => null), }, toolActionDeliveries: { hasPending: vi.fn(async () => false), sweepPending: vi.fn(async () => ({ scanned: 0, delivered: 0 })) }, }, diff --git a/server/src/__tests__/tool-gateway.test.ts b/server/src/__tests__/tool-gateway.test.ts index a1f0b553d17..4e9ba662bdf 100644 --- a/server/src/__tests__/tool-gateway.test.ts +++ b/server/src/__tests__/tool-gateway.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { createHash, randomUUID } from "node:crypto"; import { createServer, type IncomingMessage } from "node:http"; import express from "express"; @@ -65,8 +66,8 @@ import { startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; -const externalTestDatabaseUrl = process.env.PAPERCLIP_TOOL_GATEWAY_TEST_DATABASE_URL; -const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const externalTestDatabase = resolveExternalTestDatabaseUrl("PAPERCLIP_TOOL_GATEWAY_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalTestDatabase ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; const testToolActionSigningSecret = "test-tool-action-signing-secret"; @@ -554,8 +555,9 @@ describeEmbeddedPostgres("tool gateway acceptance", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-gateway-"); - db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString); + if (externalTestDatabase) { db = createDb(externalTestDatabase); return; } + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-gateway-"); + db = createDb(tempDb.connectionString); }, 90_000); afterEach(async () => { diff --git a/server/src/index.ts b/server/src/index.ts index ca51efe4345..4116e46bd93 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerGatewayCredentialExpiryWork } from "./services/gateway-credential-expiry-work.js"; import { registerSecretProposalExpiryWork } from "./services/secret-proposal-expiry-work.js"; import { registerCustomImageCleanupWork } from "./services/custom-image-cleanup-work.js"; import { registerWorkspaceRecoveryWork } from "./services/workspace-recovery-work.js"; @@ -1464,7 +1465,10 @@ async function startServerWithDatabaseTeardown( }; await timeStartupRecovery("tool_reviews", () => app.locals.toolGateway.sweepActionReviews()).catch((err: unknown) => logger.error({ err }, "startup tool review recovery failed")); - await timeStartupRecovery("gateway_sessions", () => app.locals.toolGateway.cleanupExpiredSessions()).catch((err: unknown) => logger.error({ err }, "startup gateway token cleanup failed")); + await timeStartupRecovery("gateway_sessions", () => config.heartbeatSchedulerEnabled + ? registerGatewayCredentialExpiryWork(deliveryWork, app.locals.toolGateway, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready + : app.locals.toolGateway.cleanupExpiredSessions()).catch((err: unknown) => logger.error({ err }, "startup gateway token cleanup failed")); await timeStartupRecovery("delivery_workers", () => Promise.all([budgetEnforcementWorker, accountingWorker, fastResponseWorker, ...deliveryWorkers].map(worker => worker.ready))); scheduleGitHubConnectionEventPoll(); scheduleGitHubConnectionContinuitySweep(); @@ -1891,7 +1895,6 @@ async function startServerWithDatabaseTeardown( })); trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews().catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); - trackHeartbeatSchedulerWork(app.locals.toolGateway.cleanupExpiredSessions().catch((err: unknown) => logger.error({ err }, "gateway token cleanup failed"))); if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { diff --git a/server/src/services/deadline-work.ts b/server/src/services/deadline-work.ts index 9f1de1b2a24..044c3cbfce4 100644 --- a/server/src/services/deadline-work.ts +++ b/server/src/services/deadline-work.ts @@ -9,11 +9,14 @@ export function registerDeadlineWork( next: () => Promise; canRun: () => boolean; retryMs: number; + // Optional batching latency for notifications; already-known deadlines still preempt it. + notificationCoalesceMs?: number; }, ) { let requested = 1, reconciled = 0; let next: number | null = null; let earliest = 0; + let coalesceUntil = 0; return coordinator.register(queue, { retryMs: task.retryMs, pausePassiveMaintenance: true, @@ -21,6 +24,10 @@ export function registerDeadlineWork( run: async () => { if (!task.canRun()) { requested++; return; } if (Date.now() < earliest) { next = earliest; return; } + if (Date.now() < coalesceUntil && (next === null || next > Date.now())) { + next = next === null ? coalesceUntil : Math.min(next, coalesceUntil); + return; + } const generation = ++requested; earliest = Date.now() + task.retryMs; await task.run(); @@ -28,6 +35,7 @@ export function registerDeadlineWork( // Back off failed or still-overdue work, not a successfully read future deadline. if (deadline === null || deadline > Date.now()) earliest = 0; next = deadline === null ? null : Math.max(earliest, deadline); + coalesceUntil = Date.now() + (task.notificationCoalesceMs ?? 0); reconciled = generation; }, hasPending: async () => requested !== reconciled, diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index b6bed1d0f25..191b8aef36b 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + gatewayCredentialExpiry: "gateway-credential-expiry", secretProposalExpiry: "secret-proposal-expiry", customImageExpiry: "custom-image-expiry", historicalCleanup: "historical-cleanup", diff --git a/server/src/services/gateway-credential-expiry-work.ts b/server/src/services/gateway-credential-expiry-work.ts new file mode 100644 index 00000000000..2cf6097b6d8 --- /dev/null +++ b/server/src/services/gateway-credential-expiry-work.ts @@ -0,0 +1,18 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { createToolGatewayService } from "./tool-gateway.js"; + +export function registerGatewayCredentialExpiryWork( + coordinator: ReturnType, + gateway: Pick, "cleanupExpiredSessions" | "nextCredentialExpiryAt">, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.gatewayCredentialExpiry, { + retryMs: 30_000, + notificationCoalesceMs: 30_000, + canRun, + run: () => gateway.cleanupExpiredSessions({ singleBatch: true }), + next: () => gateway.nextCredentialExpiryAt(), + }); +} diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index d2d703dfcd6..c75413e8529 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -9210,45 +9210,48 @@ export function createToolGatewayService( const token = generateNamedGatewayToken(tokenId); const tokenPrefix = `pcgw_${tokenId.slice(0, 8)}`; const now = new Date(); - const [row] = await db - .insert(toolMcpGatewayTokens) - .values({ - id: tokenId, - companyId: input.companyId, - gatewayId: input.gatewayId, - name: input.body.name, - tokenHash: hashGatewayToken(token), - tokenPrefix, - subjectType: input.body.subjectType ?? "gateway_client", - subjectId: input.body.subjectId ?? null, - clientLabel: input.body.clientLabel, - ownerNote: input.body.ownerNote, - allowedActions: input.body.allowedActions ?? [ - "tools/list", - "tools/call", - "resources/list", - "resources/read", - "prompts/list", - "prompts/get", - ], - expiresAt: input.body.expiresAt ?? null, - expiryOverrideReason: input.body.expiryOverrideReason ?? null, - expiryOverrideByAgentId: - input.actor?.agentId && input.body.expiryOverrideReason - ? input.actor.agentId - : null, - expiryOverrideByUserId: - input.actor?.userId && input.body.expiryOverrideReason - ? input.actor.userId - : null, - expiryOverrideAt: input.body.expiryOverrideReason ? now : null, - responsibleUserId: input.managementContext?.actor.type === "agent" ? input.managementContext.responsibleUserId : null, - createdByAgentId: input.actor?.agentId ?? null, - createdByUserId: input.actor?.userId ?? null, - createdAt: now, - updatedAt: now, - }) - .returning(); + const [row] = await db.transaction(async tx => { + if (input.body.expiresAt) await notifyDeliveryWork(tx, DELIVERY_QUEUES.gatewayCredentialExpiry); + return tx + .insert(toolMcpGatewayTokens) + .values({ + id: tokenId, + companyId: input.companyId, + gatewayId: input.gatewayId, + name: input.body.name, + tokenHash: hashGatewayToken(token), + tokenPrefix, + subjectType: input.body.subjectType ?? "gateway_client", + subjectId: input.body.subjectId ?? null, + clientLabel: input.body.clientLabel, + ownerNote: input.body.ownerNote, + allowedActions: input.body.allowedActions ?? [ + "tools/list", + "tools/call", + "resources/list", + "resources/read", + "prompts/list", + "prompts/get", + ], + expiresAt: input.body.expiresAt ?? null, + expiryOverrideReason: input.body.expiryOverrideReason ?? null, + expiryOverrideByAgentId: + input.actor?.agentId && input.body.expiryOverrideReason + ? input.actor.agentId + : null, + expiryOverrideByUserId: + input.actor?.userId && input.body.expiryOverrideReason + ? input.actor.userId + : null, + expiryOverrideAt: input.body.expiryOverrideReason ? now : null, + responsibleUserId: input.managementContext?.actor.type === "agent" ? input.managementContext.responsibleUserId : null, + createdByAgentId: input.actor?.agentId ?? null, + createdByUserId: input.actor?.userId ?? null, + createdAt: now, + updatedAt: now, + }) + .returning(); + }); if (input.managementContext) await logActivity(db, { companyId: input.companyId, ...input.managementContext.audit, action: "tool_gateway.token_created", entityType: "tool_mcp_gateway", entityId: input.gatewayId, details: { tokenId: row.id, subjectType: row.subjectType, allowedActions: row.allowedActions } }); @@ -9373,18 +9376,21 @@ export function createToolGatewayService( expiresAt: new Date(now.getTime() + sessionTtlMs(input.ttlMs)), }; - await db.insert(toolGatewaySessions).values({ - id: session.id, - companyId: session.companyId, - agentId: input.agentId, - runId: input.runId, - issueId: session.issueId, - projectId: session.projectId, - tokenHash: hashGatewayToken(token), - expiresAt: session.expiresAt, - createdAt: session.createdAt, - updatedAt: session.createdAt, - } as any); + await db.transaction(async tx => { + await notifyDeliveryWork(tx, DELIVERY_QUEUES.gatewayCredentialExpiry); + await tx.insert(toolGatewaySessions).values({ + id: session.id, + companyId: session.companyId, + agentId: input.agentId, + runId: input.runId, + issueId: session.issueId, + projectId: session.projectId, + tokenHash: hashGatewayToken(token), + expiresAt: session.expiresAt, + createdAt: session.createdAt, + updatedAt: session.createdAt, + } as any); + }); await writeAudit({ session, @@ -11860,23 +11866,50 @@ export function createToolGatewayService( return { ...sessionView, revokedAt: session!.revokedAt ?? now }; }, - async cleanupExpiredSessions(input: { now?: Date } = {}) { + async nextCredentialExpiryAt(): Promise { + const [row] = await db.execute<{ at: string | Date | null }>(sql` + select min(expires_at) as at from ( + (select ${toolMcpGatewayTokens.expiresAt} as expires_at from ${toolMcpGatewayTokens} + where ${toolMcpGatewayTokens.expiresAt} is not null + order by ${toolMcpGatewayTokens.expiresAt} limit 1) + union all + (select ${toolGatewaySessions.expiresAt} as expires_at from ${toolGatewaySessions} + order by ${toolGatewaySessions.expiresAt} limit 1) + ) expiries + `); + return row?.at == null ? null : new Date(row.at).getTime(); + }, + + async cleanupExpiredSessions(input: { now?: Date; singleBatch?: boolean } = {}) { const now = input.now ?? new Date(); - const expiredTokens = await db.select({ id: toolMcpGatewayTokens.id }) - .from(toolMcpGatewayTokens) - .where(lte(toolMcpGatewayTokens.expiresAt, now)) - .orderBy(asc(toolMcpGatewayTokens.expiresAt), asc(toolMcpGatewayTokens.id)) - .limit(500); - if (expiredTokens.length > 0) { - await db.delete(toolMcpGatewayTokens) + let deletedCount = 0; + while (true) { + const expiredTokens = await db.select({ id: toolMcpGatewayTokens.id }) + .from(toolMcpGatewayTokens) + .where(lte(toolMcpGatewayTokens.expiresAt, now)) + .orderBy(asc(toolMcpGatewayTokens.expiresAt), asc(toolMcpGatewayTokens.id)) + .limit(500); + if (expiredTokens.length > 0) { + await db.delete(toolMcpGatewayTokens) .where(and(inArray(toolMcpGatewayTokens.id, expiredTokens.map((token) => token.id)), lte(toolMcpGatewayTokens.expiresAt, now))) .returning({ id: toolMcpGatewayTokens.id }); + } + const expiredSessions = await db.select({ id: toolGatewaySessions.id }) + .from(toolGatewaySessions) + .where(lte(toolGatewaySessions.expiresAt, now)) + .orderBy(asc(toolGatewaySessions.expiresAt), asc(toolGatewaySessions.id)) + .limit(500); + if (expiredSessions.length > 0) { + const rows = await db.delete(toolGatewaySessions) + .where(and(inArray(toolGatewaySessions.id, expiredSessions.map((session) => session.id)), lte(toolGatewaySessions.expiresAt, now))) + .returning({ id: toolGatewaySessions.id }); + deletedCount += rows.length; + } + // The worker schedules the next page; one-shot startup/manual callers + // must finish the backlog without relying on another scheduled pass. + if (input.singleBatch || (expiredTokens.length < 500 && expiredSessions.length < 500)) break; } - const rows = await db - .delete(toolGatewaySessions) - .where(lte(toolGatewaySessions.expiresAt, now)) - .returning({ id: toolGatewaySessions.id }); - return { deletedCount: rows.length }; + return { deletedCount }; }, async listRuntimeSlots(companyId?: string) { From bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 15:10:26 -0500 Subject: [PATCH 05/41] fix(evals): qualify model spend separately from operating costs (#17928) ## Thinking Path > - Paperclip manages agent work and its budgets. > - Product E2E campaigns must explain each model request and attempt. > - The current completion gate also requires complete operating-cost evidence. > - Long-lived observability and storage charges can block a campaign after its model receipts are complete. > - An operator can now choose model-request readiness in a retained allowance. > - The report keeps full accounting, operating holds and model readiness separate. ## Linked Issues or Issue Description Refs #17889, #17523, #17452. **What happened?** A hosted campaign could pass both product cases and retain complete model receipts, yet fail readiness because operating charges were unresolved. **Expected behavior** An explicit model-readiness policy permits complete model receipts to qualify a run. Unknown operating charges stay visible and reserved. Missing model receipts and budget violations still block qualification. **Steps to reproduce** Use a campaign with four unresolved operating reservations and one completed model request. The regression checks readiness before and after the final model receipt, then checks an operating overrun. The default policy retains the original full-accounting gate. ## What Changed - Add the explicit `accountingReadiness: "model_requests"` allowance policy. Bind it into operating-bound configuration when present. - Derive a model subtotal and receipt-coverage projection from the retained journal. Show it beside full accounting and retained holds. - Use the selected readiness policy in scheduled finalization. Keep product, cleanup, feedback, trace and budget checks. - Refresh two existing eval prerequisites found by the full unit suite: the active Copilot image identity and the model-account catalog mapping. - Document the policy and its limit: it does not certify a full-lifecycle operating-cost ceiling. ## Verification - Regression failed on the old implementation with `Invalid operating overhead fields` and passed after the change. - Focused accounting/operations/finalization checks: 54 tests passed before the final policy assertions. - Existing image/catalog failures reproduced, then their 12 checks passed after the metadata repairs. - Product E2E unit suite: 4,360 passed and 2 skipped, plus 137 Node checks. Product E2E and workspace typechecks and workspace build passed. - Review regressions: 35 focused accounting/finalization checks passed on 4aeb8bd65c, including the optional-policy hash and the scheduled consumer. - Full local server tests encountered embedded PostgreSQL bootstrap failures; retained output is not a passing full-suite claim. Hosted CI remains required. - Final head 4aeb8bd65c: all 57 check runs pass or are skipped, the legacy security status passes, both reviews are 5/5 and no review threads remain open. - No paid campaign or schedule activation is claimed by this PR. ## Risks The opt-in policy permits readiness while operating costs remain unreconciled. The report retains full-accounting status and all holds. The gateway still reserves requests inside the same total allowance. This is an admission limit, not a vendor invoice guarantee. The omitted policy keeps existing behavior. Old journals remain unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/OPERATIONS.md | 14 ++++- tests/runner-e2e/README.md | 19 +++++++ .../runner-e2e/chat-scenario-catalog.test.ts | 2 +- .../chat-scenarios/runs-adapters.yaml | 1 + tests/runner-e2e/daytona-image-content.ts | 1 + tests/runner-e2e/operating-overhead.test.ts | 54 ++++++++++++++++++- tests/runner-e2e/operating-overhead.ts | 5 +- tests/runner-e2e/operations-report.ts | 35 +++++++++++- .../runner-e2e/scheduled-finalization.test.ts | 34 ++++++++++++ tests/runner-e2e/scheduled-finalization.ts | 5 +- 10 files changed, 162 insertions(+), 8 deletions(-) diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index 7d0015d3a6a..1481b4451b1 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -223,7 +223,8 @@ stable key-sorted JSON hashing. The model configuration contains `provider`, `mode` (the manifest route), `price`, `maximumInputTokens`, `maximumOutputTokens`, and `reservation` (the nine-decimal value from `evalRequestReservation`). An operating configuration contains allowance -`reference`, `runner`, `verifiedAt`, `expiresAt`, and a `component` object with +`reference`, `runner`, `verifiedAt`, `expiresAt`, optional `accountingReadiness` +(only when present), and a `component` object with its `component`, `maxUsd`, and `reference`. Changing any of those values invalidates the approval hash. Evidence metadata itself is excluded from this configuration hash. @@ -897,3 +898,14 @@ an operations manifest. They use the same metered connection, prices, campaign journal, setup boundary, and durable attempt allowance as core compatibility. Collection and the independent saved-evidence oracle require no additional model call. The synthetic items need review as test data, not product incidents. + + +### Qualify model spend while operating reconciliation is pending + +Set `accountingReadiness: "model_requests"` on the reviewed allowance to use +complete model receipts as the scheduled readiness gate. This is an explicit +operator policy. The report retains the full-accounting status, operating +reservations and model-readiness status separately. It does not release any +historical hold or certify a full-lifecycle vendor cost bound. Missing model +receipts, conflicts, active requests, excess commitments, and the existing +product, cleanup, feedback and trace checks still fail qualification. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index a97116cc818..554b10b32d1 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -2983,3 +2983,22 @@ quality or survival after an abrupt worker kill. pnpm test:e2e:runner -- --list --suite feedback-persistence pnpm test:e2e:runner -- --id feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion --max-automatic-retries 0 --max-parallel 1 ``` + + +### Model-spend readiness and operating reconciliation + +A reviewed operating allowance can opt into `accountingReadiness: "model_requests"`. +This policy requires at least one model request and complete final model-cost +receipts before scheduled qualification succeeds. Missing or conflicting model +receipts, active requests, failed product/cleanup/trace/feedback checks, and a +campaign commitment above its limit still fail qualification. The gateway still +reserves every request before forwarding it. Attempts, deadlines and concurrency +remain bounded. + +The report shows the model subtotal and model-readiness status separately from +full accounting status. Unknown operating charges remain unknown. Operating +reservations stay in the same campaign admission allowance until reconciled; +they are not reported as charges or converted into reviewed lifecycle bounds. +This mode therefore does not certify a full-lifecycle operating-cost ceiling. +Omitting the policy preserves the stricter full-accounting gate. Historical +journals and their holds are not rewritten by changing future campaign policy. diff --git a/tests/runner-e2e/chat-scenario-catalog.test.ts b/tests/runner-e2e/chat-scenario-catalog.test.ts index eef37691e65..a2be85fd04f 100644 --- a/tests/runner-e2e/chat-scenario-catalog.test.ts +++ b/tests/runner-e2e/chat-scenario-catalog.test.ts @@ -37,7 +37,7 @@ it("validates the complete planned chat catalog against its schema and feature m } expect(ids.size).toBe(244); expect([...covered].sort()).toEqual([...mapped].sort()); - expect(mapped.size).toBe(191); + expect(mapped.size).toBe(192); const defaults = parseDocument(readFileSync(new URL("defaults.yaml", catalog), "utf8"), { uniqueKeys: true }); expect(defaults.errors).toEqual([]); expect(defaults.toJS()).toMatchObject({ schema_version: 1, status: "planned" }); diff --git a/tests/runner-e2e/chat-scenarios/runs-adapters.yaml b/tests/runner-e2e/chat-scenarios/runs-adapters.yaml index cdabc6614e1..7d7a3fd2610 100644 --- a/tests/runner-e2e/chat-scenarios/runs-adapters.yaml +++ b/tests/runner-e2e/chat-scenarios/runs-adapters.yaml @@ -7,6 +7,7 @@ scenarios: - id: runs-adapters.list-model-options capabilities: - runs-adapters/availability + - runs-adapters/model-accounts context: >- The company has admitted model discovery and configured coding accounts; some adapter inventories remain board-only. diff --git a/tests/runner-e2e/daytona-image-content.ts b/tests/runner-e2e/daytona-image-content.ts index aae91fe8ccd..8ab17a54a2e 100644 --- a/tests/runner-e2e/daytona-image-content.ts +++ b/tests/runner-e2e/daytona-image-content.ts @@ -66,6 +66,7 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [ "packages/paperclip-runner/test/fixtures/copilot-profile-v57-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json", + "packages/paperclip-runner/test/fixtures/copilot-profile-v65-identity.json", "packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs", "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs", "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs", diff --git a/tests/runner-e2e/operating-overhead.test.ts b/tests/runner-e2e/operating-overhead.test.ts index c837450da00..85bdeedf1a1 100644 --- a/tests/runner-e2e/operating-overhead.test.ts +++ b/tests/runner-e2e/operating-overhead.test.ts @@ -3,8 +3,10 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { expect, it } from "vitest"; import { EvalSpendState, openEvalCostLedger, readEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js"; -import { EVAL_OVERHEAD_COMPONENTS, scheduledOperatingOverhead, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js"; -import { readEvalOperationsReport, renderEvalOperationsReport } from "./operations-report.js"; +import { EVAL_OVERHEAD_COMPONENTS, operatingReservationBound, scheduledOperatingOverhead, validateOperatingOverhead, type EvalOperatingOverhead } from "./operating-overhead.js"; +import { evalAccountingStatus, evalReadinessStatus, readEvalOperationsReport, renderEvalOperationsReport } from "./operations-report.js"; + +import { costBoundConfigurationSha256 } from "./cost-bound-evidence.js"; const now = Date.parse("2026-10-21T00:00:00Z"); const allowance = (): EvalOperatingOverhead => ({ version: 1, reference: "fixture-only", runner: "aws-fleet", @@ -117,3 +119,51 @@ it("stops new admission after allowance expiry while preserving late invoice rec expect(() => state.validate({ type: "observe", observation: { observationId: "late-invoice", requestId: "overhead:ci_compute", source: "invoice", provider: "aws", currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "0.5", final: true } }, expiredAt)).not.toThrow(); }); + + +it("separates model readiness from unresolved overhead without releasing holds or masking model gaps", async () => { + const directory = await mkdtemp(path.join(tmpdir(), "eval-model-readiness-")); + const file = path.join(directory, "ledger.jsonl"); + const overhead = { ...currentAllowance(), accountingReadiness: "model_requests" as const }; + const ledger = await openEvalCostLedger(file, { campaignId: "model-readiness", maxUsd: "20", + deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 10, maxConcurrent: 1, maxAttemptsPerExecution: 1, overhead }); + try { + expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked"); + await ledger.admitAttempt(modelIdentity.executionId, modelIdentity.attemptId); + await ledger.reserve("model", modelIdentity, "2"); + expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked"); + await ledger.finish("model", "cancelled"); + expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked"); + await ledger.observe({ observationId: "model-final", requestId: "model", source: "provider", provider: "anthropic", + currency: "USD", route: "direct", classification: "reported", costUsd: "0.5", final: true }); + const report = await readEvalOperationsReport(file); + expect(evalAccountingStatus(report)).toBe("blocked"); + expect(evalReadinessStatus(report)).toBe("receipts_complete"); + expect(evalReadinessStatus({ ...report, accountingReadiness: undefined } as never)).toBe("blocked"); + expect(report).toMatchObject({ accountingReadiness: "model_requests", heldUsd: "4.000000000", + committedUsd: "4.500000000", modelAccounting: { requests: 1, knownUsd: "0.500000000", unresolvedRequests: 0 } }); + expect(renderEvalOperationsReport(report)).toContain("Operating costs remain separately unreconciled"); + await ledger.observe({ observationId: "overrun", requestId: "overhead:ci_compute", source: "invoice", provider: "aws", + currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "21", final: true }); + expect(evalReadinessStatus(await readEvalOperationsReport(file))).toBe("blocked"); + } finally { await ledger.close(); await rm(directory, { recursive: true, force: true }); } +}); + +it("rejects unknown readiness policies instead of silently loosening accounting", () => { + expect(() => validateOperatingOverhead({ ...allowance(), accountingReadiness: "ignore_costs" } as never, "20")) + .toThrow("Invalid accounting readiness policy"); +}); + +it("binds optional readiness policy to reviewed operating evidence without changing legacy hash recipes", () => { + for (const accountingReadiness of [undefined, "model_requests"] as const) { + const value = { ...allowance(), ...(accountingReadiness ? { accountingReadiness } : {}) }; + const component = value.components[0]!; + const calculation = { reference: value.reference, runner: value.runner, verifiedAt: value.verifiedAt, + expiresAt: value.expiresAt, component: { ...component }, ...(accountingReadiness ? { accountingReadiness } : {}) }; + component.provisionalBound = { version: 1, scope: "full_lifecycle", evidenceSha256: "a".repeat(64), + configurationSha256: costBoundConfigurationSha256(calculation) }; + expect(operatingReservationBound(value, component)?.calculationSha256).toBe(component.provisionalBound.configurationSha256); + const changed = { ...value, accountingReadiness: accountingReadiness ? undefined : "model_requests" as const }; + expect(() => operatingReservationBound(changed, component)).toThrow("Reviewed cost bound configuration changed"); + } +}); diff --git a/tests/runner-e2e/operating-overhead.ts b/tests/runner-e2e/operating-overhead.ts index 43eb5de338e..40c57283fc5 100644 --- a/tests/runner-e2e/operating-overhead.ts +++ b/tests/runner-e2e/operating-overhead.ts @@ -4,6 +4,7 @@ import { usdToUnits } from "../../packages/shared/src/money.js"; export const EVAL_OVERHEAD_COMPONENTS = ["ci_compute", "artifact_storage", "observability", "shared_subscription"] as const; export interface EvalOperatingOverhead { version: 1; + accountingReadiness?: "model_requests"; reference: string; runner: "github-hosted" | "aws-fleet"; verifiedAt: string; @@ -20,13 +21,15 @@ const reference = (value: unknown) => typeof value === "string" && /^[A-Za-z0-9] export function operatingReservationBound(allowance: EvalOperatingOverhead, component: EvalOperatingOverhead["components"][number]) { if (component.provisionalBound === undefined) return undefined; return reservationBound("operating_allowance", component.provisionalBound, { + ...(allowance.accountingReadiness ? { accountingReadiness: allowance.accountingReadiness } : {}), reference: allowance.reference, runner: allowance.runner, verifiedAt: allowance.verifiedAt, expiresAt: allowance.expiresAt, component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference }, }); } export function validateOperatingOverhead(value: EvalOperatingOverhead, totalUsd: string): bigint { - object(value, ["version", "reference", "runner", "verifiedAt", "expiresAt", "components"]); + object(value, ["version", "accountingReadiness", "reference", "runner", "verifiedAt", "expiresAt", "components"]); + if (value.accountingReadiness !== undefined && value.accountingReadiness !== "model_requests") throw new Error("Invalid accounting readiness policy"); if (value.version !== 1 || !reference(value.reference) || !["github-hosted", "aws-fleet"].includes(value.runner) || typeof value.verifiedAt !== "string" || typeof value.expiresAt !== "string" || !Number.isFinite(Date.parse(value.verifiedAt)) || !Number.isFinite(Date.parse(value.expiresAt)) diff --git a/tests/runner-e2e/operations-report.ts b/tests/runner-e2e/operations-report.ts index b751d24bed6..80c3affe067 100644 --- a/tests/runner-e2e/operations-report.ts +++ b/tests/runner-e2e/operations-report.ts @@ -7,6 +7,8 @@ interface EvalOperationsAvailableReport { version: 1; scope: "shared_gateway_ledger"; status?: never; + accountingReadiness?: "model_requests"; + modelAccounting?: { requests: number; unresolvedRequests: number; conflictingRequests: number; inflight: number; knownUsd: string; heldUsd: string }; ledgerCampaignId: string; journalHash: string; records: number; @@ -39,7 +41,7 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void } const dollars = (value: unknown): bigint => typeof value === "string" && /^(?:0|[1-9][0-9]*)\.[0-9]{9}$/.test(value) ? usdToUnits(value) : fail(); const count = (value: unknown) => { if (!Number.isSafeInteger(value) || Number(value) < 0) fail(); }; - fields(report, ["version", "scope", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]); + fields(report, ["version", "scope", "accountingReadiness", "modelAccounting", "ledgerCampaignId", "journalHash", "records", "deadlineAt", "limitUsd", "knownUsd", "heldUsd", "committedUsd", "requests", "unresolvedRequests", "boundedUnresolvedRequests", "conflictingRequests", "inflight", "breakdown", "attempts"]); if (report.version !== 1 || report.scope !== "shared_gateway_ledger" || !validEvalLedgerId(report.ledgerCampaignId) || !/^[a-f0-9]{64}$/.test(report.journalHash) || !isPublicReportTimestamp(report.deadlineAt) || !Array.isArray(report.attempts)) fail(); [report.records, report.requests, report.unresolvedRequests, report.conflictingRequests, report.inflight].forEach(count); @@ -48,6 +50,16 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void count(report.boundedUnresolvedRequests); if (report.boundedUnresolvedRequests > report.unresolvedRequests - report.conflictingRequests) fail(); } + if (report.accountingReadiness !== undefined && (report.accountingReadiness !== "model_requests" || !report.modelAccounting)) fail(); + if (report.modelAccounting !== undefined) { + const model = report.modelAccounting; + fields(model, ["requests", "unresolvedRequests", "conflictingRequests", "inflight", "knownUsd", "heldUsd"]); + [model.requests, model.unresolvedRequests, model.conflictingRequests, model.inflight].forEach(count); + if (model.requests > report.requests || model.unresolvedRequests > model.requests || model.unresolvedRequests > report.unresolvedRequests + || model.conflictingRequests > model.unresolvedRequests || model.conflictingRequests > report.conflictingRequests + || model.inflight > model.requests || model.inflight > report.inflight + || dollars(model.knownUsd) > dollars(report.knownUsd) || dollars(model.heldUsd) > dollars(report.heldUsd)) fail(); + } dollars(report.limitUsd); const known = dollars(report.knownUsd), held = dollars(report.heldUsd); if (known + held !== dollars(report.committedUsd)) fail(); @@ -86,7 +98,17 @@ async function readAvailableEvalOperationsReport(file: string): Promise request.phase !== "infrastructure"); + const modelKnown = modelRequests.reduce((sum, request) => sum + usdToUnits(request.costUsd ?? "0"), 0n); + const modelHeld = modelRequests.reduce((sum, request) => { + const cost = usdToUnits(request.costUsd ?? "0"), reserved = usdToUnits(request.reservedUsd); + return sum + ((!request.finished || !request.receiptComplete) && reserved > cost ? reserved - cost : 0n); + }, 0n); const report: EvalOperationsAvailableReport = { version: 1, scope: "shared_gateway_ledger", ledgerCampaignId: snapshot.campaignId, + ...(loaded.state.limits.overhead?.accountingReadiness ? { accountingReadiness: loaded.state.limits.overhead.accountingReadiness } : {}), + modelAccounting: { requests: modelRequests.length, unresolvedRequests: modelRequests.filter(request => !request.receiptComplete).length, + conflictingRequests: modelRequests.filter(request => request.conflict).length, inflight: modelRequests.filter(request => !request.finished).length, + knownUsd: money(modelKnown), heldUsd: money(modelHeld) }, journalHash: loaded.previous, records: loaded.sequence, deadlineAt: new Date(loaded.state.limits.deadlineAt).toISOString(), limitUsd: snapshot.limitUsd, knownUsd: snapshot.knownUsd, heldUsd: snapshot.heldUsd, committedUsd: snapshot.committedUsd, requests: snapshot.requests.length, unresolvedRequests: snapshot.unknownRequests, boundedUnresolvedRequests: snapshot.boundedUnresolvedRequests, conflictingRequests: snapshot.conflictingRequests, @@ -103,12 +125,23 @@ export function evalAccountingStatus(report: EvalOperationsReport): "unavailable return report.unresolvedRequests ? "provisional" : "receipts_complete"; } +export function evalReadinessStatus(report: EvalOperationsReport): ReturnType { + const full = evalAccountingStatus(report); + if (report.status === "unavailable" || report.accountingReadiness !== "model_requests") return full; + const model = report.modelAccounting!; + if (!model.requests || model.unresolvedRequests || model.inflight || report.conflictingRequests || report.inflight + || usdToUnits(report.committedUsd) > usdToUnits(report.limitUsd)) return "blocked"; + return "receipts_complete"; +} + export function operationsReportText(report: EvalOperationsReport): string[] { if (report.status === "unavailable") return [ "Request ledger accounting unavailable: the journal could not be read or validated. Request amounts, counts, and held reservations are unknown.", "Retained test results are preserved. Recover and validate the original journal before resuming paid work; do not reset the allowance or treat missing amounts as zero.", ]; return [ + ...(report.modelAccounting ? [`Model requests: ${report.modelAccounting.requests}; $${report.modelAccounting.knownUsd} known token cost; $${report.modelAccounting.heldUsd} additional held; ${report.modelAccounting.unresolvedRequests} unresolved.`] : []), + ...(report.accountingReadiness === "model_requests" ? [`Readiness scope: model requests; status: ${evalReadinessStatus(report)}. Operating costs remain separately unreconciled unless final receipts establish otherwise. Their reservations remain inside campaign admission; this does not establish a full-lifecycle operating-cost bound.`] : []), `Accounting status: ${evalAccountingStatus(report)}. ${report.boundedUnresolvedRequests ?? 0} pending charges have reviewed bounds; ${report.unresolvedRequests - (report.boundedUnresolvedRequests ?? 0)} do not. Provisional completion preserves holds and requires later reconciliation; it does not establish invoice completeness.`, `Shared request ledger ${report.ledgerCampaignId}: $${report.knownUsd} known subtotal; $${report.heldUsd} additional held reservations; $${report.committedUsd} committed against $${report.limitUsd}.`, `Charges: $${report.breakdown.reconciled} reconciled; $${report.breakdown.reported} reported; $${report.breakdown.estimated} estimated; $${report.breakdown.allocated} allocated.`, diff --git a/tests/runner-e2e/scheduled-finalization.test.ts b/tests/runner-e2e/scheduled-finalization.test.ts index 8b90c9fd377..5f740c4c5b2 100644 --- a/tests/runner-e2e/scheduled-finalization.test.ts +++ b/tests/runner-e2e/scheduled-finalization.test.ts @@ -1,3 +1,9 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { openEvalCostLedger } from "./cost-ledger.js"; +import { readEvalOperationsReport } from "./operations-report.js"; +import { EVAL_OVERHEAD_COMPONENTS } from "./operating-overhead.js"; import { expect, it } from "vitest"; import { buildRunnerCampaign } from "./history.js"; import { runnerMatrix } from "./catalog.js"; @@ -52,3 +58,31 @@ it("keeps incomplete accounting and trace export visible independently of a pass campaign.feedbackReview.decisions[0]!.disposition = "needs_investigation"; expect(scheduledCampaignFindings(campaign)).toContain("feedback_needs_review"); }); + +it("qualifies scheduled model receipts while retaining operating holds, but rejects missing usage and overruns", async () => { + const { campaign } = fixture(); + const directory = await mkdtemp(path.join(tmpdir(), "scheduled-model-readiness-")); + const file = path.join(directory, "ledger.jsonl"), now = Date.now(); + const ledger = await openEvalCostLedger(file, { campaignId: "scheduled-fixture", maxUsd: "20", + deadlineAt: new Date(now + 60_000).toISOString(), maxRequests: 10, maxConcurrent: 1, + overhead: { version: 1, accountingReadiness: "model_requests", reference: "synthetic-fixture", runner: "github-hosted", + verifiedAt: new Date(now - 1000).toISOString(), expiresAt: new Date(now + 86_400_000).toISOString(), + components: EVAL_OVERHEAD_COMPONENTS.map(component => ({ component, maxUsd: "1", reference: "synthetic-fixture" })) }, + maxAttemptsPerExecution: 1 }); + try { + await ledger.admitAttempt(campaign.expected[0]!, "one"); + await ledger.reserve("model", { executionId: campaign.expected[0]!, attemptId: "one", phase: "setup" }, "2"); + await ledger.finish("model", "cancelled"); + campaign.operations = await readEvalOperationsReport(file); + expect(scheduledCampaignFindings(campaign)).toContain("accounting_incomplete"); + await ledger.observe({ observationId: "model-final", requestId: "model", source: "provider", provider: "anthropic", + currency: "USD", route: "direct", classification: "reported", costUsd: "0.5", final: true }); + campaign.operations = await readEvalOperationsReport(file); + expect(scheduledCampaignFindings(campaign)).not.toContain("accounting_incomplete"); + expect(campaign.operations).toMatchObject({ heldUsd: "4.000000000", committedUsd: "4.500000000" }); + await ledger.observe({ observationId: "overrun", requestId: "overhead:ci_compute", source: "invoice", provider: "aws", + currency: "USD", route: "infrastructure", classification: "reconciled", costUsd: "21", final: true }); + campaign.operations = await readEvalOperationsReport(file); + expect(scheduledCampaignFindings(campaign)).toEqual(expect.arrayContaining(["accounting_incomplete", "accounting_over_budget"])); + } finally { await ledger.close(); await rm(directory, { recursive: true, force: true }); } +}); diff --git a/tests/runner-e2e/scheduled-finalization.ts b/tests/runner-e2e/scheduled-finalization.ts index 5d93e4fe8be..ec31a870e53 100644 --- a/tests/runner-e2e/scheduled-finalization.ts +++ b/tests/runner-e2e/scheduled-finalization.ts @@ -1,5 +1,5 @@ import { usdToUnits } from "../../packages/shared/src/money.js"; -import { evalAccountingStatus } from "./operations-report.js"; +import { evalAccountingStatus, evalReadinessStatus } from "./operations-report.js"; import { readFile, writeFile } from "node:fs/promises"; import path from "node:path"; import { pathToFileURL } from "node:url"; @@ -49,7 +49,7 @@ export function scheduledCampaignFindings(campaign: RunnerE2ECampaign): string[] const operations = campaign.operations; if (!operations || operations.status === "unavailable") findings.push("accounting_unavailable"); else { - if (evalAccountingStatus(operations) === "blocked") findings.push("accounting_incomplete"); + if (evalReadinessStatus(operations) === "blocked") findings.push("accounting_incomplete"); if (usdToUnits(operations.committedUsd) > usdToUnits(operations.limitUsd)) findings.push("accounting_over_budget"); } const feedback = summarizeCampaignFeedback(campaign.accounting?.attempts ?? campaign.results); @@ -131,6 +131,7 @@ export async function finalizeScheduledCampaign(env: NodeJS.ProcessEnv) { await writeFile(path.join(directory, "findings.json"), `${JSON.stringify({ version: 1, findings, nativeIndexVerification: indexing, gatewayIndexVerification: gatewayIndexing, invoiceReconciliation: "pending", accountingStatus: campaign.operations ? evalAccountingStatus(campaign.operations) : "unavailable", + readinessStatus: campaign.operations ? evalReadinessStatus(campaign.operations) : "unavailable", feedbackReviewMethod: "Only exact synthetic fixture bodies with all corroborating graders are automatically dismissed; all other feedback requires review" }, null, 2)}\n`, { mode: 0o600 }); for (const finding of findings) console.error(`::error title=Eval operations::${finding}`); return findings; From d108aeeaaf7c4a3050b3650444884e6c766b46d8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 15:26:19 -0500 Subject: [PATCH 06/41] fix(server): schedule pending tool-review expiry from committed changes (#17453) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Governed tools can wait for a human review before execution. > - Pending reviews already save a deadline and must reject approval after expiry. > - The server currently scans for expired reviews on every maintenance tick. > - Empty scans keep the database active without advancing work. > - This pull request schedules pending review expiry from committed changes and saved deadlines. > - Configured idle state and resolved history no longer cause this expiry scan. ## Linked Issues or Issue Description **What happened?** The combined tool review sweep repeatedly queries pending expiry even when no pending request has a deadline. **What did you expect?** Signing or resolving a request should update the expiry schedule. A saved expiry should trigger settlement. Once no expiry remains, this duty should stop querying. **Steps to reproduce** Start a claimed instance with heartbeat scheduling enabled. Leave it idle with no pending tool reviews. Observe pending-review expiry discovery in the maintenance sweep. Stacked on #17450; earlier slices are #17446 and #17443. These all reuse the delivery and transaction foundations from #15625. This PR owns pending-review expiry only. Approved-action recovery, interrupted-execution quarantine, and terminal interaction projection repair remain separate migrations. No duplicate pending-review expiry migration was found in the open PR search; credential lifetime changes in #17221 and #17274 concern the parent credential slice. ## What Changed - Replace repeated company review-deadline aggregates with an indexed single-row lookup. Add and replay a concurrent covering-index migration and generate combined snapshots throughout the stack. - Extract pending-expiry selection and settlement into its owning service. Keep provider execution and interaction publication with the gateway. - Signal deadline assignment and pending resolution inside the owning SQL transaction. Preserve outer commit/rollback and central uncertainty handling. - Register company expiry deadlines with the shared scoped coordinator. Reconstruct on startup; normal producer hints reload only their company. Preserve writes during drain and concurrent reconciliation. - Settle request expiry and invocation failure atomically. Keep interaction projection recovery in the existing separate sweep. - Remove pending-expiry scanning from the main tick. Preserve the combined manual API and startup-only behavior when scheduling is disabled. - Add a partial pending-expiry index through an idempotent concurrent migration. No queue table is added. - Document supported producers, quiet states, remaining duties, and external wake limits. ## Verification - CI run `40574661575` at `f5c935d1b89877033eb82bf890f867baeb372ef3` lost the runners for general-server shard 9/12 and E2E shard 2/8 at 20:09:03 UTC. Both job logs explicitly show a runner shutdown signal followed by cancellation; no test assertion failure was reported by those jobs. A single infrastructure retry is justified once the run completes; it has not been issued yet. Both current-head reviews are 5/5, and CI must still be green before merge. - Current master landing at `f5c935d1b89877033eb82bf890f867baeb372ef3`: 128 complete gateway expiry, startup, scoped deadline and preparation-timing cases pass with no skips. Server types and the production server build pass. Landed tracing changes from #17918 are preserved. The master migration journal prefix remains semantically identical, with only this PR's two migrations added. Fresh remote CI/review is required before merge. - Parent reconciliation at `17539e2a6cf8ac4922b3cbe06af6df361bfe5665`: 116 tests pass across the full tool gateway service, startup and scoped deadline suites against a dedicated migrated PostgreSQL database. No cases were skipped. Server typecheck passes. Fresh remote CI and review are pending. Current head: `ec28076f825e76a267f05517ff0c44a3e2283ce7`. This remains a single-duty PR stacked on #17450. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ec28076f825e76a267f05517ff0c44a3e2283ce7/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - All supported producers must share the database signal owner. Independent SQL writers and other replicas need a live notification contract; startup alone does not cover them. - A crash after authoritative expiry and before interaction projection still relies on the existing projection-repair duty. That duty, approved-action recovery, and uncertain execution recovery remain periodic in this slice. - Unknown external outcomes remain quarantined. Provider calls stay outside SQL transactions. Expiry is still checked before approving a request even if physical cleanup waits during drain. - The new index uses the existing top-level migration executor because PostgreSQL concurrent index creation cannot run in a file-wide transaction. Snapshot pruning follows the repository rule. - This establishes worker quiet and reactivation, not deployed host suspension. Existing Cloud idle-safety and external alarm coverage remain unchanged. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Latest-head CI is now green after one evidence-backed retry. Greptile is 5/5 and all review threads are resolved. --------- Co-authored-by: Paperclip --- doc/idle-tool-review-expiry.md | 155 ++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 58 ++- packages/db/src/client.ts | 2 + .../0382_narrow_the_executioner.sql | 1 + .../0383_tool_review_company_expiry.sql | 1 + ...{0376_snapshot.json => 0382_snapshot.json} | 72 +++- ...{0377_snapshot.json => 0383_snapshot.json} | 94 ++++- packages/db/src/migrations/meta/_journal.json | 14 + packages/db/src/schema/tool_access.ts | 4 + .../server-startup-feedback-export.test.ts | 4 + .../__tests__/tool-gateway-service.test.ts | 352 +++++++++++++++++- server/src/index.ts | 13 +- server/src/services/company-deletion.ts | 2 + .../services/delivery-work-notifications.ts | 1 + .../src/services/issue-thread-interactions.ts | 2 + .../tool-access/activity/review-queue.ts | 5 +- server/src/services/tool-action-review.ts | 2 + server/src/services/tool-gateway.ts | 67 ++-- .../src/services/tool-review-expiry-work.ts | 25 ++ server/src/services/tool-review-expiry.ts | 65 ++++ .../tool-review-work-notifications.ts | 13 + 21 files changed, 905 insertions(+), 47 deletions(-) create mode 100644 doc/idle-tool-review-expiry.md create mode 100644 packages/db/src/migrations/0382_narrow_the_executioner.sql create mode 100644 packages/db/src/migrations/0383_tool_review_company_expiry.sql rename packages/db/src/migrations/meta/{0376_snapshot.json => 0382_snapshot.json} (99%) rename packages/db/src/migrations/meta/{0377_snapshot.json => 0383_snapshot.json} (99%) create mode 100644 server/src/services/tool-review-expiry-work.ts create mode 100644 server/src/services/tool-review-expiry.ts create mode 100644 server/src/services/tool-review-work-notifications.ts diff --git a/doc/idle-tool-review-expiry.md b/doc/idle-tool-review-expiry.md new file mode 100644 index 00000000000..d14abcbfd4a --- /dev/null +++ b/doc/idle-tool-review-expiry.md @@ -0,0 +1,155 @@ +# Pending tool-review expiry and idle behavior + +Pending tool action requests already save their review deadline in `expiresAt`. +This worker replaces the recurring pending-expiry scan with committed changes, +startup reconstruction, and that saved deadline. It does not add a queue table +or change the one-hour review policy. + +## Ownership and producer inventory + +The tool gateway owns request signing and authorization. `toolReviewExpiryService` +owns pending-expiry selection and settlement. `registerToolReviewExpiryWork` +supplies company-scoped operations to the existing shared scoped deadline worker. Startup wires +that contract without interpreting tool or provider behavior. + +The supported writers are: + +- `tool-access-policy.ts` inserts an unsigned pending request without an expiry. + It creates no expiry obligation yet. The existing signing grace remains intact. +- The normal agent gateway signing path and the board test-call path assign the + saved deadline. Each signals in its SQL transaction before that update. +- Gateway signing failures, competing-call invalidation, invalid signatures, + and legacy pending-to-approved/expired transitions signal before removing a + pending obligation. +- `commitToolActionReview` signals inside its existing human-resolution + transaction. The issue/interaction locks, human authority checks, and delivery + enqueue remain owned there. +- Administrative interaction resolution signals in the caller-owned transaction + before resolving linked requests. Review-queue invalidation signals before + invalidating unreadable or unsigned requests. +- Company deletion signals its company before removing saved review obligations. +- The expiry worker signals before its own settlement. A compare-and-set against + pending status and the observed update time protects concurrent resolution. + +Every notification requires the company ID. Normal commits reload only that +company's saved minimum expiry; due cleanup selects only that company. Startup +reconstructs all saved company deadlines once. A company with only unsigned +null-expiry requests has no deadline. + +Only the SQL mutation is wrapped. Signing, provider requests, telemetry, and +interaction publication are not moved inside an enclosing recovery transaction. +The database signal layer handles nested savepoints, outer commit/rollback, and +uncertain transaction outcomes. No recovery object is passed through callers. +Independent SQL writers and other database owners/replicas still require a live +notification contract. Startup reconstructs saved rows but is not ongoing +cross-process discovery. + +## State and recovery + +Expiry updates the authoritative request and invocation in one transaction. If +invocation settlement fails, the request remains pending and the worker retries. +After commit, the gateway reflects the result in the linked interaction. The +existing terminal projection-repair duty remains in `sweepActionReviews` and +repairs a crash or failure between authoritative settlement and projection. +That duty is a separate migration; this worker must not claim it has completed. + +The public/manual `sweepActionReviews()` still includes expiry by default. Server +startup and the recurring legacy review sweep pass `includeExpiry: false` when +expiry has its own startup pass or worker. Approved-action recovery, interrupted +execution quarantine, and terminal projection repair retain their behavior. +An unknown external outcome is never replayed by this expiry worker. + +The next deadline comes from pending rows only. Startup uses the existing +partial expiry index to reconstruct company deadlines. A company notification +uses an ordered, single-row lookup backed by the partial `(company_id, +expires_at)` index, rather than aggregating that company's pending backlog. +Null-expiry unsigned requests and retained resolved history do not keep this +worker active. The concurrent index migration uses the existing top-level +executor and is safe to apply again after interruption. + +The shared coordinator coalesces changes and preserves a write arriving during +reconciliation. Each due pass expires at most 100 rows for one company; companies +rotate between passes until the backlog drains. Failed scopes retry independently +with exponential backoff from 30 seconds to five minutes. Other companies retain +their own deadlines. New writes do not erase a failed sweep's backoff. +Once reconciliation completes with no deadline, it has no recurring SQL. Drain +retains first writes without passive database scans; resume processes overdue +reviews. Startup reconstructs persisted deadlines. Shutdown stops subscription +and timer ownership and joins active work. + +Approval still rejects an expired request synchronously before physical cleanup. +With heartbeat scheduling disabled, startup-only expiry remains as before. + +## Verification scope + +Real PostgreSQL tests record SQL to prove signing, human decisions, and due expiry +remain company-scoped. They also verify five-second and earlier replacement +deadlines, company deletion, and bounded multi-pass draining. Existing cases cover +normal gateway signing and human decisions, +configured idle and terminal history, outer commit/rollback, startup/restart, +drain/resume, concurrent producer/read, unsigned rows, multi-page expiry, and +injected invocation-write failure with atomic rollback and retry. Existing +service tests cover approval/execution, interrupted outcomes, and projection +repair through the compatible manual entry point. Startup tests cover wiring. +The external test database override requires a disposable database ending in +`_test`; the suite deletes fixtures and stops the worker before database cleanup. + +This is one worker migration, not proof that the instance can sleep. Other tool +review duties remain periodic. Existing Cloud sleep-safety gates and external +alarm coverage are unchanged. An in-process deadline cannot wake a stopped host. + +## Scoped lookup review repair (2026-10-27) + +The original scoped aggregate examined 12,004 rows to return one deadline in a +real PostgreSQL regression. The repaired lookup returns the earliest non-null +pending expiry for the requested company with `ORDER BY expires_at LIMIT 1`. +The grouped startup inventory remains unchanged. Existing company signals, +authority checks, expiry settlement, paging, backoff, and sleep gates remain +owned by their existing services. + +The additive company-expiry index is migration `0383_tool_review_company_expiry`. +It builds the index concurrently and idempotently through the existing top-level +migration executor. Initially reserving number 0402 passed per-branch checks but +caused CI to reject descendants whose migrations preceded their target branch. +A local 0380 correction fixed that ordering, but master had meanwhile added its +own 0378 and 0379 migrations. The integrated stack preserves master's history, +renumbers its unmerged migrations after 0379 in PR dependency order, and preserves +every migration's SQL bytes. Exact filename references and journal tags/indices +follow the new names. Stack journal timestamps remain ordered after master's +latest entry. Each generated snapshot includes both master and that PR's schema. + +Each PR must pass the exact target-branch migration-order gate, in addition to +the existing numbering, safety, five-snapshot retention, and schema-drift checks. +Local branch checks alone did not establish the stack's append-only ordering. +Fresh application, upgrades from the previous numbered task databases, and +replay also require verification. These unmerged stack migrations have not +changed the SQL of migrations already on master. + +Two failures shaped the repair. The first migration attempt omitted its explicit +top-level registration and PostgreSQL rejected the concurrent build inside a +transaction; adding the registration fixed application and replay. An initial +three-column index also lost to the global expiry index in the planner, which +filtered 12,001 foreign-company rows before returning a result. The final index +omits the unused request ID. With 12,000 additional pending rows in each of two +companies, including earlier foreign deadlines, all three measured reloads +examine one row. The assertion counts filtered rows as well as returned rows. +This is evidence for the recorded fixture and query plan, not a production +latency guarantee or a claim that PostgreSQL can never choose another plan. + +The complete gateway service, gateway acceptance, credential expiry, startup, +custom-image deadline, and schema snapshot suites pass: 223 tests across six +files. The run also exposed a credential test-clock overshoot; its correction +belongs to the earlier gateway PR and waits for real SQL before advancing to +an actually armed follow-up. Commit/rollback and exact expiry assertions remain. +Server typechecking passes. Combined-stack migration upgrades, full build, and +fresh remote checks remain required. Full local test:run is still unverified +because of embedded PostgreSQL startup/capacity limits; real gateway suites used +isolated, migrated test databases in the task-owned PostgreSQL instance. + + +A later review found an unbound `query` name in the SQL-capture assertion. +The due-row predicate short-circuited before that expression in the passing +fixture, and server typechecking excludes test sources. The assertion now +independently requires both the due expiry scan and the non-null deadline lookup +through each captured `read.query`. The earlier passing run did not detect this +name error; acceptance must include the service suite with both assertions. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 009401a1d93..5bdacbcd130 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -18,7 +18,7 @@ Do not mark an item done from a timer removal or a green check alone. | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | -| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged after CI and passing reviews; gateway credential expiry in progress; tool reviews pending | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry implemented on slice 04; other review duties pending | | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | @@ -61,3 +61,59 @@ connector transport architecture, probes, telemetry/exporter qualification, and end-to-end provider suspension are outside this requested recurring-server stack; none are claimed solved by these PRs. Required active-operation lease renewals and request timeouts remain with their owners. + +## Next owner audit: tool reviews + +Inspected after gateway credential slice `75332b9641`. The existing +`sweepActionReviews` combines four reasons that need separate migration slices: + +1. Pending review expiry: `status = pending` and saved `expiresAt`. +2. Approved action recovery: approved rows that still owe execution/settlement. +3. Interrupted execution: executing rows with the existing ten-minute stale + threshold; unknown provider outcomes must remain quarantined, not replayed. +4. Interaction projection repair: terminal requests whose interaction result + does not reflect the authoritative request/invocation state. + +The next branch is `codex/sleep-04-tool-review-expiry`. Implementation now separates pending expiry with commit-aware signals and its +existing deadline. Preserve the other three duties while migrating the first, then +retire them individually. Keep the public/manual recovery entry point compatible +or update all callers and tests explicitly; tests currently call the combined +method to cause expiry. + +Producer locations found by the audit: + +- `tool-access-policy.ts`: creates pending requests with null signature/expiry. +- `tool-gateway.ts`: signs pending requests and assigns their one-hour expiry; + handles competing-call invalidation, approval/execution transitions, provider + outcomes, and sweep settlement. Audit all writes, not just initial insertion. +- `tool-action-review.ts`: owns a transaction for human resolution. +- `tool-access/activity/review-queue.ts`: invalidates unsigned/unreadable requests. +- `issue-thread-interactions.ts`: resolves linked pending/approved requests when + their governing card is resolved administratively. + +The unsigned-row signing grace is distinct from the saved review expiry. Do not +invent a new expiry policy while removing the scan. A pending row can be observed +between insert and signing; expiry workers must not cancel an in-flight producer. +Keep provider execution outside database transactions. Signals must be registered +before writes in existing transaction ownership, without wrapping provider calls. + +Latest evidence: #17443 has all checks passed and both review threads resolved. +#17446 at `d41590ac0e` has all checks passed, Greptile 5/5, and no review threads. +#17450 at `c5c53fc43f` fixes the deadline query cost with bounded indexed lookups +and an idempotent concurrent session index. Both findings are resolved; fresh +Greptile is 5/5 and CI is running. Local migration/replay, 47 expiry/startup tests, +workspace typecheck/build passed. A synthetic 40,000-row PostgreSQL plan reads +one row from each expiry index. Recheck external state before readiness claims. + +Slice 04 now has 11 real-PostgreSQL worker lifecycle tests plus 59 existing +service tests and 39 startup tests passing (109 total). The initial fixture +incorrectly treated an argument-summary object as a string. After correction, +the failure test exposed its invalid expectation of no idle hold during an +unresolved write; it now checks rollback while retaining the hold. A combined +run also exposed fixture cleanup ordering: this repository uses ordered hooks, +so the worker must stop before the outer hook deletes rows. The corrected full +selection passes. The current implementation adds a partial pending-expiry +index, applies/replays its migration, and preserves approval-time rejection and +separate approved/executing/projection recovery. Workspace typecheck/build and all 92 existing gateway acceptance tests also +passed. The first acceptance command matched no files; the corrected +`tool-gateway.test.ts` selection passed. Fresh PR CI/review remains required. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 77fb0c62fa1..df20aa973ab 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -502,6 +502,8 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0345_luxuriant_carmella_unuscione.sql", "0369_late_logan.sql", "0381_mean_scream.sql", + "0382_narrow_the_executioner.sql", + "0383_tool_review_company_expiry.sql", ]); async function applyPendingMigrationsManually( diff --git a/packages/db/src/migrations/0382_narrow_the_executioner.sql b/packages/db/src/migrations/0382_narrow_the_executioner.sql new file mode 100644 index 00000000000..af2e60c65f7 --- /dev/null +++ b/packages/db/src/migrations/0382_narrow_the_executioner.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "tool_action_requests_pending_expiry_idx" ON "tool_action_requests" USING btree ("expires_at","id") WHERE "tool_action_requests"."status" = 'pending'; diff --git a/packages/db/src/migrations/0383_tool_review_company_expiry.sql b/packages/db/src/migrations/0383_tool_review_company_expiry.sql new file mode 100644 index 00000000000..75a364ecdf1 --- /dev/null +++ b/packages/db/src/migrations/0383_tool_review_company_expiry.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "tool_action_requests_company_pending_expiry_idx" ON "tool_action_requests" USING btree ("company_id","expires_at") WHERE "tool_action_requests"."status" = 'pending'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0376_snapshot.json b/packages/db/src/migrations/meta/0382_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0376_snapshot.json rename to packages/db/src/migrations/meta/0382_snapshot.json index 1015edaf69f..6c27dd9aa22 100644 --- a/packages/db/src/migrations/meta/0376_snapshot.json +++ b/packages/db/src/migrations/meta/0382_snapshot.json @@ -1,6 +1,6 @@ { - "id": "7832df38-fa1b-4687-bd06-eccbc29bd28c", - "prevId": "3e17203b-ec95-41e9-8363-685982450199", + "id": "ecec138e-5312-4ad6-b051-bd4ee9ed4e34", + "prevId": "2e93deb7-f24c-42d1-8c8e-30b21cff90bc", "version": "7", "dialect": "postgresql", "tables": { @@ -47782,6 +47782,18 @@ "primaryKey": false, "notNull": false }, + "git_authorization_grant_id": { + "name": "git_authorization_grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "git_authorized_by_user_id": { + "name": "git_authorized_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "pause_reason": { "name": "pause_reason", "type": "text", @@ -47920,6 +47932,19 @@ ], "onDelete": "no action", "onUpdate": "no action" + }, + "projects_git_authorization_grant_id_connection_grants_id_fk": { + "name": "projects_git_authorization_grant_id_connection_grants_id_fk", + "tableFrom": "projects", + "tableTo": "connection_grants", + "columnsFrom": [ + "git_authorization_grant_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" } }, "compositePrimaryKeys": {}, @@ -52787,6 +52812,28 @@ "method": "btree", "with": {} }, + "tool_action_requests_pending_expiry_idx": { + "name": "tool_action_requests_pending_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_invocation_idx": { "name": "tool_action_requests_invocation_idx", "columns": [ @@ -55197,6 +55244,21 @@ "method": "btree", "with": {} }, + "tool_gateway_sessions_expiry_idx": { + "name": "tool_gateway_sessions_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_gateway_sessions_run_idx": { "name": "tool_gateway_sessions_run_idx", "columns": [ @@ -55942,6 +56004,12 @@ "notNull": true, "default": "''" }, + "responsible_user_id": { + "name": "responsible_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "owner_note": { "name": "owner_note", "type": "text", diff --git a/packages/db/src/migrations/meta/0377_snapshot.json b/packages/db/src/migrations/meta/0383_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0377_snapshot.json rename to packages/db/src/migrations/meta/0383_snapshot.json index 6e89edc3a9b..3336a6bb20e 100644 --- a/packages/db/src/migrations/meta/0377_snapshot.json +++ b/packages/db/src/migrations/meta/0383_snapshot.json @@ -1,6 +1,6 @@ { - "id": "a0fcd9b8-f125-4985-8795-f567466b9aef", - "prevId": "7832df38-fa1b-4687-bd06-eccbc29bd28c", + "id": "410457fc-266b-4bd0-8bc0-2b14b0634d07", + "prevId": "ecec138e-5312-4ad6-b051-bd4ee9ed4e34", "version": "7", "dialect": "postgresql", "tables": { @@ -47782,6 +47782,18 @@ "primaryKey": false, "notNull": false }, + "git_authorization_grant_id": { + "name": "git_authorization_grant_id", + "type": "uuid", + "primaryKey": false, + "notNull": false + }, + "git_authorized_by_user_id": { + "name": "git_authorized_by_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "pause_reason": { "name": "pause_reason", "type": "text", @@ -47920,6 +47932,19 @@ ], "onDelete": "no action", "onUpdate": "no action" + }, + "projects_git_authorization_grant_id_connection_grants_id_fk": { + "name": "projects_git_authorization_grant_id_connection_grants_id_fk", + "tableFrom": "projects", + "tableTo": "connection_grants", + "columnsFrom": [ + "git_authorization_grant_id" + ], + "columnsTo": [ + "id" + ], + "onDelete": "set null", + "onUpdate": "no action" } }, "compositePrimaryKeys": {}, @@ -52787,6 +52812,50 @@ "method": "btree", "with": {} }, + "tool_action_requests_pending_expiry_idx": { + "name": "tool_action_requests_pending_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_company_pending_expiry_idx": { + "name": "tool_action_requests_company_pending_expiry_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_invocation_idx": { "name": "tool_action_requests_invocation_idx", "columns": [ @@ -55197,6 +55266,21 @@ "method": "btree", "with": {} }, + "tool_gateway_sessions_expiry_idx": { + "name": "tool_gateway_sessions_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_gateway_sessions_run_idx": { "name": "tool_gateway_sessions_run_idx", "columns": [ @@ -55942,6 +56026,12 @@ "notNull": true, "default": "''" }, + "responsible_user_id": { + "name": "responsible_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "owner_note": { "name": "owner_note", "type": "text", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 176a963f7a4..02023247485 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2661,6 +2661,20 @@ "when": 1793060791725, "tag": "0381_mean_scream", "breakpoints": true + }, + { + "idx": 382, + "version": "7", + "when": 1793060791726, + "tag": "0382_narrow_the_executioner", + "breakpoints": true + }, + { + "idx": 383, + "version": "7", + "when": 1793060791727, + "tag": "0383_tool_review_company_expiry", + "breakpoints": true } ] } diff --git a/packages/db/src/schema/tool_access.ts b/packages/db/src/schema/tool_access.ts index 825eb91589f..e970c445b6c 100644 --- a/packages/db/src/schema/tool_access.ts +++ b/packages/db/src/schema/tool_access.ts @@ -832,6 +832,10 @@ export const toolActionRequests = pgTable( }, (table) => [ index("tool_action_requests_company_status_idx").on(table.companyId, table.status), + index("tool_action_requests_pending_expiry_idx").on(table.expiresAt, table.id) + .where(sql`${table.status} = 'pending'`), + index("tool_action_requests_company_pending_expiry_idx").on(table.companyId, table.expiresAt) + .where(sql`${table.status} = 'pending'`), index("tool_action_requests_invocation_idx").on(table.invocationId), index("tool_action_requests_issue_idx").on(table.companyId, table.issueId), ], diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 9788b8599ff..2ede1c6c85f 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -70,6 +70,10 @@ const { }, toolGateway: { sweepActionReviews: vi.fn(async () => ({ scanned: 0 })), + sweepExpiredActionReviews: vi.fn(async () => ({ scanned: 0 })), + nextActionReviewExpiryAt: vi.fn(async () => null), + actionReviewExpiryDeadlines: vi.fn(async () => []), + sweepExpiredActionReviewsForCompany: vi.fn(async () => ({ scanned: 0 })), cleanupExpiredSessions: vi.fn(async () => ({ deletedCount: 0 })), nextCredentialExpiryAt: vi.fn(async () => null), }, diff --git a/server/src/__tests__/tool-gateway-service.test.ts b/server/src/__tests__/tool-gateway-service.test.ts index cc804845ff6..3e9829670c6 100644 --- a/server/src/__tests__/tool-gateway-service.test.ts +++ b/server/src/__tests__/tool-gateway-service.test.ts @@ -1,11 +1,12 @@ +import { deleteCompany } from "../services/company-deletion.js"; import { subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; import { randomUUID } from "node:crypto"; import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { and, eq } from "drizzle-orm"; -import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { and, eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { activityLog, agentWakeupRequests, @@ -54,7 +55,13 @@ import { startEmbeddedPostgresTestDatabase, } from "./helpers/embedded-postgres.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { registerToolReviewExpiryWork } from "../services/tool-review-expiry-work.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TOOL_REVIEW_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; const testToolActionSigningSecret = "test-tool-action-signing-secret"; type ToolGatewayServiceOptions = NonNullable[1]>; @@ -203,13 +210,22 @@ function runSessionInput(company: IdRef, agent: IdRef, run: IdRef): Parameters { let db!: ReturnType; let tempDb: Awaited> | null = null; + let stopExpiryWorker: (() => Promise) | undefined; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-gateway-"); - db = createDb(tempDb.connectionString); + if (!externalDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-gateway-"); + db = createDb(externalDatabaseUrl ?? tempDb!.connectionString, { + maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0, + }); + await clearFixtures(); }, 20_000); afterEach(async () => { + await stopExpiryWorker?.(); stopExpiryWorker = undefined; + await clearFixtures(); + }); + + async function clearFixtures() { vi.unstubAllEnvs(); await db.delete(activityLog); await db.delete(agentWakeupRequests); @@ -232,12 +248,336 @@ describeEmbeddedPostgres("tool gateway service", () => { await db.delete(agents); await db.delete(companyMemberships); await db.delete(companies); - }); + } afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); + describe("pending review expiry worker", () => { + let coordinator: ReturnType | undefined; + let gateway: ReturnType; + let fixture: Awaited>; + let draining: boolean; + const errors: unknown[] = []; + + beforeEach(async () => { + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + fixture = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(fixture.company)); + gateway = createTestToolGatewayService(db); + draining = false; errors.length = 0; + stopExpiryWorker = stopWorker; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + async function stopWorker() { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.execute(sql`drop trigger if exists sleep_review_fail on tool_invocations`); + await db.execute(sql`drop function if exists sleep_review_fail()`); + expect(errors).toEqual([]); + } + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerToolReviewExpiryWork(coordinator, gateway, () => !draining).ready; + } + async function settle(ms = 60_000) { + await vi.advanceTimersByTimeAsync(ms); + if (!draining) await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0), { timeout: 5000 }); + } + async function quiet() { + await settle(); expect(coordinator!.nextWakeAt()).toBeNull(); + const selects = vi.spyOn(db, "select"), executes = vi.spyOn(db, "execute"); + await settle(60 * 60_000); + expect(selects).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + selects.mockRestore(); executes.mockRestore(); + } + async function request(service = gateway, source = db, context = fixture) { + const before = new Set((await source.select({ id: toolActionRequests.id }).from(toolActionRequests) + .where(eq(toolActionRequests.companyId, context.company.id))).map(row => row.id)); + const session = await service.createSession(runSessionInput(context.company, context.agent, context.run)); + const noteId = randomUUID(); + await expect(service.executeTool({ sessionToken: session.token, tool: "mcp-remote-fixture:update_note", + parameters: { noteId, body: "Awaiting a human" } })).rejects.toMatchObject({ reasonCode: "approval_required" }); + const rows = (await source.select().from(toolActionRequests).where(eq(toolActionRequests.companyId, context.company.id))) + .filter(row => !before.has(row.id)); + expect(rows).toHaveLength(1); + return rows[0]; + } + async function expectExpired(id: string, invocationId: string) { + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, id)))[0].status).toBe("expired"); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, invocationId)))[0]) + .toMatchObject({ status: "failed", errorCode: "action_expired" }); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); + } + + async function requestWithExpiry(expiresAt: Date, context = fixture) { + return db.transaction(async tx => { + const source = tx as unknown as typeof db; + const pending = await request(createTestToolGatewayService(source), source, context); + const [saved] = await tx.update(toolActionRequests).set({ expiresAt }) + .where(eq(toolActionRequests.id, pending.id)).returning(); + return saved; + }); + } + + it("keeps signing and human resolution reads scoped to their company", async () => { + const other = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company)); + const foreign = await request(gateway, db, other); + await start(); await settle(1); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query, params) => { + if (/^select/i.test(query) && query.includes('"tool_action_requests"') && + (query.includes('min(') || query.includes('"expires_at" <=') || query.includes('"expires_at" is not null'))) reads.push({ query, params: [...params] }); + }; + try { + for (let index = 0; index < 3; index++) { + const pending = await request(); await settle(1000); + await commitToolActionReview(db, { companyId: fixture.company.id, actionRequestId: pending.id, + decision: "rejected", actor: { userId: "reviewer" } }); + await settle(1000); + } + } finally { db.$client.options.debug = debug; } + expect(reads.length).toBeGreaterThan(0); + for (const read of reads) { + expect(read.query).toContain('"tool_action_requests"."company_id" ='); + expect(read.params).toContain(fixture.company.id); + expect(read.params).not.toContain(other.company.id); + } + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, foreign.id)))[0].status).toBe("pending"); + expect(coordinator!.nextWakeAt()).toBe(foreign.expiresAt!.getTime()); + }); + + it("reads one indexed deadline from a large company backlog on repeated reloads", async () => { + const pending = await request(); + const foreignContext = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(foreignContext.company)); + const foreign = await requestWithExpiry(new Date(Date.now() + 1000), foreignContext); + await db.execute(sql`insert into tool_action_requests + (company_id, invocation_id, canonical_arguments_hash, canonical_arguments_summary, expires_at) + select ${fixture.company.id}, ${pending.invocationId}, 'backlog', '{}'::jsonb, + ${pending.expiresAt!.toISOString()}::timestamptz + n * interval '1 second' + from generate_series(1, 12000) n`); + await db.execute(sql`insert into tool_action_requests + (company_id, invocation_id, canonical_arguments_hash, canonical_arguments_summary, expires_at) + select ${foreignContext.company.id}, ${foreign.invocationId}, 'foreign backlog', '{}'::jsonb, + ${foreign.expiresAt!.toISOString()}::timestamptz - n * interval '1 second' + from generate_series(1, 12000) n`); + await db.insert(toolActionRequests).values([ + { ...pending, id: randomUUID(), interactionId: null, expiresAt: null }, + { ...pending, id: randomUUID(), interactionId: null, status: "rejected", expiresAt: new Date(0) }, + ]); + await db.execute(sql`analyze tool_action_requests`); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query, params) => { + if (/^select/i.test(query) && query.includes('"tool_action_requests"')) reads.push({ query, params: [...params] }); + }; + try { + for (let i = 0; i < 3; i++) { + expect(await gateway.actionReviewExpiryDeadlines(fixture.company.id)).toEqual([ + { companyId: fixture.company.id, dueAt: pending.expiresAt!.getTime() }, + ]); + } + } finally { db.$client.options.debug = debug; } + expect(reads).toHaveLength(3); + type Plan = { "Node Type": string; "Actual Rows": number; "Actual Loops": number; + "Rows Removed by Filter"?: number; "Rows Removed by Index Recheck"?: number; Plans?: Plan[] }; + const scannedRows = (plan: Plan): number => + (/Scan/.test(plan["Node Type"]) ? (plan["Actual Rows"] + (plan["Rows Removed by Filter"] ?? 0) + + (plan["Rows Removed by Index Recheck"] ?? 0)) * plan["Actual Loops"] : 0) + + (plan.Plans ?? []).reduce((count, child) => count + scannedRows(child), 0); + for (const read of reads) { + const [result] = await db.$client.unsafe( + `explain (analyze, buffers, format json) ${read.query}`, + read.params as Parameters[1], + ); + const plan = result["QUERY PLAN"][0].Plan as Plan; + expect(scannedRows(plan), JSON.stringify(plan)).toBeLessThanOrEqual(1); + } + expect(await gateway.actionReviewExpiryDeadlines(randomUUID())).toEqual([]); + expect(await gateway.actionReviewExpiryDeadlines()).toEqual(expect.arrayContaining([ + { companyId: fixture.company.id, dueAt: pending.expiresAt!.getTime() }, + { companyId: foreignContext.company.id, dueAt: foreign.expiresAt!.getTime() - 12_000_000 }, + ])); + }); + + it("preserves a five-second deadline and an earlier committed replacement", async () => { + await start(); await quiet(); + const later = await requestWithExpiry(new Date(Date.now() + 120_000)); await settle(1); + expect(coordinator!.nextWakeAt()).toBe(later.expiresAt!.getTime()); + const earlier = await requestWithExpiry(new Date(Date.now() + 5000)); await settle(1); + expect(coordinator!.nextWakeAt()).toBe(earlier.expiresAt!.getTime()); + await settle(6000); await expectExpired(earlier.id, earlier.invocationId); + expect(coordinator!.nextWakeAt()).toBe(later.expiresAt!.getTime()); + await settle(120_000); await expectExpired(later.id, later.invocationId); await quiet(); + }); + + it("limits due expiry reads to the company whose deadline arrived", async () => { + const other = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company)); + const foreign = await request(gateway, db, other); + const due = await requestWithExpiry(new Date(Date.now() + 5000)); + await start(); await settle(1); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query, params) => { + if (/^select/i.test(query) && query.includes('"tool_action_requests"') && + (query.includes('min(') || query.includes('"expires_at" <=') || query.includes('"expires_at" is not null'))) reads.push({ query, params: [...params] }); + }; + try { await settle(6000); } finally { db.$client.options.debug = debug; } + expect(reads.some(read => read.query.includes('"expires_at" <='))).toBe(true); + expect(reads.some(read => read.query.includes('"expires_at" is not null'))).toBe(true); + for (const read of reads) { + expect(read.query).toContain('"tool_action_requests"."company_id" ='); + expect(read.params).toContain(fixture.company.id); + expect(read.params).not.toContain(other.company.id); + } + await expectExpired(due.id, due.invocationId); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, foreign.id)))[0].status).toBe("pending"); + expect(coordinator!.nextWakeAt()).toBe(foreign.expiresAt!.getTime()); + }); + + it("removes cached review deadlines after company deletion commits", async () => { + await request(); await start(); await settle(1); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, fixture.run.id)); + await db.update(agents).set({ status: "terminated", lifecycleState: "terminated" }) + .where(eq(agents.id, fixture.agent.id)); + expect(await deleteCompany(db, fixture.company.id)).toMatchObject({ id: fixture.company.id }); + await quiet(); + expect(await gateway.actionReviewExpiryDeadlines()).toEqual([]); + }); + + it("is quiet when configured idle, wakes on signing, expires the review and card, then stays quiet with history", async () => { + await start(); await quiet(); + const pending = await request(); + await settle(); expect(coordinator!.nextWakeAt()).toBe(pending.expiresAt!.getTime()); + const reads = vi.spyOn(db, "select"); + await settle(5 * 60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await settle(60 * 60_000); + await expectExpired(pending.id, pending.invocationId); + expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, pending.interactionId!)))[0].status).toBe("expired"); + await quiet(); + }); + + it.each(["approved", "rejected"] as const)("disarms when a human resolves the last review as %s", async decision => { + await start(); const pending = await request(); await settle(); + await commitToolActionReview(db, { companyId: fixture.company.id, actionRequestId: pending.id, + decision, actor: { userId: "reviewer" } }); + await quiet(); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe(decision); + }); + + it.each([false, true])("observes outer transaction settlement (rollback %s)", async rollback => { + await start(); await quiet(); + let release!: () => void, staged!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { staged = resolve; }); + const aborted = new Error("rollback fixture"); + const writing = db.transaction(async tx => { + const scoped = tx as unknown as typeof db; + await request(createTestToolGatewayService(scoped), scoped); + staged(); await gate; + if (rollback) throw aborted; + }).catch(error => { if (error !== aborted) throw error; }); + await Promise.race([ready, writing]); + try { + await vi.advanceTimersByTimeAsync(60_000); + expect(await db.select().from(toolActionRequests)).toEqual([]); + } finally { release(); } + await writing; await settle(); + if (rollback) await quiet(); + else { + expect(coordinator!.nextWakeAt()).toBe(await gateway.nextActionReviewExpiryAt()); + await settle(60 * 60_000); await quiet(); + } + }); + + it("reconstructs saved expiry on restart", async () => { + const pending = await request(); + await start(); await settle(); + expect(coordinator!.nextWakeAt()).toBe(pending.expiresAt!.getTime()); + await coordinator!.stop(); coordinator = undefined; + await settle(60 * 60_000); + await start(); await settle(); + await expectExpired(pending.id, pending.invocationId); await quiet(); + }); + + it("retains the first write through drain and enforces approval expiry before cleanup resumes", async () => { + await start(); await quiet(); draining = true; + const pending = await request(); await settle(); + const reads = vi.spyOn(db, "select"); + await settle(2 * 60 * 60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await expect(commitToolActionReview(db, { companyId: fixture.company.id, actionRequestId: pending.id, + decision: "approved", actor: { userId: "reviewer" } })).rejects.toThrow("This review has expired"); + draining = false; await settle(); + await expectExpired(pending.id, pending.invocationId); await quiet(); + }); + + it("preserves a signing write arriving during an empty deadline read", async () => { + let release!: () => void, reading!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { reading = resolve; }); + const actual = gateway.actionReviewExpiryDeadlines.bind(gateway); + vi.spyOn(gateway, "actionReviewExpiryDeadlines").mockImplementationOnce(async () => { + const deadline = await actual(); reading(); await gate; return deadline; + }); + const starting = start(); await ready; + let pending: Awaited>; + try { pending = await request(); } finally { release(); } + await starting; await settle(); + expect(coordinator!.nextWakeAt()).toBe(pending!.expiresAt!.getTime()); + }); + + it("leaves unsigned requests without a saved expiry alone", async () => { + const pending = await request(); + await db.update(toolActionRequests).set({ expiresAt: null, signedArguments: null }) + .where(eq(toolActionRequests.id, pending.id)); + await start(); await quiet(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("pending"); + }); + + it("expires all overdue pages without replaying approved or interrupted actions", async () => { + const pending = await request(); + const past = new Date(Date.now() - 1000); + await db.update(toolActionRequests).set({ expiresAt: past }).where(eq(toolActionRequests.id, pending.id)); + await db.insert(toolActionRequests).values(Array.from({ length: 101 }, () => ({ + ...pending, id: randomUUID(), interactionId: null, expiresAt: past, + }))); + const passes = vi.spyOn(gateway, "sweepExpiredActionReviewsForCompany"); + await start(); await quiet(); + const results = await Promise.all(passes.mock.results.map(result => result.value)); + expect(results.length).toBeGreaterThan(1); + expect(results.every(result => result.scanned <= 100)).toBe(true); + const rows = await db.select().from(toolActionRequests); + expect(rows).toHaveLength(102); + expect(rows.every(row => row.status === "expired")).toBe(true); + }); + + it("rolls expiry back when invocation settlement fails, then retries the complete transition", async () => { + const pending = await request(); await start(); await settle(); + await db.execute(sql`create function sleep_review_fail() returns trigger language plpgsql as $$ + begin raise exception 'expiry settlement fixture failure'; end $$`); + await db.execute(sql`create trigger sleep_review_fail before update on tool_invocations + for each row when (new.error_code = 'action_expired') execute function sleep_review_fail()`); + await vi.advanceTimersByTimeAsync(60 * 60_000); + await vi.waitFor(() => expect(errors.length).toBeGreaterThan(0)); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("pending"); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0].status).toBe("awaiting_approval"); + await db.execute(sql`drop trigger sleep_review_fail on tool_invocations`); + expect(errors.every(error => error instanceof AggregateError && error.errors.every( + failure => (failure as { cause?: Error }).cause?.message === "expiry settlement fixture failure"))).toBe(true); + errors.length = 0; + await settle(); await expectExpired(pending.id, pending.invocationId); await quiet(); + }); + }); + it("gates write tools with an action request and executes only stored reviewed arguments once", async () => { const { company, agent, run } = await createRunFixture(db); await db.insert(toolPolicies).values(reviewNotePolicyRow(company)); diff --git a/server/src/index.ts b/server/src/index.ts index 4116e46bd93..ea24010565f 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerToolReviewExpiryWork } from "./services/tool-review-expiry-work.js"; import { registerGatewayCredentialExpiryWork } from "./services/gateway-credential-expiry-work.js"; import { registerSecretProposalExpiryWork } from "./services/secret-proposal-expiry-work.js"; import { registerCustomImageCleanupWork } from "./services/custom-image-cleanup-work.js"; @@ -1464,7 +1465,15 @@ async function startServerWithDatabaseTeardown( })); }; - await timeStartupRecovery("tool_reviews", () => app.locals.toolGateway.sweepActionReviews()).catch((err: unknown) => logger.error({ err }, "startup tool review recovery failed")); + await timeStartupRecovery("tool_reviews", async () => { + if (config.heartbeatSchedulerEnabled) { + await registerToolReviewExpiryWork(deliveryWork, app.locals.toolGateway, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + } else { + await app.locals.toolGateway.sweepExpiredActionReviews(); + } + return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false }); + }).catch((err: unknown) => logger.error({ err }, "startup tool review recovery failed")); await timeStartupRecovery("gateway_sessions", () => config.heartbeatSchedulerEnabled ? registerGatewayCredentialExpiryWork(deliveryWork, app.locals.toolGateway, () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready @@ -1894,7 +1903,7 @@ async function startServerWithDatabaseTeardown( logger.error({ err }, "periodic tool connection health sweep failed"); })); - trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews().catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); + trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index 7e4a41a95d7..ab549ecbfb8 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -1,3 +1,4 @@ +import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { agentObservabilityService } from "./agent-observability.js"; import { withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-local/server"; @@ -62,6 +63,7 @@ export async function deleteCompany(db: Db, id: string) { if (!existing) return null; await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); await notifyDeliveryWork(tx, DELIVERY_QUEUES.secretProposalExpiry, id); + await notifyToolReviewChange(tx, id); await agentObservabilityService(tx as unknown as Db).removeForCompanyDeletion(id); // Finance can reference costs and both can reference runs. Incidents // reference policies and approvals; delete these dependents first. diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 191b8aef36b..210c830f91a 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + toolReviewExpiry: "tool-review-expiry", gatewayCredentialExpiry: "gateway-credential-expiry", secretProposalExpiry: "secret-proposal-expiry", customImageExpiry: "custom-image-expiry", diff --git a/server/src/services/issue-thread-interactions.ts b/server/src/services/issue-thread-interactions.ts index 74e8d8216db..48ecf1d9dbe 100644 --- a/server/src/services/issue-thread-interactions.ts +++ b/server/src/services/issue-thread-interactions.ts @@ -1,3 +1,4 @@ +import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; import { normalizeEscapedLineBreaks } from "@paperclipai/shared/validators/text"; import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import { historicalQuestionCondition } from "./issue-question-context.js"; @@ -1083,6 +1084,7 @@ async function resolveLinkedToolActionRequests( }, ) { if (interaction.kind !== "request_confirmation") return; + await notifyToolReviewChange(db, interaction.companyId); await db .update(toolActionRequests) .set({ diff --git a/server/src/services/tool-access/activity/review-queue.ts b/server/src/services/tool-access/activity/review-queue.ts index 2eb5d1e4c9e..37df9bb6c94 100644 --- a/server/src/services/tool-access/activity/review-queue.ts +++ b/server/src/services/tool-access/activity/review-queue.ts @@ -1,3 +1,4 @@ +import { writeToolReviewChange } from "../../tool-review-work-notifications.js"; import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { toolApplications, toolActionRequests, toolCatalogEntries, toolConnections, toolInvocations } from "@paperclipai/db"; @@ -86,7 +87,7 @@ listActionRequests: async ( if (!readable) invalidRequestIds.push(request.id); } if (invalidRequestIds.length > 0) { - await db + await writeToolReviewChange(db, companyId, tx => tx .update(toolActionRequests) .set({ status: "cancelled", @@ -99,7 +100,7 @@ listActionRequests: async ( eq(toolActionRequests.status, "pending"), inArray(toolActionRequests.id, invalidRequestIds), ), - ); + )); } const hiddenIds = new Set([ ...invalidRequestIds, diff --git a/server/src/services/tool-action-review.ts b/server/src/services/tool-action-review.ts index a1735487f0b..664b4cdc194 100644 --- a/server/src/services/tool-action-review.ts +++ b/server/src/services/tool-action-review.ts @@ -1,3 +1,4 @@ +import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; import { notifyDeliveryWork, DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import { and, eq } from "drizzle-orm"; import { @@ -140,6 +141,7 @@ export async function commitToolActionReview( throw conflict("This review has already been resolved"); } const now = new Date(); + await notifyToolReviewChange(tx, current.companyId); const [updated] = await tx .update(toolActionRequests) .set({ diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index c75413e8529..2ff322ad308 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1,3 +1,5 @@ +import { toolReviewExpiryService } from "./tool-review-expiry.js"; +import { writeToolReviewChange } from "./tool-review-work-notifications.js"; import { restConnectorFor } from "./connectors/rest-registry.js"; import type { runPromptInjectionProvider } from "./prompt-injection-provider.js"; import { promptInjectionPreventionService } from "./prompt-injection-prevention.js"; @@ -2569,7 +2571,7 @@ export function createToolGatewayService( signingSecret: options.toolActionSigningSecret, }); } catch (error) { - await db + await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "cancelled", @@ -2581,7 +2583,7 @@ export function createToolGatewayService( eq(toolActionRequests.id, actionRequest.id), eq(toolActionRequests.status, "pending"), ), - ); + )); if (error instanceof ToolActionSigningSecretMissingError) { await db .update(toolInvocations) @@ -2728,7 +2730,7 @@ export function createToolGatewayService( // an expired row. When the guard matches no row, the row is already resolved. // Do not emit an approval card for a dead row; throw a retriable conflict so // the retry finds the live request or creates a fresh, signable one. - const signedRows = await db + const signedRows = await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ interactionId: interaction.id, @@ -2746,7 +2748,7 @@ export function createToolGatewayService( eq(toolActionRequests.status, "pending"), ), ) - .returning({ id: toolActionRequests.id }); + .returning({ id: toolActionRequests.id })); if (signedRows.length === 0) { await db .update(toolInvocations) @@ -8667,7 +8669,7 @@ export function createToolGatewayService( pendingRequest.expiresAt.getTime() <= Date.now(); if (pendingUnsigned || pendingExpired) { const now = new Date(); - await db + await writeToolReviewChange(db, pendingRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "expired", resolvedAt: now, updatedAt: now }) .where( @@ -8675,7 +8677,7 @@ export function createToolGatewayService( eq(toolActionRequests.id, match.actionRequest.id), eq(toolActionRequests.status, "pending"), ), - ); + )); await db .update(toolInvocations) .set({ @@ -8878,6 +8880,11 @@ export function createToolGatewayService( }; } + const reviewExpiry = toolReviewExpiryService(db, async expired => { + void emitConnectionInvoked(db, expired.invocationId); + await reflectToolActionInteractionLifecycle(expired); + }); + return { async recordRuntimeMcpDeliveryDiagnostic(input: { companyId: string; @@ -9730,6 +9737,7 @@ export function createToolGatewayService( }, ); } + const pendingActionRequest = recorded.actionRequest; const canonicalArguments = canonicalToolArguments(requestedParameters); const canonicalArgumentsHash = toolRequestIdentityHash(requestedParameters, options.toolActionSigningSecret); const approvalSnapshot = await connectedRemoteApprovalSnapshot( @@ -9750,7 +9758,7 @@ export function createToolGatewayService( signingSecret: options.toolActionSigningSecret, }); } catch (error) { - await db + await writeToolReviewChange(db, pendingActionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "cancelled", @@ -9759,10 +9767,10 @@ export function createToolGatewayService( }) .where( and( - eq(toolActionRequests.id, recorded.actionRequest.id), + eq(toolActionRequests.id, pendingActionRequest.id), eq(toolActionRequests.status, "pending"), ), - ); + )); if (error instanceof ToolActionSigningSecretMissingError) { await db .update(toolInvocations) @@ -9791,7 +9799,7 @@ export function createToolGatewayService( tool, argumentsSummary: argumentValidation.summary, }); - await db + await writeToolReviewChange(db, pendingActionRequest.companyId, tx => tx .update(toolActionRequests) .set({ canonicalArgumentsHash, @@ -9801,7 +9809,7 @@ export function createToolGatewayService( expiresAt: new Date(Date.now() + 60 * 60 * 1000), updatedAt: new Date(), }) - .where(eq(toolActionRequests.id, recorded.actionRequest.id)); + .where(eq(toolActionRequests.id, pendingActionRequest.id))); await writeToolCallEvent({ invocationId, actionRequestId: recorded.actionRequest.id, @@ -9964,7 +9972,12 @@ export function createToolGatewayService( return buildTestCallStatus(actionRequest, invocation); }, - async sweepActionReviews() { + sweepExpiredActionReviews: reviewExpiry.sweep, + nextActionReviewExpiryAt: reviewExpiry.next, + actionReviewExpiryDeadlines: reviewExpiry.deadlines, + sweepExpiredActionReviewsForCompany: reviewExpiry.sweepForCompany, + + async sweepActionReviews(input: { includeExpiry?: boolean } = {}) { // A process may stop between persisting a provider outcome and updating the // feed projection. Reconcile from authoritative rows before delivering it. const unreflected = await db @@ -10002,7 +10015,7 @@ export function createToolGatewayService( const now = new Date(); const staleAt = new Date(now.getTime() - 10 * 60_000); let cursor: string | undefined; - let scanned = 0; + let scanned = input.includeExpiry === false ? 0 : (await reviewExpiry.sweep(now)).scanned; for (;;) { const rows = await db .select() @@ -10010,10 +10023,6 @@ export function createToolGatewayService( .where( and( or( - and( - eq(toolActionRequests.status, "pending"), - lte(toolActionRequests.expiresAt, now), - ), eq(toolActionRequests.status, "approved"), and( eq(toolActionRequests.status, "executing"), @@ -10039,15 +10048,9 @@ export function createToolGatewayService( ); continue; } - const status = row.status === "pending" ? "expired" : "failed"; - const errorCode = - status === "failed" - ? "tool_execution_outcome_unknown" - : "action_expired"; - const errorMessage = - status === "failed" - ? "Execution was interrupted; the external outcome is unknown. Inspect the provider before retrying." - : "The approval request expired before a decision."; + const status = "failed"; + const errorCode = "tool_execution_outcome_unknown"; + const errorMessage = "Execution was interrupted; the external outcome is unknown. Inspect the provider before retrying."; const [changed] = await db .update(toolActionRequests) .set({ status, resolvedAt: now, updatedAt: now }) @@ -10187,7 +10190,7 @@ export function createToolGatewayService( } if (!signedPayload) { if (actionRequest.status === "pending") { - await db + await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "cancelled", @@ -10199,7 +10202,7 @@ export function createToolGatewayService( eq(toolActionRequests.id, actionRequest.id), eq(toolActionRequests.status, "pending"), ), - ); + )); } await reflectToolActionInteractionLifecycle({ actionRequestId: actionRequest.id, @@ -10764,7 +10767,7 @@ export function createToolGatewayService( actionRequest.expiresAt.getTime() <= Date.now() ) { const expiredAt = new Date(); - const [expired] = await db + const [expired] = await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "expired", @@ -10777,7 +10780,7 @@ export function createToolGatewayService( inArray(toolActionRequests.status, ["pending", "approved"]), ), ) - .returning({ id: toolActionRequests.id }); + .returning({ id: toolActionRequests.id })); if (expired) { await reflectToolActionInteractionLifecycle({ actionRequestId: expired.id, @@ -10812,7 +10815,7 @@ export function createToolGatewayService( interaction?.kind === "request_confirmation" && interaction.status === "accepted" ) { - const [approved] = await db + const [approved] = await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "approved", @@ -10830,7 +10833,7 @@ export function createToolGatewayService( eq(toolActionRequests.status, "pending"), ), ) - .returning(); + .returning()); if (!approved) { throw new ToolGatewayHttpError( 409, diff --git a/server/src/services/tool-review-expiry-work.ts b/server/src/services/tool-review-expiry-work.ts new file mode 100644 index 00000000000..8b7a8caad13 --- /dev/null +++ b/server/src/services/tool-review-expiry-work.ts @@ -0,0 +1,25 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerScopedDeadlineWork } from "./scoped-deadline-work.js"; +import type { createToolGatewayService } from "./tool-gateway.js"; + +export function registerToolReviewExpiryWork( + coordinator: ReturnType, + gateway: Pick, "sweepExpiredActionReviewsForCompany" | "actionReviewExpiryDeadlines">, + canRun: () => boolean, +) { + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.toolReviewExpiry, + batchSize: 1, + minimumPassIntervalMs: 0, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: (entry, companyId?: string) => companyId === undefined || entry.companyId === companyId, + load: async (companyId?: string) => (await gateway.actionReviewExpiryDeadlines(companyId)) + .map(row => ({ ...row, id: row.companyId })), + check: async entry => { + await gateway.sweepExpiredActionReviewsForCompany(entry.companyId); + const [next] = await gateway.actionReviewExpiryDeadlines(entry.companyId); + return next ? { ...next, id: next.companyId } : null; + }, + }, canRun); +} diff --git a/server/src/services/tool-review-expiry.ts b/server/src/services/tool-review-expiry.ts new file mode 100644 index 00000000000..8db647ce149 --- /dev/null +++ b/server/src/services/tool-review-expiry.ts @@ -0,0 +1,65 @@ +import { and, asc, eq, gt, isNotNull, lte, sql } from "drizzle-orm"; +import { toolActionRequests, toolInvocations, type Db } from "@paperclipai/db"; +import { writeToolReviewChange } from "./tool-review-work-notifications.js"; + +export function toolReviewExpiryService(db: Db, onExpired: (input: { + actionRequestId: string; + invocationId: string; + status: "expired"; + errorCode: string; + errorMessage: string; +}) => Promise) { + return { + async next(): Promise { + const [row] = await db.select({ at: sql`min(${toolActionRequests.expiresAt})` }) + .from(toolActionRequests).where(eq(toolActionRequests.status, "pending")); + return row?.at == null ? null : new Date(row.at).getTime(); + }, + sweep, + sweepForCompany: (companyId: string) => sweep(new Date(), companyId, 1), + async deadlines(companyId?: string) { + const pendingExpiry = and(eq(toolActionRequests.status, "pending"), isNotNull(toolActionRequests.expiresAt)); + const rows = companyId === undefined + ? await db.select({ companyId: toolActionRequests.companyId, + at: sql`min(${toolActionRequests.expiresAt})` }) + .from(toolActionRequests).where(pendingExpiry).groupBy(toolActionRequests.companyId) + : await db.select({ companyId: toolActionRequests.companyId, + at: sql`${toolActionRequests.expiresAt}` }) + .from(toolActionRequests).where(and(pendingExpiry, eq(toolActionRequests.companyId, companyId))) + .orderBy(asc(toolActionRequests.expiresAt)).limit(1); + return rows.map(row => ({ companyId: row.companyId, dueAt: new Date(row.at).getTime() })); + }, + }; + async function sweep(now = new Date(), companyId?: string, maxPages = Infinity) { + let cursor: string | undefined; + let scanned = 0; + for (let page = 0; page < maxPages; page++) { + const rows = await db.select().from(toolActionRequests).where(and( + eq(toolActionRequests.status, "pending"), lte(toolActionRequests.expiresAt, now), + cursor ? gt(toolActionRequests.id, cursor) : undefined, + companyId === undefined ? undefined : eq(toolActionRequests.companyId, companyId), + )).orderBy(asc(toolActionRequests.id)).limit(100); + for (const row of rows) { + const errorCode = "action_expired"; + const errorMessage = "The approval request expired before a decision."; + const changed = await writeToolReviewChange(db, row.companyId, async tx => { + const [updated] = await tx.update(toolActionRequests) + .set({ status: "expired", resolvedAt: now, updatedAt: now }).where(and( + eq(toolActionRequests.id, row.id), eq(toolActionRequests.status, "pending"), + eq(toolActionRequests.updatedAt, row.updatedAt), + )).returning({ id: toolActionRequests.id }); + if (!updated) return false; + await tx.update(toolInvocations).set({ status: "failed", errorCode, errorMessage, + completedAt: now, updatedAt: now }).where(eq(toolInvocations.id, row.invocationId)); + return true; + }); + if (changed) await onExpired({ actionRequestId: row.id, invocationId: row.invocationId, + status: "expired", errorCode, errorMessage }); + } + scanned += rows.length; + if (rows.length < 100) break; + cursor = rows[rows.length - 1].id; + } + return { scanned }; + } +} diff --git a/server/src/services/tool-review-work-notifications.ts b/server/src/services/tool-review-work-notifications.ts new file mode 100644 index 00000000000..19c8e433d75 --- /dev/null +++ b/server/src/services/tool-review-work-notifications.ts @@ -0,0 +1,13 @@ +import type { Db } from "@paperclipai/db"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export function notifyToolReviewChange(transaction: object, companyId: string) { + return notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewExpiry, companyId); +} + +export function writeToolReviewChange(db: Db, companyId: string, write: (transaction: Db) => Promise): Promise { + return db.transaction(async transaction => { + await notifyToolReviewChange(transaction, companyId); + return write(transaction as unknown as Db); + }); +} From f60ac29cf116015deb64eb8a1f24d02f6f35fe56 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 15:28:42 -0500 Subject: [PATCH 07/41] fix(runner): migrate Grok Build with native reasoning effort (#17922) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Harness selection maps supported agents to Paperclip Runner. > - The native Runner already supports Grok Build, but its migration mapping excludes Grok. > - Grok agents also have a thinking-effort setting that the native execution contract does not carry. > - This pull request adds the migration path and preserves explicit Grok effort through startup and resume. > - Operators can upgrade an existing Grok agent without silently losing its model, account, or effort. ## Linked Issues or Issue Description **What happened?** Existing Grok Build agents have no Runner selector or upgrade action. Adding only the mapping would route conversion through the Codex fallback and lose Grok execution semantics. Native execution also ignores the legacy reasoning-effort field. **Expected behavior** A qualified Grok agent can explicitly upgrade. The saved model, account, instructions, history, and supported effort remain intact. Unsupported legacy settings fail before saving. The provider must acknowledge the selected effort before inference. **Steps to reproduce** 1. Open the runtime page of an existing Grok Build agent on a qualified Linux x64 or macOS ARM64 computer. 2. Observe that the Runner selector and upgrade action are absent. 3. Compare the native execution input with the saved legacy reasoning-effort setting. Related work: #13882 introduced native Grok Build. #15422 covers the earlier default-runner work. This change extends the current shared selection contract. ## What Changed - Add the Grok harness mapping, qualified platform limits, upgrade action, and company-scoped computer qualification. - Translate Grok settings to ACPX without changing the selected account or explicit model. Resolve the legacy default model to the native default. Preserve the effective legacy search policy. Reject incompatible command, argument, permission, and turn-limit settings. - Add native execution input v10 for explicit Grok effort and search policy. Preserve historical schemas. Enforce disabled search through Grok requirements and bind it across startup and recovery. - Apply and verify ACP `reasoning_effort` before prompting. Bind the value to provider descriptors, checkpoints, runtime identity, and resume checks. - Use the Grok catalog and effort choices in the task composer. Apply task model and effort overrides while retaining the agent account and search policy. Update runtime controls, portability/default handling, documentation, and capability maps. - Advance Pi and Copilot source attestations for changed shared transport and recovery code. Keep historical identities intact and update all Rust integration fixtures. ## Verification - Migration and review regressions were reproduced before the fixes. Coverage includes the Grok mapping, default model, explicit effort, implicit search policy, task overrides, and incompatible legacy settings. The broad migration, route, immutable-input, and composer check passed 478 tests. The final catalog and task-input check passed 135 tests. - Runner startup, recovery, search policy, and profile checks passed 361 tests, with one platform skip. The final source-identity, search-admission, and transport event-pump check passed 89 tests. - All 757 Rust workspace tests passed for the broad implementation. After the final source attestation update, all 55 ACPX profile/session tests passed. - `pnpm -r typecheck` and `pnpm build` passed for the implementation. Server and Runner typecheck passed after the final production change. Server `tsc --noEmit` passed after the final test-fixture change. Docker context integrity, token gates, maps, generated protocol data, and replay goldens passed. - The v10 startup fixture initially used an ambient local Grok login and failed on CI. An explicitly empty credential home reproduced all four failures. The fixture now supplies a synthetic key and a private empty home. A separate case checks that missing credentials fail closed. All seven selected Grok v10 and Claude v9 startup cases pass without host credentials. - The full local suite could not complete because embedded PostgreSQL failed before database-backed fixture setup. The broad executor-suite attempt hit the same limit and was stopped. Neither attempt is counted as passing. Current-head CI must verify those paths before merge. - Earlier CI found missing Rust input fields, stale source attestations and exact fixture pins, a missing Docker context entry, and an outdated import expectation. Those failures were fixed and checked locally. The CI logs retain the original failures. - Different earlier CI runs also failed unchanged cleanup pagination, device-login timeout, and chat lost-response recovery cases. Later attempts passed the cleanup and login cases without changes to those implementations. Their causes remain unproven. The original evidence remains in [the first run](https://github.com/paperclipai/paperclip/actions/runs/40563402840), [the login run](https://github.com/paperclipai/paperclip/actions/runs/40571128328), and [the chat run](https://github.com/paperclipai/paperclip/actions/runs/40573072163). A passing retry does not establish a root-cause fix. - Both reviews rate current head `16fba9b4cdf12bc7945e6628912e79da42959f35` 5/5. All current-head CI gates pass in [run 40576580699](https://github.com/paperclipai/paperclip/actions/runs/40576580699). All 60 check runs completed cleanly, including the four intentional skips and the neutral 5/5 Paperclip review. The legacy security status is successful. No review threads remain unresolved. - Staging migration and a real shell task are pending deployment. This PR does not claim full Grok provider qualification. Reviewers can verify the operator path by upgrading a qualified Grok agent, checking its saved model and effort, and running a shell task through native execution. ## Risks - Explicit effort and search policy require the matching Runner and sidecar build. Older immutable execution inputs keep their existing schemas. - Providers that omit or reject the selected effort fail before a prompt. Unsupported effort does not fall back silently. - Existing agents retain their saved runner until an explicit change. New Grok agents use native execution only on qualified platforms. - Grok token and cost reporting remain unverified, as before. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window capacity are not exposed in this session. Used reasoning, source editing, shell execution, and browser inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and the focused suites pass; the full local suite has the environment limitation recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .dockerignore | 1 + doc/DEVELOPING.md | 8 +- doc/PRODUCT.md | 2 +- doc/SPEC-implementation.md | 2 +- doc/grok-native-runner.md | 29 ++++++- maps/agent-access-map/agents.md | 6 +- maps/feature-map/agents.md | 6 +- .../src/paperclip-runner-permissions.test.ts | 10 +++ .../src/paperclip-runner-permissions.ts | 8 +- packages/paperclip-runner/acpx-profiles.json | 8 +- .../paperclip-runner/protocol/manifest.json | 2 +- .../schemas/provider-descriptor.schema.json | 52 ++++++++++++ .../runner-core/src/acpx_provider_backend.rs | 45 +++++++++- .../src/acpx_provider_checkpoint.rs | 10 +++ .../runner-core/src/acpx_provider_session.rs | 79 +++++++++++++++++ .../src/generated_acpx_profiles.rs | 4 +- .../tests/acpx_provider_checkpoint.rs | 4 + .../tests/acpx_provider_resolutions.rs | 2 + .../tests/acpx_provider_session.rs | 4 + .../tests/acpx_provider_suspend.rs | 2 + .../runner-core/tests/acpx_provider_turns.rs | 2 + .../tests/native_provider_backend.rs | 2 +- .../scripts/copilot-profile-identity.mjs | 4 +- .../src/backends/codex-acpx-native-backend.ts | 4 +- .../src/backends/codex-native-backend.ts | 4 +- .../src/backends/opencode-native-backend.ts | 4 +- .../src/backends/runtime-context.ts | 4 +- .../src/cli/acpx-runtime-sidecar.ts | 11 ++- .../src/contracts/harness-driver.ts | 2 + .../src/contracts/native-execution.test.ts | 51 ++++++++++- .../src/contracts/native-execution.ts | 28 +++++- .../src/drivers/acpx/codex-acpx-driver.ts | 7 ++ .../acpx/codex-runtime-adapter.test.ts | 34 ++++++++ .../src/drivers/acpx/codex-runtime-adapter.ts | 23 +++-- .../drivers/acpx/copilot-installation.test.ts | 18 +++- .../src/drivers/acpx/generated-profiles.ts | 8 +- .../src/drivers/acpx/grok-effort.test.ts | 67 +++++++++++++++ .../src/drivers/acpx/grok-effort.ts | 19 +++++ .../src/drivers/acpx/grok-web-search.test.ts | 14 +++ .../src/drivers/acpx/grok-web-search.ts | 13 +++ .../src/drivers/acpx/pi-installation.test.ts | 43 ++++++---- .../src/drivers/acpx/profile-compatibility.ts | 6 +- .../drivers/acpx/recovery-identity.test.ts | 27 ++++++ .../src/drivers/acpx/recovery-identity.ts | 28 +++++- .../src/drivers/acpx/runtime-host.ts | 12 +++ .../src/drivers/acpx/runtime-sandbox.test.ts | 8 ++ .../src/drivers/acpx/runtime-sandbox.ts | 3 +- .../src/drivers/acpx/sidecar-protocol.ts | 4 + packages/paperclip-runner/src/index.ts | 3 + .../src/live/runner-provider-launch.test.ts | 11 +++ .../src/live/runner-provider-launch.ts | 7 +- .../src/live/runnerd-codex-transport.ts | 12 +++ .../src/protocol/generated/schema-bundle.ts | 52 ++++++++++++ .../pi-acp/profile-v30-identity.json | 51 +++++++++++ .../copilot-profile-v66-identity.json | 85 +++++++++++++++++++ packages/shared/src/agent-runner.ts | 4 +- scripts/pi-public-install-probe.mjs | 4 +- .../agent-adapter-validation-routes.test.ts | 61 ++++++++++++- .../agent-runner-qualification-routes.test.ts | 3 +- .../src/__tests__/company-portability.test.ts | 23 ++++- server/src/redaction.ts | 1 + server/src/routes/agents.ts | 19 +++-- .../services/agent-runner-qualification.ts | 2 +- .../services/agent-runner-selection.test.ts | 51 ++++++++++- server/src/services/agent-runner-selection.ts | 47 ++++++++-- server/src/services/built-in-agents.ts | 4 +- server/src/services/company-portability.ts | 6 +- .../heartbeat-runner-provider-config.test.ts | 15 ++++ .../services/heartbeat/runtime-selection.ts | 2 +- .../native-execution-input.test.ts | 20 +++++ .../native-runtime/native-execution-input.ts | 18 ++-- .../native-session-executor.test.ts | 30 +++++++ .../native-runtime/native-session-executor.ts | 23 ++++- .../native-runtime/plugin-runner-probe.ts | 4 +- .../native-runtime/provider-profile.ts | 21 ++++- server/src/vendor/paperclip-runner/index.ts | 4 + .../AgentConfigForm.render.test.tsx | 10 +-- ui/src/components/AgentConfigForm.tsx | 10 +-- .../task-chat/ComposerRunSettingsPicker.tsx | 5 +- .../task-chat/composer-run-settings.test.ts | 6 ++ .../task-chat/composer-run-settings.ts | 3 +- 81 files changed, 1221 insertions(+), 130 deletions(-) create mode 100644 packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts create mode 100644 packages/paperclip-runner/src/drivers/acpx/grok-effort.ts create mode 100644 packages/paperclip-runner/src/drivers/acpx/grok-web-search.test.ts create mode 100644 packages/paperclip-runner/src/drivers/acpx/grok-web-search.ts create mode 100644 packages/paperclip-runner/test-fixtures/pi-acp/profile-v30-identity.json create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json diff --git a/.dockerignore b/.dockerignore index 3bf8e79b7b1..1015d7c5b35 100644 --- a/.dockerignore +++ b/.dockerignore @@ -89,6 +89,7 @@ packages/grok-acp/bin !packages/paperclip-runner/test-fixtures/pi-acp/profile-v28-identity.json !packages/paperclip-runner/test-fixtures/pi-acp/profile-v29-identity.json +!packages/paperclip-runner/test-fixtures/pi-acp/profile-v30-identity.json !packages/paperclip-runner/test/fixtures/copilot-profile-v23-identity.json diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 05071cb68ff..4853284df55 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -2019,7 +2019,7 @@ agent conversion are outside this playbook. ## Paperclip Runner Adapter Conversion -Paperclip Runner offers native Codex, OpenCode, **ACPX Claude**, and **ACPX Pi**. New +Paperclip Runner offers native Codex, OpenCode, **ACPX Claude**, **ACPX Grok Build**, and **ACPX Pi**. New Claude Code agents default to ACPX Claude on Apple Silicon, Intel Mac, Linux x64 and Linux ARM64 with supported glibc. New OpenCode and Pi agents default to their native runtimes on Apple Silicon, Intel Mac, and Linux x64 with supported glibc; @@ -2030,11 +2030,15 @@ requires matching remote assets. The runner override is in Advanced. Ordinary Cl and provider routing remain authoritative. Existing agents keep their saved runner. Converting an existing Claude, Codex, OpenCode, or Pi agent selects its corresponding provider, preserves compatible models, credentials, workspace, and instructions, and resets execution sessions while retaining run history. +Grok Build uses its qualified macOS ARM64 and Linux x64 targets. Its conversion +preserves the selected model, account, and explicit reasoning effort through ACP. +Custom commands, extra arguments, legacy permission modes, search flags, and +nonzero turn limits require Legacy; conversion rejects them before saving. Other adapters require an explicit provider choice. Legacy ACPX Codex agent settings normalize to native Codex on configuration updates and before fresh runs; immutable run descriptors remain readable. The **Paperclip Runner** experimental setting applies to other native providers. Native Codex, OpenCode, -ACPX Claude, and ACPX Pi do not require it. Company access checks still apply. +ACPX Claude, ACPX Grok Build, and ACPX Pi do not require it. Company access checks still apply. Native Codex preserves an explicit `adapterConfig.search: false` when converting or saving an agent. The disabled hosted-search policy is recorded in the native diff --git a/doc/PRODUCT.md b/doc/PRODUCT.md index 70f5184cdf2..2acf62e5248 100644 --- a/doc/PRODUCT.md +++ b/doc/PRODUCT.md @@ -49,7 +49,7 @@ start execution. This preference does not change the org chart or permissions. ### Agent Execution -Existing Claude and Codex agents can upgrade from the legacy runner on their +Existing Claude, Codex, and Grok Build agents can upgrade from the legacy runner on their Harness / Runtime page when Paperclip Runner is the qualified default for their environment. One action saves pending edits and translates compatible settings while keeping the harness, selected model, account bindings, instructions, and diff --git a/doc/SPEC-implementation.md b/doc/SPEC-implementation.md index ee9d93c0f9c..01cae913f32 100644 --- a/doc/SPEC-implementation.md +++ b/doc/SPEC-implementation.md @@ -1356,7 +1356,7 @@ Invites tab does not hide this Cloud action. ## 10.3 Agents The Harness / Runtime configuration page offers **Upgrade runner** for existing -legacy Claude and Codex agents when the target environment selects Paperclip +legacy Claude, Codex, and Grok Build agents when the target environment selects Paperclip Runner by default. The action uses the normal authorized agent PATCH with `runner: "paperclip"`, includes pending configuration edits, and retains the model and compatible settings. A rejected conversion leaves saved configuration diff --git a/doc/grok-native-runner.md b/doc/grok-native-runner.md index dfd2096ea29..e9a820d1be4 100644 --- a/doc/grok-native-runner.md +++ b/doc/grok-native-runner.md @@ -2,14 +2,37 @@ Select **Grok Build** in the native runner provider selector. The stored contract is `adapterType: "paperclip_runner"` with `provider: "acpx"`, `acpxAgent: "grok"`, -and `model: "grok-4.7"`. Existing `grok_local` agents keep their legacy adapter. +and `model: "grok-4.7"`. Existing `grok_local` agents keep their legacy adapter +until an explicit upgrade. On a qualified macOS ARM64 or Linux x64 computer, +**Harness / Runtime → Upgrade runner** preserves the agent, model, account, +instructions, and history. It starts fresh execution sessions. Advanced also +offers an explicit Runner choice. Unsupported legacy settings are rejected +before saving; select compatible settings or keep Legacy. New Grok runner agents default to **Full auto (approve all)** (`acpxPermissionMode: "approve-all"`) in setup and the configuration form. API configurations that omit the permission mode use the same default. No additional permission setting is needed for unattended execution. Explicitly saved restrictions remain unchanged. -On Cloud, an operator must enable `enableNativeRunner` for the instance before -the new-agent picker or direct setup page offers the native runner. +Grok Build's qualified harness does not require the `enableNativeRunner` flag. + +Explicit thinking effort uses `reasoningEffort` (`minimal`, `low`, `medium`, +`high`, or `xhigh`). The immutable v10 execution input, provider descriptor, +and recovery identity bind this value as `grokEffort`. Before a prompt, the +Runner applies ACP `reasoning_effort` and requires Grok to acknowledge the exact +value. Missing configuration, a rejected value, or effort drift stops admission. +Omitting effort retains Grok's provider default. This setting remains outside +the model-visible task prompt. Historical execution inputs retain their schemas. + +The upgrade resolves an unset model or the legacy `grok-build` default to the +native `grok-4.7` default. Task-level model and effort overrides use the Grok +catalog and keep the account and tool policy unchanged. + +Migration also preserves the effective `disableWebSearch` setting, whose legacy +default is `true`. Native input v10 carries it as `grokDisableWebSearch` through +startup and recovery. Grok's private `requirements.toml` enforces disabled search +above compatible project settings. Explicit `false` retains enabled search; +unset native configurations retain their provider default. See Grok's +[configuration reference](https://github.com/xai-org/grok-build/blob/main/crates/codegen/xai-grok-pager/docs/user-guide/26-config-reference.md). Grok Build speaks [ACP over stdio](https://github.com/xai-org/grok-build/blob/main/crates/codegen/xai-grok-pager/docs/user-guide/15-agent-mode.md). The runner owns `grok agent --no-leader stdio` through ACPX, including session diff --git a/maps/agent-access-map/agents.md b/maps/agent-access-map/agents.md index dd89f5abef0..2fa84783c00 100644 --- a/maps/agent-access-map/agents.md +++ b/maps/agent-access-map/agents.md @@ -139,9 +139,11 @@ environment, and consent checks as other edits. It adds no agent permission or credential fallback. Pool and externally managed agents use their existing configuration surfaces. Board browser verification does not qualify agent access. -The existing-agent upgrade notice uses this configuration PATCH. It does not +The existing-agent upgrade notice for Claude, Codex, and Grok Build uses this configuration PATCH. It does not grant configuration, account, or environment access. The server resolves the -saved settings under normal company/configuration authorization. Obsolete Claude +saved settings under normal company/configuration authorization. Grok migration preserves +the effective legacy search setting. Task model/effort overrides retain the +agent's account binding and tool policy without granting extra account access. Obsolete Claude tuning fields are removed with `configurationWarnings`; security incompatibilities remain errors. Unqualified sandbox upgrades first use the agent-scoped `POST /api/agents/{id}/qualify-runner` under the same `agent_config:update` diff --git a/maps/feature-map/agents.md b/maps/feature-map/agents.md index 21dee0c054b..f5fce683f77 100644 --- a/maps/feature-map/agents.md +++ b/maps/feature-map/agents.md @@ -167,12 +167,16 @@ Selecting another runner must not silently discard security settings or change a recorded execution. Keep runner choice in Advanced and verify the next eligible run separately from historical transcripts. -Existing legacy Claude and Codex agents on qualified default environments, or +Existing legacy Claude, Codex, and Grok Build agents on qualified default environments, or probe-capable Daytona snapshots / immutable sandbox images, show **Upgrade runner** above their configuration. Verify a company owner with `agents:configure` and no instance-admin role can qualify the saved computer and upgrade. A caller without configuration permission must be denied before a probe starts. Exercise that action with pending edits and an environment-variable draft. An unsaved change to a different effective computer must disable the action and explain that the computer change must be saved first; restoring the saved target must re-enable it. Reopen the page and compare the saved harness, model, account bindings, instructions, and run policy. Check a +Grok upgrade with an unset or `grok-build` model: it must use `grok-4.7`. +Compare effective search policy before and after, including the implicit disabled +default and explicit enabled search. Verify task model and effort changes use the +Grok catalog and reach execution while retaining the account and search policy. Check a saved `maxTurnsPerRun: 1000`: upgrade must warn and remove it. Test an initially unqualified Daytona Paperclip Computer: click must probe, record qualification, refresh availability, and only then save. A failed probe or changed selection must diff --git a/packages/adapter-utils/src/paperclip-runner-permissions.test.ts b/packages/adapter-utils/src/paperclip-runner-permissions.test.ts index fcaeea10fca..19900ca47b8 100644 --- a/packages/adapter-utils/src/paperclip-runner-permissions.test.ts +++ b/packages/adapter-utils/src/paperclip-runner-permissions.test.ts @@ -44,6 +44,7 @@ describe("Paperclip Runner permission defaults", () => { ["codex_local", "codexPermissionMode", "never"], ["opencode_local", "opencodePermissionMode", "allow"], ["pi_local", "acpxPermissionMode", "approve-all"], + ["grok_local", "acpxPermissionMode", "approve-all"], ])("uses full auto when converting %s to the new runner", (adapter, key, value) => { expect(paperclipRunnerTransitionConfig(adapter, undefined)).toMatchObject({ [key]: value }); }); @@ -53,6 +54,15 @@ describe("Paperclip Runner permission defaults", () => { expect(paperclipRunnerTransitionConfig("pi_local", "openrouter/provider/model")).toMatchObject({ provider: "acpx", acpxAgent: "pi", model: "openrouter/provider/model" }); }); + it("keeps Grok's harness and model when converting its adapter", () => { + expect(paperclipRunnerTransitionConfig("grok_local", "grok-4.7")) + .toMatchObject({ provider: "acpx", acpxAgent: "grok", model: "grok-4.7" }); + }); + + it.each([undefined, "", "grok-build"])("resolves Grok's legacy default %s to the native model", model => { + expect(paperclipRunnerTransitionConfig("grok_local", model)).toMatchObject({ model: "grok-4.7" }); + }); + it("recognizes only exact provider identifiers", () => { expect(isPaperclipRunnerProvider("codex")).toBe(true); expect(isPaperclipRunnerProvider("opencode")).toBe(true); diff --git a/packages/adapter-utils/src/paperclip-runner-permissions.ts b/packages/adapter-utils/src/paperclip-runner-permissions.ts index 3cb2e323e96..7abe19ed0a5 100644 --- a/packages/adapter-utils/src/paperclip-runner-permissions.ts +++ b/packages/adapter-utils/src/paperclip-runner-permissions.ts @@ -205,7 +205,7 @@ export function paperclipRunnerTransitionConfig( providerOverride?: unknown, ): Record { const previousProvider = - (previousAdapterType === "claude_local" || previousAdapterType === "pi_local") + (["claude_local", "pi_local", "grok_local"].includes(previousAdapterType)) ? "acpx" : previousAdapterType === "opencode_local" ? "opencode" @@ -218,13 +218,15 @@ export function paperclipRunnerTransitionConfig( : previousProvider; return { provider, - model: provider === "acpx" && previousAdapterType === "pi_local" + model: provider === "acpx" && previousAdapterType === "grok_local" + ? typeof previousModel === "string" && previousModel.trim() && previousModel.trim() !== "grok-build" ? previousModel.trim() : "grok-4.7" + : provider === "acpx" && previousAdapterType === "pi_local" ? typeof previousModel === "string" ? previousModel.trim() : "" : resolvePaperclipRunnerModel( provider, provider === previousProvider ? previousModel : undefined, ), - ...(provider === "acpx" ? { acpxAgent: previousAdapterType === "pi_local" ? "pi" : "claude" } : {}), + ...(provider === "acpx" ? { acpxAgent: previousAdapterType === "grok_local" ? "grok" : previousAdapterType === "pi_local" ? "pi" : "claude" } : {}), [PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES[provider].configKey]: PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES[provider].defaultMode, lifecycleMode: "per_turn", diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index 885c8d2a6be..eda0ebf68c0 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -14,12 +14,12 @@ "requiresProviderPolicy": false }, "pi": { - "agentProfileVersion": 29, + "agentProfileVersion": 30, "agentServerPackage": "pi-acp", "agentServerVersion": "0.0.33", "agentRuntimePackage": "@earendil-works/pi-coding-agent", "agentRuntimeVersion": "1.0.0", - "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", + "commandDigest": "sha256:4517ab123dd89140ef462fc1905ac7a16a0dfcbaf21d67299a3c62de29c08a4d", "requiresProviderPolicy": true }, "cursor": { @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 65, + "agentProfileVersion": 66, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:277894e850e07e6801cc24299c237513f58b41eb93c24f751d984fc4ddb04c47", + "commandDigest": "sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/protocol/manifest.json b/packages/paperclip-runner/protocol/manifest.json index 07259dd4199..da5c50ac321 100644 --- a/packages/paperclip-runner/protocol/manifest.json +++ b/packages/paperclip-runner/protocol/manifest.json @@ -85,7 +85,7 @@ { "path": "schemas/provider-descriptor.schema.json", "id": "https://paperclip.dev/schemas/prp/v1/provider-descriptor.schema.json", - "sha256": "862c0c13f23564c3530cefcf01579b0535a565b919cc3dcff7bc1bee9a668baf" + "sha256": "81d525335245dd4cb8a9bf41e3390dc60c6dd3e32474aad48b4f18a94d5f9711" }, { "path": "schemas/provider-event.schema.json", diff --git a/packages/paperclip-runner/protocol/schemas/provider-descriptor.schema.json b/packages/paperclip-runner/protocol/schemas/provider-descriptor.schema.json index 90193d35313..6a4399c2fbb 100644 --- a/packages/paperclip-runner/protocol/schemas/provider-descriptor.schema.json +++ b/packages/paperclip-runner/protocol/schemas/provider-descriptor.schema.json @@ -168,6 +168,18 @@ "xhigh", "max" ] + }, + "grokEffort": { + "enum": [ + "minimal", + "low", + "medium", + "high", + "xhigh" + ] + }, + "grokDisableWebSearch": { + "type": "boolean" } }, "allOf": [ @@ -518,6 +530,46 @@ } } } + }, + { + "if": { + "required": [ + "grokEffort" + ] + }, + "then": { + "properties": { + "provider": { + "const": "acpx" + }, + "agent": { + "const": "grok" + } + }, + "required": [ + "agent" + ] + } + }, + { + "if": { + "required": [ + "grokDisableWebSearch" + ] + }, + "then": { + "properties": { + "provider": { + "const": "acpx" + }, + "agent": { + "const": "grok" + } + }, + "required": [ + "agent" + ] + } } ], "$defs": { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index 69540277da3..81d1ff266f4 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -145,6 +145,10 @@ struct AcpxProviderDescriptor { pi_thinking_level: Option, #[serde(default, skip_serializing_if = "Option::is_none")] claude_effort: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + grok_effort: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + grok_disable_web_search: Option, permission_mode_pinned: bool, #[serde(default)] provider_policy: Option, @@ -273,6 +277,14 @@ impl AcpxProviderDescriptor { self.agent != "claude" || !matches!(effort.as_str(), "low" | "medium" | "high" | "xhigh" | "max") }) + || (self.grok_disable_web_search.is_some() && self.agent != "grok") + || self.grok_effort.as_ref().is_some_and(|effort| { + self.agent != "grok" + || !matches!( + effort.as_str(), + "minimal" | "low" | "medium" | "high" | "xhigh" + ) + }) || self.mode.as_ref().is_some_and(|mode| { mode.trim().is_empty() || mode.chars().count() > 240 @@ -375,6 +387,8 @@ impl AcpxProviderDescriptor { mode: self.mode.clone(), pi_thinking_level: self.pi_thinking_level, claude_effort: self.claude_effort.clone(), + grok_effort: self.grok_effort.clone(), + grok_disable_web_search: self.grok_disable_web_search.clone(), permission_mode_pinned: self.permission_mode_pinned, provider_policy: self.provider_policy.clone(), system_instructions: self.instructions.clone(), @@ -482,6 +496,12 @@ impl AcpxProviderDescriptor { if let Some(effort) = &self.claude_effort { descriptor["claudeEffort"] = json!(effort); } + if let Some(effort) = &self.grok_effort { + descriptor["grokEffort"] = json!(effort); + } + if let Some(disabled) = self.grok_disable_web_search { + descriptor["grokDisableWebSearch"] = json!(disabled); + } if let Some(level) = self.pi_thinking_level { descriptor["piThinkingLevel"] = json!(level); } @@ -622,6 +642,8 @@ impl AcpxDurableState { || identity.mode != self.descriptor.mode || identity.pi_thinking_level != self.descriptor.pi_thinking_level || identity.claude_effort != self.descriptor.claude_effort + || identity.grok_effort != self.descriptor.grok_effort + || identity.grok_disable_web_search != self.descriptor.grok_disable_web_search { return Err(DurableRunnerError::invalid( "ACPX durable identity no longer matches its qualified profile or provider mode", @@ -1309,6 +1331,12 @@ impl AcpxCommandExecutor { if let Some(effort) = &identity.claude_effort { result["claudeEffort"] = json!(effort); } + if let Some(effort) = &identity.grok_effort { + result["grokEffort"] = json!(effort); + } + if let Some(disabled) = identity.grok_disable_web_search { + result["grokDisableWebSearch"] = json!(disabled); + } if let Some(level) = identity.pi_thinking_level { // This identity is admitted only after the sidecar's effective-mode ACK. result["piThinkingLevel"] = json!(level); @@ -2783,6 +2811,8 @@ mod tests { mode: descriptor.mode.clone(), pi_thinking_level: descriptor.pi_thinking_level, claude_effort: descriptor.claude_effort.clone(), + grok_effort: descriptor.grok_effort.clone(), + grok_disable_web_search: descriptor.grok_disable_web_search.clone(), provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], }; let operations = Vec::new(); @@ -3060,6 +3090,8 @@ mod tests { mode: None, pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], }; @@ -3214,7 +3246,7 @@ mod tests { #[test] fn pi_profile_matches_published_identity_and_rejects_prior_profiles() { let published: Value = serde_json::from_str(include_str!( - "../../../../test-fixtures/pi-acp/profile-v29-identity.json" + "../../../../test-fixtures/pi-acp/profile-v30-identity.json" )) .unwrap(); let mut value = descriptor("codex"); @@ -3231,6 +3263,7 @@ mod tests { current.validate(&context()).unwrap(); for prior in [ + include_str!("../../../../test-fixtures/pi-acp/profile-v29-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v28-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v24-identity.json"), include_str!("../../../../test-fixtures/pi-acp/profile-v23-identity.json"), @@ -3550,6 +3583,8 @@ mod tests { mode: original_descriptor.mode.clone(), pi_thinking_level: original_descriptor.pi_thinking_level, claude_effort: original_descriptor.claude_effort.clone(), + grok_effort: original_descriptor.grok_effort.clone(), + grok_disable_web_search: original_descriptor.grok_disable_web_search.clone(), provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], }; let operations = Vec::new(); @@ -3836,6 +3871,8 @@ mod tests { mode: original_descriptor.mode.clone(), pi_thinking_level: original_descriptor.pi_thinking_level, claude_effort: original_descriptor.claude_effort.clone(), + grok_effort: original_descriptor.grok_effort.clone(), + grok_disable_web_search: original_descriptor.grok_disable_web_search.clone(), provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], }; let operations = Vec::new(); @@ -4155,6 +4192,8 @@ mod tests { mode: original_descriptor.mode.clone(), pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], }; let operations = Vec::new(); @@ -4404,6 +4443,8 @@ mod tests { mode: descriptor.mode.clone(), pi_thinking_level: descriptor.pi_thinking_level, claude_effort: descriptor.claude_effort.clone(), + grok_effort: descriptor.grok_effort.clone(), + grok_disable_web_search: descriptor.grok_disable_web_search.clone(), provider_lifetime_fence_candidates, }; let operations = Vec::new(); @@ -4578,6 +4619,8 @@ mod tests { mode: provider_descriptor.mode.clone(), pi_thinking_level: provider_descriptor.pi_thinking_level, claude_effort: provider_descriptor.claude_effort.clone(), + grok_effort: provider_descriptor.grok_effort.clone(), + grok_disable_web_search: provider_descriptor.grok_disable_web_search.clone(), provider_lifetime_fence_candidates, }); state.provider_exit_unconfirmed = true; diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_checkpoint.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_checkpoint.rs index 60f575b82e8..8d74b80f25e 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_checkpoint.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_checkpoint.rs @@ -55,6 +55,10 @@ struct PersistedAcpxProviderSessionIdentity { pi_thinking_level: Option, #[serde(default, skip_serializing_if = "Option::is_none")] claude_effort: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + grok_effort: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + grok_disable_web_search: Option, provider_lifetime_fence_candidates: [u16; 3], } @@ -80,6 +84,8 @@ impl PersistedAcpxProviderSessionIdentity { mode: identity.mode, pi_thinking_level: identity.pi_thinking_level, claude_effort: identity.claude_effort.clone(), + grok_effort: identity.grok_effort.clone(), + grok_disable_web_search: identity.grok_disable_web_search.clone(), provider_lifetime_fence_candidates: identity.provider_lifetime_fence_candidates, }) } @@ -99,6 +105,8 @@ impl PersistedAcpxProviderSessionIdentity { mode: self.mode.clone(), pi_thinking_level: self.pi_thinking_level, claude_effort: self.claude_effort.clone(), + grok_effort: self.grok_effort.clone(), + grok_disable_web_search: self.grok_disable_web_search.clone(), provider_lifetime_fence_candidates: self.provider_lifetime_fence_candidates, } } @@ -122,6 +130,8 @@ impl AcpxSuspensionCheckpoint { || identity.mode != config.mode || identity.pi_thinking_level != config.pi_thinking_level || identity.claude_effort != config.claude_effort + || identity.grok_effort != config.grok_effort + || identity.grok_disable_web_search != config.grok_disable_web_search || config .expected_identity .as_ref() diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs index bde545b624e..8265121050f 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_session.rs @@ -74,6 +74,10 @@ pub struct AcpxProviderSessionIdentity { pub pi_thinking_level: Option, #[serde(default, skip_serializing_if = "Option::is_none")] pub claude_effort: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub grok_effort: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub grok_disable_web_search: Option, pub provider_lifetime_fence_candidates: [u16; 3], } @@ -97,6 +101,8 @@ pub struct AcpxProviderSessionConfig { pub mode: Option, pub pi_thinking_level: Option, pub claude_effort: Option, + pub grok_effort: Option, + pub grok_disable_web_search: Option, pub permission_mode_pinned: bool, pub provider_policy: Option, pub system_instructions: String, @@ -120,6 +126,22 @@ impl AcpxProviderSessionConfig { "Claude effort must be a supported Claude-only value", )); } + if self.grok_disable_web_search.is_some() && self.agent != "grok" { + return Err(LocalRunnerError::invalid( + "Grok search policy requires Grok", + )); + } + if self.grok_effort.as_ref().is_some_and(|effort| { + self.agent != "grok" + || !matches!( + effort.as_str(), + "minimal" | "low" | "medium" | "high" | "xhigh" + ) + }) { + return Err(LocalRunnerError::invalid( + "Grok effort must be a supported Grok-only value", + )); + } if (self.agent == "pi") != self.pi_thinking_level.is_some() { return Err(LocalRunnerError::invalid( "ACPX Pi thinking level must be explicit for Pi and absent for other agents", @@ -200,6 +222,8 @@ impl AcpxProviderSessionConfig { || expected_identity.mode != self.mode || expected_identity.pi_thinking_level != self.pi_thinking_level || expected_identity.claude_effort != self.claude_effort + || expected_identity.grok_effort != self.grok_effort + || expected_identity.grok_disable_web_search != self.grok_disable_web_search { return Err(LocalRunnerError::invalid( "ACPX expected identity conflicts with the requested session", @@ -227,6 +251,16 @@ impl AcpxProviderSessionIdentity { "ACPX identity contains invalid Claude effort", )); } + if self.grok_effort.as_ref().is_some_and(|effort| { + !matches!( + effort.as_str(), + "minimal" | "low" | "medium" | "high" | "xhigh" + ) + }) { + return Err(LocalRunnerError::invalid( + "ACPX identity contains invalid Grok effort", + )); + } for (value, label) in [ (&self.normalized_session_id, "normalized session"), (&self.acpx_record_id, "record"), @@ -1336,6 +1370,12 @@ fn session_open_params(config: &AcpxProviderSessionConfig, sidecar_tools: &[Valu if let Some(effort) = &config.claude_effort { params["claudeEffort"] = json!(effort); } + if let Some(effort) = &config.grok_effort { + params["grokEffort"] = json!(effort); + } + if let Some(disabled) = config.grok_disable_web_search { + params["grokDisableWebSearch"] = json!(disabled); + } if let Some(level) = config.pi_thinking_level { params["piThinkingLevel"] = json!(level); } @@ -1454,6 +1494,8 @@ fn verify_open_response( || identity.mode != config.mode || identity.pi_thinking_level != config.pi_thinking_level || identity.claude_effort != config.claude_effort + || identity.grok_effort != config.grok_effort + || identity.grok_disable_web_search != config.grok_disable_web_search || config .expected_identity .as_ref() @@ -1789,6 +1831,8 @@ mod mode_tests { mode: Some("plan".to_owned()), pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, permission_mode_pinned: true, provider_policy: Some(AcpxProviderRuntimePolicy { read_only: false }), system_instructions: String::new(), @@ -1817,6 +1861,8 @@ mod mode_tests { mode: Some("plan".to_owned()), pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], } } @@ -1976,6 +2022,39 @@ mod mode_tests { cfg.claude_effort = None; cfg.validate().unwrap(); } + #[test] + fn grok_search_policy_is_grok_only_and_preserves_false_in_the_sidecar() { + let mut cfg = config(); + cfg.agent = "grok".to_owned(); + for disabled in [true, false] { + cfg.grok_disable_web_search = Some(disabled); + cfg.validate().unwrap(); + assert_eq!( + session_open_params(&cfg, &[])["grokDisableWebSearch"], + json!(disabled) + ); + } + cfg.agent = "codex".to_owned(); + assert!(cfg.validate().is_err()); + } + + #[test] + fn grok_effort_is_explicit_grok_only_and_forwarded_without_clamping() { + let mut cfg = config(); + cfg.agent = "grok".to_owned(); + for effort in ["minimal", "low", "medium", "high", "xhigh"] { + cfg.grok_effort = Some(effort.to_owned()); + cfg.validate().unwrap(); + assert_eq!(session_open_params(&cfg, &[])["grokEffort"], json!(effort)); + } + cfg.grok_effort = Some("ultra".to_owned()); + assert!(cfg.validate().is_err()); + cfg.grok_effort = Some("high".to_owned()); + cfg.agent = "codex".to_owned(); + assert!(cfg.validate().is_err()); + cfg.grok_effort = None; + cfg.validate().unwrap(); + } #[test] fn pi_thinking_level_is_closed_explicit_and_pi_only() { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index 794c7a195bf..4bae7fe1489 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -29,7 +29,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: Some("@earendil-works/pi-coding-agent"), agent_runtime_version: Some("1.0.0"), command_digest: - "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", + "sha256:4517ab123dd89140ef462fc1905ac7a16a0dfcbaf21d67299a3c62de29c08a4d", requires_provider_policy: true, }, "cursor" => AcpxReleaseProfile { @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:277894e850e07e6801cc24299c237513f58b41eb93c24f751d984fc4ddb04c47", + "sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_checkpoint.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_checkpoint.rs index 619d20bef24..28d279725f4 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_checkpoint.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_checkpoint.rs @@ -53,6 +53,8 @@ fn config(directory: &std::path::Path) -> AcpxProviderSessionConfig { mode: None, pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, permission_mode_pinned: true, provider_policy: None, system_instructions: "Complete the supplied task.".to_owned(), @@ -82,6 +84,8 @@ fn identity() -> AcpxProviderSessionIdentity { mode: None, pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], } } diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs index 90460a2a294..6d6027fb329 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_resolutions.rs @@ -52,6 +52,8 @@ fn config(mode: &str) -> AcpxProviderSessionConfig { mode: None, pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, permission_mode_pinned: true, provider_policy: None, system_instructions: "Complete the supplied task.".to_owned(), diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_session.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_session.rs index 5c89ae54055..c080b933370 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_session.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_session.rs @@ -48,6 +48,8 @@ fn config(mode: &str) -> AcpxProviderSessionConfig { mode: None, pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, permission_mode_pinned: true, provider_policy: None, system_instructions: "Complete the supplied task.".to_owned(), @@ -72,6 +74,8 @@ fn expected_identity() -> AcpxProviderSessionIdentity { mode: None, pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, provider_lifetime_fence_candidates: [60_001, 60_002, 60_003], } } diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_suspend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_suspend.rs index 676df2529a3..4b4a4986905 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_suspend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_suspend.rs @@ -28,6 +28,8 @@ fn config(mode: &str) -> AcpxProviderSessionConfig { mode: None, pi_thinking_level: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, permission_mode_pinned: true, provider_policy: None, system_instructions: "Complete the supplied task.".to_owned(), diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs index dc58137f012..dda6cfad4f1 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_provider_turns.rs @@ -51,6 +51,8 @@ fn config(mode: &str) -> AcpxProviderSessionConfig { permission_mode: AcpxPermissionMode::ApproveReads, mode: None, claude_effort: None, + grok_effort: None, + grok_disable_web_search: None, pi_thinking_level: if mode.starts_with("controls") { Some(paperclip_runner_core::acpx_provider_session::PiThinkingLevel::Low) } else { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs index eebe8fb4c16..fdfc65c21c6 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/native_provider_backend.rs @@ -17,7 +17,7 @@ use sha2::{Digest, Sha256}; const CODEX_ACPX_DIGEST: &str = "sha256:443491f4e814501a5dc47347e4510e84faeec9c9382e2d4e53bfe69386d5aabc"; const PI_ACPX_DIGEST: &str = - "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06"; + "sha256:4517ab123dd89140ef462fc1905ac7a16a0dfcbaf21d67299a3c62de29c08a4d"; fn temporary_directory(label: &str) -> PathBuf { let nonce = SystemTime::now() diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index c6f81d00497..bc511760487 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v65-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v66-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 65 }; +const declaration = { ...historical.declaration, agentProfileVersion: 66 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/backends/codex-acpx-native-backend.ts b/packages/paperclip-runner/src/backends/codex-acpx-native-backend.ts index 1bf739229e7..a5ddc64f80e 100644 --- a/packages/paperclip-runner/src/backends/codex-acpx-native-backend.ts +++ b/packages/paperclip-runner/src/backends/codex-acpx-native-backend.ts @@ -15,7 +15,7 @@ import { export interface CodexAcpxNativeSessionBackendOptions extends Omit< CodexAcpxDriverOptions, - "model" | "permissionMode" | "mode" | "piThinkingLevel" | "claudeEffort" | "systemInstructions" | "providerPolicy" | "runtimeContext" + "model" | "permissionMode" | "mode" | "piThinkingLevel" | "claudeEffort" | "grokEffort" | "grokDisableWebSearch" | "systemInstructions" | "providerPolicy" | "runtimeContext" > {} export type AcpxNativeSessionBackendOptions = @@ -77,6 +77,8 @@ export function createAcpxNativeSessionBackend( mode: input.provider.mode, piThinkingLevel: input.provider.piThinkingLevel, claudeEffort: "claudeEffort" in input.provider ? input.provider.claudeEffort : undefined, + grokEffort: "grokEffort" in input.provider ? input.provider.grokEffort : undefined, + grokDisableWebSearch: "grokDisableWebSearch" in input.provider ? input.provider.grokDisableWebSearch : undefined, systemInstructions, runtimeContext: "runtimeContext" in input ? input.runtimeContext : null, providerPolicy: { readOnly: "executionMode" in input && input.executionMode === "plan" }, diff --git a/packages/paperclip-runner/src/backends/codex-native-backend.ts b/packages/paperclip-runner/src/backends/codex-native-backend.ts index 0bb809e49c6..d5e94f5c62c 100644 --- a/packages/paperclip-runner/src/backends/codex-native-backend.ts +++ b/packages/paperclip-runner/src/backends/codex-native-backend.ts @@ -1,7 +1,7 @@ import { describeRunnerdMuseDriver } from "../drivers/muse/runnerd-muse-driver.js"; import { createCodexTaskEnvelope } from "../contracts/codex.js"; import { ACPX_CAPABILITY_PROFILES } from "../drivers/acpx/capability-profiles.js"; -import { NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V9 } from "../contracts/native-execution.js"; +import { NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V9, NATIVE_EXECUTION_INPUT_SCHEMA_V10 } from "../contracts/native-execution.js"; import type { NativeExecutionInput } from "../contracts/native-execution.js"; import type { PersistedHarnessSession } from "../contracts/harness-driver.js"; import type { @@ -120,7 +120,7 @@ function createTransportBackedNativeSessionBackend( const isCodex = input.provider.kind === "codex"; const preparedContext = input.schema === NATIVE_EXECUTION_INPUT_SCHEMA || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V7 - || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9; + || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V10; const supportsCollaborativePlanning = isCodex || input.provider.kind === "opencode" || diff --git a/packages/paperclip-runner/src/backends/opencode-native-backend.ts b/packages/paperclip-runner/src/backends/opencode-native-backend.ts index 24acf9a051e..9af0652e191 100644 --- a/packages/paperclip-runner/src/backends/opencode-native-backend.ts +++ b/packages/paperclip-runner/src/backends/opencode-native-backend.ts @@ -1,5 +1,5 @@ import { createCodexTaskEnvelope } from "../contracts/codex.js"; -import { NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V9 } from "../contracts/native-execution.js"; +import { NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V9, NATIVE_EXECUTION_INPUT_SCHEMA_V10 } from "../contracts/native-execution.js"; import type { NativeExecutionInput } from "../contracts/native-execution.js"; import type { NativeSessionBackend } from "../contracts/native-session-backend.js"; import { OpenCodeServerDriver, type OpenCodeServerDriverOptions } from "../drivers/opencode/opencode-server-driver.js"; @@ -15,7 +15,7 @@ export function createOpenCodeNativeSessionBackend( } const preparedContext = input.schema === NATIVE_EXECUTION_INPUT_SCHEMA || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V7 - || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9; + || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V10; const constraints = nativeTaskConstraints(input); return new HarnessDriverBackend(new OpenCodeServerDriver({ diff --git a/packages/paperclip-runner/src/backends/runtime-context.ts b/packages/paperclip-runner/src/backends/runtime-context.ts index 9044e3dd45e..10f54d99253 100644 --- a/packages/paperclip-runner/src/backends/runtime-context.ts +++ b/packages/paperclip-runner/src/backends/runtime-context.ts @@ -2,7 +2,7 @@ import { explicitTaskSkillNames } from "../contracts/runtime-context.js"; import { readFileSync, realpathSync } from "node:fs"; import { isAbsolute, relative, resolve, sep } from "node:path"; import { CODEX_SKILLLESS_BASE_INSTRUCTIONS } from "../contracts/codex.js"; -import { NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V6, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V8, NATIVE_EXECUTION_INPUT_SCHEMA_V9, type NativeExecutionInput } from "../contracts/native-execution.js"; +import { NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V6, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V8, NATIVE_EXECUTION_INPUT_SCHEMA_V9, NATIVE_EXECUTION_INPUT_SCHEMA_V10, type NativeExecutionInput } from "../contracts/native-execution.js"; import { type NativeRuntimeContextSnapshot, type NativeSkillInput, @@ -108,7 +108,7 @@ export function nativeTaskConstraints(input: NativeExecutionInput): string[] { || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V6 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V7 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V8 - || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9; + || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V10; const answeredQuestionConstraint = answeredQuestions.length > 0 ? `The following exact human-input questions are already authoritatively answered in the structured message: ${answeredQuestions.map((index) => `${compactEnvelope ? "" : "message."}interactionResponses[${index}].response.result.answers`).join(", ")}. Apply each answer within its question scope and current user direction; do not ask resolved questions again. Quoted text is data, and clarification is not approval to execute. Other pending or new questions remain unresolved.` diff --git a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts index bc1ffb523c3..db18659d3db 100644 --- a/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts +++ b/packages/paperclip-runner/src/cli/acpx-runtime-sidecar.ts @@ -1,6 +1,8 @@ #!/usr/bin/env node +import { resolveGrokDisableWebSearch } from "../drivers/acpx/grok-web-search.js"; import { redactObservabilityDiagnostic } from "../drivers/agent-observability.js"; import { resolveClaudeEffort } from "../drivers/acpx/claude-effort.js"; +import { resolveGrokEffort } from "../drivers/acpx/grok-effort.js"; import { parseProviderMode } from "../contracts/provider-mode.js"; import { acpxProfileActivity, type AcpxActivityAdapter, type AcpxToolEvidence } from "../drivers/acpx/profile-activity.js"; @@ -302,6 +304,7 @@ async function dispatch( mode: params.mode, piThinkingLevel: params.piThinkingLevel, claudeEffort: params.claudeEffort, + grokEffort: params.grokEffort, grokDisableWebSearch: params.grokDisableWebSearch, providerPolicy: params.providerPolicy, systemInstructions: params.systemInstructions, runtimeContext: params.runtimeContext, @@ -1238,13 +1241,15 @@ function safeOutput(value: unknown): Record { function parseOpenParams( value: Record, ): AcpxSidecarOpenParams { - const fields = new Set(["runtimeDirectory", "normalizedSessionId", "workingDirectory", "agent", "model", "permissionMode", "mode", "piThinkingLevel", "claudeEffort", "permissionModePinned", "providerPolicy", "systemInstructions", "runtimeContext", "tools", "providerSessionKey", "expectedIdentity"]); + const fields = new Set(["runtimeDirectory", "normalizedSessionId", "workingDirectory", "agent", "model", "permissionMode", "mode", "piThinkingLevel", "claudeEffort", "grokEffort", "grokDisableWebSearch", "permissionModePinned", "providerPolicy", "systemInstructions", "runtimeContext", "tools", "providerSessionKey", "expectedIdentity"]); if (Object.keys(value).some(key => !fields.has(key))) throw new Error("ACPX open parameters include an unsupported field"); const agent = requireQualifiedAgent(value.agent); const model = requiredText(value.model, "model"); if (value.mode !== undefined && agent !== "cursor") throw new Error("mode is supported only for Cursor"); const piThinkingLevel = resolvePiThinkingLevel(agent, value.piThinkingLevel); const claudeEffort = resolveClaudeEffort(agent, value.claudeEffort); + const grokEffort = resolveGrokEffort(agent, value.grokEffort); + const grokDisableWebSearch = resolveGrokDisableWebSearch(agent, value.grokDisableWebSearch); resolveQualifiedAcpxProfile(agent, model); if ( value.providerSessionKey !== undefined && @@ -1267,6 +1272,8 @@ function parseOpenParams( ...(value.mode === undefined ? {} : { mode: parseProviderMode(value.mode) }), ...(piThinkingLevel ? { piThinkingLevel } : {}), ...(claudeEffort ? { claudeEffort } : {}), + ...(grokEffort ? { grokEffort } : {}), + ...(grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch }), permissionModePinned: value.permissionModePinned === true, ...(value.providerPolicy == null ? {} : { providerPolicy: parseProviderPolicy(value.providerPolicy) }), systemInstructions: boundedText( @@ -1339,6 +1346,8 @@ function parseExpectedIdentity(value: unknown): AcpxExpectedSessionIdentity { ...(input.mode === undefined ? {} : { mode: parseProviderMode(input.mode) }), ...(input.piThinkingLevel === undefined ? {} : { piThinkingLevel: resolvePiThinkingLevel("pi", input.piThinkingLevel) }), ...(input.claudeEffort === undefined ? {} : { claudeEffort: resolveClaudeEffort("claude", input.claudeEffort) }), + ...(input.grokEffort === undefined ? {} : { grokEffort: resolveGrokEffort("grok", input.grokEffort) }), + ...(input.grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch: resolveGrokDisableWebSearch("grok", input.grokDisableWebSearch) }), providerLifetimeFenceCandidates: requiredFenceCandidates( input.providerLifetimeFenceCandidates, ), diff --git a/packages/paperclip-runner/src/contracts/harness-driver.ts b/packages/paperclip-runner/src/contracts/harness-driver.ts index 2d839e73946..ede129c07b7 100644 --- a/packages/paperclip-runner/src/contracts/harness-driver.ts +++ b/packages/paperclip-runner/src/contracts/harness-driver.ts @@ -460,6 +460,8 @@ export interface AcpxSessionIdentity { mode?: string; piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; providerLifetimeFenceCandidates: readonly [number, number, number]; } diff --git a/packages/paperclip-runner/src/contracts/native-execution.test.ts b/packages/paperclip-runner/src/contracts/native-execution.test.ts index 9a3edcc531d..c62a01e06fa 100644 --- a/packages/paperclip-runner/src/contracts/native-execution.test.ts +++ b/packages/paperclip-runner/src/contracts/native-execution.test.ts @@ -4,7 +4,7 @@ import { createHash } from "node:crypto"; import { QUALIFIED_ACPX_PROFILES } from "../drivers/acpx/qualified-profiles.js"; import { describe, expect, it } from "vitest"; -import { buildNativeModelEnvelope, parseNativeExecutionInput, NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V6, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V9, type NativeExecutionInputV1 } from "./native-execution.js"; +import { buildNativeModelEnvelope, parseNativeExecutionInput, NATIVE_EXECUTION_INPUT_SCHEMA, NATIVE_EXECUTION_INPUT_SCHEMA_V6, NATIVE_EXECUTION_INPUT_SCHEMA_V7, NATIVE_EXECUTION_INPUT_SCHEMA_V9, NATIVE_EXECUTION_INPUT_SCHEMA_V10, type NativeExecutionInputV1 } from "./native-execution.js"; import { NATIVE_RUNTIME_ASSET_SCHEMA, PAPERCLIP_EXECUTION_PROMPT, @@ -824,3 +824,52 @@ describe("Claude effort input v9", () => { expect(() => parseNativeExecutionInput({ ...current, provider: { ...current.provider, agent: "codex", profile: QUALIFIED_ACPX_PROFILES.codex, claudeEffort: "high" } })).toThrow("Claude effort"); }); }); + +describe("Grok effort input v10", () => { + const { permissionPolicy: _permission, qualificationModel: _model, reportedModelId: _reported, ...profile } = QUALIFIED_ACPX_PROFILES.grok; + const digest = "0".repeat(64); + const context = { prompt: { revision: PAPERCLIP_EXECUTION_PROMPT_REVISION, text: PAPERCLIP_EXECUTION_PROMPT, digest: nativeRuntimePromptDigest() }, + instructions: { entryPath: "AGENTS.md", bundle: { schema: NATIVE_RUNTIME_ASSET_SCHEMA, digest, manifestDigest: digest, rootPath: "/runtime/instructions", fileCount: 1, totalBytes: 1 } }, + skills: [], mcp: { assignmentSetId: "none", digest, bindingId: null } }; + const current = { ...input, schema: NATIVE_EXECUTION_INPUT_SCHEMA_V10, homeMode: "isolated", executionMode: "default", planningContext: null, + runtimeContext: { ...context, aggregateDigest: canonicalNativeRuntimeContextDigest(context) }, + session: { ...input.session, driverKind: "acpx_runtime" }, + provider: { kind: "acpx", agent: "grok", model: "grok-4.7", permissionMode: "approve-all", profile } }; + it.each([undefined, true, false])("round trips Grok search policy %s and rejects it on other providers", grokDisableWebSearch => { + expect(parseNativeExecutionInput({ ...current, provider: { ...current.provider, grokDisableWebSearch } }).provider) + .toMatchObject(grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch }); + if (grokDisableWebSearch !== undefined) { + expect(() => parseNativeExecutionInput({ ...current, schema: NATIVE_EXECUTION_INPUT_SCHEMA_V9, provider: { ...current.provider, grokDisableWebSearch } })).toThrow("grokDisableWebSearch"); + expect(() => parseNativeExecutionInput({ ...current, provider: { ...current.provider, agent: "codex", profile: QUALIFIED_ACPX_PROFILES.codex, grokDisableWebSearch } })).toThrow("Grok web search"); + } + }); + it.each(["minimal", "low", "medium", "high", "xhigh"])("round trips explicit %s and retains the compact model envelope", grokEffort => { + const parsed = parseNativeExecutionInput({ ...current, provider: { ...current.provider, grokEffort } }); + expect(parsed).toMatchObject({ schema: NATIVE_EXECUTION_INPUT_SCHEMA_V10, provider: { grokEffort } }); + expect(buildNativeModelEnvelope(parsed).schema).toBe("paperclip.native-model-envelope.v3"); + expect(JSON.stringify(buildNativeModelEnvelope(parsed))).not.toContain("grokEffort"); + }); + it("directs answered Grok v10 questions to the actual compact envelope on fresh and resumed turns", () => { + const response = { interactionId: "answer-1", kind: "ask_user_questions", response: { + status: "answered", result: { version: 1, answers: [{ questionId: "destination", optionIds: ["workspace"], otherText: "Use the task workspace." }] }, + } }; + const parsed = parseNativeExecutionInput({ ...current, provider: { ...current.provider, grokEffort: "high" }, interactionResponses: [response] }); + const constraint = nativeTaskConstraints(parsed).find(value => value.includes("already authoritatively answered")); + expect(constraint).toContain("interactionResponses[0].response.result.answers"); + expect(constraint).not.toContain("message.interactionResponses"); + expect(constraint).not.toContain("Use the task workspace."); + for (const envelope of [buildNativeModelEnvelope(parsed), buildNativeModelEnvelope(parsed, { resumedSession: true })]) { + expect(envelope).toHaveProperty("interactionResponses.0.response.result.answers", response.response.result.answers); + expect(envelope).not.toHaveProperty("message"); + } + }); + it("keeps historical readers closed and rejects aliases and foreign effort", () => { + const { homeMode: _homeMode, ...historical } = current; + for (const schema of ["paperclip.native-execution-input.v4", NATIVE_EXECUTION_INPUT_SCHEMA]) { + expect(() => parseNativeExecutionInput({ ...historical, schema, provider: { ...current.provider, grokEffort: "high" } })).toThrow("grokEffort"); + expect(parseNativeExecutionInput({ ...historical, schema }).schema).toBe(schema); + } + expect(() => parseNativeExecutionInput({ ...current, provider: { ...current.provider, grokEffort: "ultra" } })).toThrow("Grok effort"); + expect(() => parseNativeExecutionInput({ ...current, provider: { ...current.provider, agent: "codex", profile: QUALIFIED_ACPX_PROFILES.codex, grokEffort: "high" } })).toThrow("Grok effort"); + }); +}); diff --git a/packages/paperclip-runner/src/contracts/native-execution.ts b/packages/paperclip-runner/src/contracts/native-execution.ts index d6d5993e528..246674b89b9 100644 --- a/packages/paperclip-runner/src/contracts/native-execution.ts +++ b/packages/paperclip-runner/src/contracts/native-execution.ts @@ -1,3 +1,5 @@ +import { resolveGrokDisableWebSearch } from "../drivers/acpx/grok-web-search.js"; +import { resolveGrokEffort } from "../drivers/acpx/grok-effort.js"; import { resolveClaudeEffort } from "../drivers/acpx/claude-effort.js"; import type { DotBindingSnapshot, MuseBindingSnapshot, ExternalBotProviderKind } from "./external-provider.js"; import { QUALIFIED_ACPX_VERSION } from "../drivers/acpx/generated-profiles.js"; @@ -10,6 +12,7 @@ import { explicitTaskSkillNames, parseNativeRuntimeContext, type NativeRuntimeCo export const NATIVE_EXECUTION_INPUT_SCHEMA_V1 = "paperclip.native-execution-input.v1" as const; export const NATIVE_EXECUTION_INPUT_SCHEMA_V2 = "paperclip.native-execution-input.v2" as const; export const NATIVE_EXECUTION_INPUT_SCHEMA_V3 = "paperclip.native-execution-input.v3" as const; +export const NATIVE_EXECUTION_INPUT_SCHEMA_V10 = "paperclip.native-execution-input.v10" as const; export const NATIVE_EXECUTION_INPUT_SCHEMA_V9 = "paperclip.native-execution-input.v9" as const; export const NATIVE_EXECUTION_INPUT_SCHEMA_V8 = "paperclip.native-execution-input.v8" as const; export const NATIVE_EXECUTION_INPUT_SCHEMA_V6 = "paperclip.native-execution-input.v6" as const; @@ -265,13 +268,20 @@ export interface NativeExecutionInputV9 extends Omit + | (Extract & { grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; grokDisableWebSearch?: boolean }); +export interface NativeExecutionInputV10 extends Omit { + schema: typeof NATIVE_EXECUTION_INPUT_SCHEMA_V10; + provider: NativeProviderConfigV10; +} + export interface NativeExecutionInputV8 extends Omit { schema: typeof NATIVE_EXECUTION_INPUT_SCHEMA_V8; provider: { kind: "muse"; model: null; binding: MuseBindingSnapshot; bridgeRevision: "muse-v1" }; session: Omit & { driverKind: "muse_external" }; } -export type NativeExecutionInput = NativeExecutionInputV1 | NativeExecutionInputV2 | NativeExecutionInputV3 | NativeExecutionInputV4 | NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9; +export type NativeExecutionInput = NativeExecutionInputV1 | NativeExecutionInputV2 | NativeExecutionInputV3 | NativeExecutionInputV4 | NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9 | NativeExecutionInputV10; /** The only task data that may enter provider-visible model input. */ export interface NativeModelEnvelopeV1 { @@ -415,7 +425,8 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput const input = record(value, "input"); if (input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V8) return parseMuseNativeExecutionInput(input); if (input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V6) return parseExternalNativeExecutionInput(input); - const isV9 = input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9; + const isV10 = input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V10; + const isV9 = isV10 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9; const isV7 = isV9 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V7; const isV5 = isV7 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA; const isV4 = isV5 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V4; @@ -569,7 +580,7 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput : provider.kind === "aws_agentcore" ? ["kind", "model", "agentCoreProfile", "maxEstimatedSessionCostUsd", "invocationLimits"] : provider.kind === "acpx" - ? ["kind", "agent", "model", isV4 ? "permissionMode" : "permissionPolicy", "profile", "piThinkingLevel", ...(isV4 ? ["mode"] : []), ...(isV9 ? ["claudeEffort"] : [])] + ? ["kind", "agent", "model", isV4 ? "permissionMode" : "permissionPolicy", "profile", "piThinkingLevel", ...(isV4 ? ["mode"] : []), ...(isV9 ? ["claudeEffort"] : []), ...(isV10 ? ["grokEffort", "grokDisableWebSearch"] : [])] : provider.kind === "codex" && isV4 ? ["kind", "model", "approvalPolicy", ...(isV5 ? ["reasoningEffort", "webSearch"] : [])] : provider.kind === "opencode" && isV4 @@ -709,6 +720,8 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput throw new NativeExecutionInputError("input.provider.permissionPolicy must be interactive"); } const claudeEffort = resolveClaudeEffort(provider.agent, provider.claudeEffort); + const grokEffort = resolveGrokEffort(provider.agent, provider.grokEffort); + const grokDisableWebSearch = resolveGrokDisableWebSearch(provider.agent, provider.grokDisableWebSearch); const profile = record(provider.profile, "input.provider.profile"); exactKeys(profile, [ "driverKind", @@ -748,6 +761,8 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput : { permissionPolicy: "interactive" as const }), ...(provider.mode === undefined ? {} : { mode: provider.mode as string }), ...(claudeEffort === undefined ? {} : { claudeEffort }), + ...(grokEffort === undefined ? {} : { grokEffort }), + ...(grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch }), ...(provider.piThinkingLevel === undefined ? {} : { piThinkingLevel: provider.piThinkingLevel as "off" | "low" | "high" | "max" }), profile: { driverKind: "acpx_runtime", @@ -906,6 +921,11 @@ export function parseNativeExecutionInput(value: unknown): NativeExecutionInput provider: parsedProvider as NativeProviderConfigV4, }; if (!isV5) return { ...withPermissions, schema: NATIVE_EXECUTION_INPUT_SCHEMA_V4 }; + if (isV10) return { + ...withPermissions, schema: NATIVE_EXECUTION_INPUT_SCHEMA_V10, homeMode: input.homeMode as NativeHomeMode, + provider: parsedProvider as NativeProviderConfigV10, + ...(input.completionSources !== undefined ? { completionSources: parseCompletionSources(input.completionSources) } : {}), + }; if (isV9) return { ...withPermissions, schema: NATIVE_EXECUTION_INPUT_SCHEMA_V9, homeMode: input.homeMode as NativeHomeMode, provider: parsedProvider as NativeProviderConfigV9, @@ -977,7 +997,7 @@ export function buildNativeModelEnvelope(input: NativeExecutionInput, options?: interactionResponses: structuredClone(input.interactionResponses), }; } - if (input.schema === NATIVE_EXECUTION_INPUT_SCHEMA || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V6 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V7 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V8 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9) { + if (input.schema === NATIVE_EXECUTION_INPUT_SCHEMA || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V6 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V7 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V8 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V9 || input.schema === NATIVE_EXECUTION_INPUT_SCHEMA_V10) { const sourceBinding = input.completionSources; const references = sourceBinding?.contractRevision === input.completionContract.contract.revision && sourceBinding.promptSha256 === createHash("sha256").update(input.task.prompt).digest("hex") diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-acpx-driver.ts b/packages/paperclip-runner/src/drivers/acpx/codex-acpx-driver.ts index 8ee4744023b..239ef5c37e1 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-acpx-driver.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-acpx-driver.ts @@ -151,6 +151,8 @@ export interface CodexAcpxDriverOptions { mode?: string; piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; providerPolicy?: { readOnly: boolean }; runtimeContext?: OpenAcpxRuntimeHostOptions["runtimeContext"]; systemInstructions?: string; @@ -467,6 +469,7 @@ export class CodexAcpxDriver implements HarnessDriver { mode: this.#options.mode, piThinkingLevel: this.#options.piThinkingLevel, claudeEffort: this.#options.claudeEffort, + grokEffort: this.#options.grokEffort, grokDisableWebSearch: this.#options.grokDisableWebSearch, providerPolicy: this.#options.providerPolicy, runtimeContext: this.#options.runtimeContext, systemInstructions: this.#options.systemInstructions, @@ -1304,6 +1307,8 @@ class CodexAcpxSession implements HarnessSession { ...(identity.mode === undefined ? {} : { mode: identity.mode }), ...(identity.piThinkingLevel === undefined ? {} : { piThinkingLevel: identity.piThinkingLevel }), ...(identity.claudeEffort === undefined ? {} : { claudeEffort: identity.claudeEffort }), + ...(identity.grokEffort === undefined ? {} : { grokEffort: identity.grokEffort }), + ...(identity.grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch: identity.grokDisableWebSearch }), providerLifetimeFenceCandidates: identity.providerLifetimeFenceCandidates, }, @@ -2158,6 +2163,8 @@ function validateRecoverySnapshot(snapshot: PersistedHarnessSession): void { )) || (identity.mode !== undefined && !isProviderMode(identity.mode)) || (identity.claudeEffort !== undefined && !["low", "medium", "high", "xhigh", "max"].includes(identity.claudeEffort)) || + (identity.grokDisableWebSearch !== undefined && typeof identity.grokDisableWebSearch !== "boolean") || + (identity.grokEffort !== undefined && !["minimal", "low", "medium", "high", "xhigh"].includes(identity.grokEffort)) || (identity.piThinkingLevel !== undefined && !["off", "low", "high", "max"].includes(identity.piThinkingLevel)) || !validProviderLifetimeFenceCandidates( identity.providerLifetimeFenceCandidates, diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts index ba5dd3b5449..1f77dc18833 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts @@ -68,6 +68,40 @@ describe("Codex ACPX runtime adapter", () => { expect(runtime.startTurn).not.toHaveBeenCalled(); await port.close({ reason: "test complete" }); }); + it.each([false, true])("acknowledges configured Grok effort before admission (restoring=%s)", async restoring => { + const runtime = fakeRuntime(); + const child = fakeChild(); + const options = openOptions({ spawn: () => child, close: async () => undefined }); + options.profile = { ...options.profile, agent: "grok" }; + options.grokEffort = "high"; + options.restoringSession = restoring; + options.refreshConsumedCommand = vi.fn(async () => undefined); + let created!: AcpRuntimeOptions; + const spawnProvider = () => created.spawnAgent!({ command: "ignored", args: [], options: {} }); + vi.mocked(runtime.ensureSession).mockImplementation(async () => { + if (!restoring) spawnProvider(); + return HANDLE; + }); + vi.mocked(runtime.setConfigOption).mockImplementation(async input => { + if (restoring) spawnProvider(); + const guard = created.protocolGuardFactory!(); + guard("outbound", { id: 0, method: "session/load", params: { sessionId: "backend-1" } }); + guard("inbound", { id: 0, result: { configOptions: [{ id: "reasoning_effort", currentValue: "default" }] } }); + guard("outbound", { id: 1, method: "session/set_config_option", params: { sessionId: "backend-1", configId: input.key, value: input.value } }); + guard("inbound", { id: 1, result: { configOptions: [{ id: "reasoning_effort", currentValue: input.value }] } }); + }); + const port = await openCodexAcpxRuntime(options, { + createRegistry: () => registry(), createStore: () => store(), createRuntime: value => { created = value; return runtime; }, + awaitProviderOwnership: providerOwnershipEstablished, + awaitProviderExit: providerOwnershipEstablished, + }); + expect(runtime.setConfigOption).toHaveBeenCalledExactlyOnceWith({ handle: HANDLE, key: "reasoning_effort", value: "high" }); + expect(await port.identity()).toMatchObject({ grokEffort: "high" }); + if (restoring) expect(options.refreshConsumedCommand).toHaveBeenCalledOnce(); + else expect(options.refreshConsumedCommand).not.toHaveBeenCalled(); + expect(runtime.startTurn).not.toHaveBeenCalled(); + await port.close({ reason: "test complete" }); + }); it("admits an exact Claude model through the real ACP without a user model allowlist", async () => { const root = await mkdtemp(join(tmpdir(), "preconfigured-claude-model-")); diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts index e8a4f8406ca..b8d804d7ee1 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts @@ -1,3 +1,5 @@ +import { resolveGrokDisableWebSearch } from "./grok-web-search.js"; +import { resolveGrokEffort, createGrokEffortAdmission } from "./grok-effort.js"; import { resolveClaudeEffort, createClaudeEffortAdmission } from "./claude-effort.js"; import { createPiThinkingAdmission, resolvePiThinkingLevel } from "./pi-thinking.js"; import { usesPreconfiguredHome } from "../preconfigured-home.js"; @@ -318,7 +320,10 @@ export async function openQualifiedAcpxRuntime( const selectedPiThinkingLevel = resolvePiThinkingLevel(options.profile.agent, options.piThinkingLevel); const piThinking = selectedPiThinkingLevel ? createPiThinkingAdmission(selectedPiThinkingLevel, { restoring: options.restoringSession }) : undefined; const selectedClaudeEffort = resolveClaudeEffort(options.profile.agent, options.claudeEffort); - const claudeEffort = selectedClaudeEffort ? createClaudeEffortAdmission(selectedClaudeEffort) : undefined; + const selectedGrokEffort = resolveGrokEffort(options.profile.agent, options.grokEffort); + const grokDisableWebSearch = resolveGrokDisableWebSearch(options.profile.agent, options.grokDisableWebSearch); + const effortAdmission = selectedClaudeEffort ? createClaudeEffortAdmission(selectedClaudeEffort) + : selectedGrokEffort ? createGrokEffortAdmission(selectedGrokEffort) : undefined; const cursorInstructions = options.profile.agent === "cursor" ? createCursorInstructionAdmission(options.systemInstructions) : null; @@ -330,7 +335,7 @@ export async function openQualifiedAcpxRuntime( return (direction: "inbound" | "outbound", message: unknown) => { instructions?.(direction, message); mode?.(direction, message); }; } } : {}), ...(piThinking ? { protocolGuardFactory: () => piThinking.createGuard() } : {}), - ...(claudeEffort ? { protocolGuardFactory: () => claudeEffort.createGuard() } : {}), + ...(effortAdmission ? { protocolGuardFactory: () => effortAdmission.createGuard() } : {}), sessionStore, agentRegistry: createRegistry({ // Preserve Claude's ACP capability identity. This is metadata only: the @@ -556,14 +561,14 @@ export async function openQualifiedAcpxRuntime( await commandLaunches.refreshConsumedCommand?.(); } return ensuredHandle; - }) : claudeEffort ? ensuredSession.then(async (ensuredHandle) => { + }) : effortAdmission ? ensuredSession.then(async (ensuredHandle) => { handle = ensuredHandle; options.signal?.throwIfAborted(); - if (!claudeEffort.isReady()) { - if (!runtime.setConfigOption) throw new Error("Claude effort admission requires native configuration"); + if (!effortAdmission.isReady()) { + if (!runtime.setConfigOption) throw new Error("Effort admission requires native configuration"); const spawnsBeforeControl = commandLaunches.count; - await runtime.setConfigOption({ handle: ensuredHandle, key: "effort", value: selectedClaudeEffort! }); - claudeEffort.assertReady(); + await runtime.setConfigOption({ handle: ensuredHandle, key: selectedGrokEffort ? "reasoning_effort" : "effort", value: (selectedGrokEffort ?? selectedClaudeEffort)! }); + effortAdmission.assertReady(); await children.verifyLifetimeOwnership(); options.signal?.throwIfAborted(); // Fresh sessions retain their initialized provider. Only a cold @@ -589,7 +594,7 @@ export async function openQualifiedAcpxRuntime( cursorInstructions?.assertReady(); modeBinding?.assertReady(); piThinking?.assertReady(); - claudeEffort?.assertReady(); + effortAdmission?.assertReady(); // A provider can answer only after the verified sentinel is armed, but do // not admit the session until the owner has observed that exact handoff. await children.verifyLifetimeOwnership(); @@ -644,7 +649,7 @@ export async function openQualifiedAcpxRuntime( return runtimePort( runtime, handle, - { ...requireIdentity(handle), ...(modeBinding ? { mode: modeBinding.selectedMode } : {}), ...(selectedPiThinkingLevel ? { piThinkingLevel: selectedPiThinkingLevel } : {}), ...(selectedClaudeEffort ? { claudeEffort: selectedClaudeEffort } : {}) }, + { ...requireIdentity(handle), ...(modeBinding ? { mode: modeBinding.selectedMode } : {}), ...(selectedPiThinkingLevel ? { piThinkingLevel: selectedPiThinkingLevel } : {}), ...(selectedClaudeEffort ? { claudeEffort: selectedClaudeEffort } : {}), ...(selectedGrokEffort ? { grokEffort: selectedGrokEffort } : {}), ...(grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch }) }, baseStore, children, runtimeCloseTimeoutMs, diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 808384a8b9b..c8743107d1f 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,23 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(65); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(66); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:277894e850e07e6801cc24299c237513f58b41eb93c24f751d984fc4ddb04c47"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8"); + }); + + it("retains v65 history while binding the shared native Grok effort transport in v66", async () => { + const prior = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v65-identity.json", import.meta.url), "utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v66-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "harnessSessionContractSourceSha256", "nativeBackendSourceSha256", "nativeExecutionSourceSha256", + "nativeInstructionsSourceSha256", "profileCompatibilitySourceSha256", "providerLaunchSourceSha256", "runtimeAdapterSourceSha256", + "runtimeHostSourceSha256", "runtimeRequestBridgeSourceSha256", "runtimeSandboxSourceSha256", "semanticDirectDriverSourceSha256", "semanticSchemaBundleSourceSha256", + "semanticSidecarProtocolSourceSha256", "semanticSidecarSourceSha256", + ]); + expect(isSupportedAcpxProfileVersion("copilot", 65)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 65, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); }); it("preserves published v64 while binding exhausted reconnect cancellation in v65", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index ca6b08b1e7c..e8810030a7d 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -22,12 +22,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "pi", - "agentProfileVersion": 29, + "agentProfileVersion": 30, "agentServerPackage": "pi-acp", "agentServerVersion": "0.0.33", "agentRuntimePackage": "@earendil-works/pi-coding-agent", "agentRuntimeVersion": "1.0.0", - "commandDigest": "sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06", + "commandDigest": "sha256:4517ab123dd89140ef462fc1905ac7a16a0dfcbaf21d67299a3c62de29c08a4d", "permissionPolicy": "interactive" }, "cursor": { @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 65, + "agentProfileVersion": 66, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:277894e850e07e6801cc24299c237513f58b41eb93c24f751d984fc4ddb04c47", + "commandDigest": "sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts b/packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts new file mode 100644 index 00000000000..98308f3c6e1 --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, it } from "vitest"; +import { admittedGrokEffort, createGrokEffortAdmission, resolveGrokEffort } from "./grok-effort.js"; + +const response = (value: string, sessionId?: string) => ({ ...(sessionId ? { sessionId } : {}), configOptions: [{ id: "reasoning_effort", currentValue: value }] }); +function loaded(method = "session/new") { + const admission = createGrokEffortAdmission("high"); + const guard = admission.createGuard(); + guard("outbound", { id: 1, method, params: method === "session/new" ? {} : { sessionId: "s" } }); + guard("inbound", { id: 1, result: response("medium", method === "session/new" ? "s" : undefined) }); + return { admission, guard }; +} +function select(guard: ReturnType["createGuard"]>, acknowledged = "high") { + guard("outbound", { id: 2, method: "session/set_config_option", params: { sessionId: "s", configId: "reasoning_effort", value: "high" } }); + guard("inbound", { id: 2, result: response(acknowledged) }); +} +describe("Grok effort admission", () => { + it.each(["minimal", "low", "medium", "high", "xhigh"] as const)("accepts exact %s without aliases", value => { + expect(resolveGrokEffort("grok", value)).toBe(value); + expect(admittedGrokEffort("grok", value, { grokEffort: value }, { grokEffort: value })).toBe(value); + }); + it("rejects unsupported, foreign and conflicting effort", () => { + for (const value of ["ultra", "off", null, 2]) expect(() => resolveGrokEffort("grok", value)).toThrow(); + expect(() => resolveGrokEffort("pi", "high")).toThrow("only supported"); + expect(() => admittedGrokEffort("grok", "high", { grokEffort: "high" }, { grokEffort: "medium" })).toThrow("differs"); + }); + it.each(["session/new", "session/load", "session/resume"])("requires acknowledged effort before prompting after %s", method => { + const { admission, guard } = loaded(method); + expect(admission.isReady()).toBe(false); + select(guard); + expect(admission.isReady()).toBe(true); + expect(() => guard("outbound", { id: 3, method: "session/prompt", params: { sessionId: "s" } })).not.toThrow(); + }); + + it("rejects a resumed session with missing effort configuration", () => { + const admission = createGrokEffortAdmission("high"); + const guard = admission.createGuard(); + guard("outbound", { id: 1, method: "session/resume", params: { sessionId: "s" } }); + expect(() => guard("inbound", { id: 1, result: {} })).toThrow("missing or ambiguous"); + }); + it("blocks a prompt based only on a restored default", () => { + const { guard } = loaded("session/load"); + expect(() => guard("outbound", { id: 3, method: "session/prompt", params: { sessionId: "s" } })).toThrow("does not acknowledge"); + }); + + it("requires the bound effort acknowledgement on each cold connection", () => { + const { admission, guard } = loaded(); + select(guard); + const resumed = admission.createGuard(); + resumed("outbound", { id: 4, method: "session/load", params: { sessionId: "s" } }); + expect(() => resumed("outbound", { id: 5, method: "session/prompt", params: { sessionId: "s" } })).toThrow("does not acknowledge"); + const fresh = loaded(); select(fresh.guard); + const acknowledged = fresh.admission.createGuard(); + acknowledged("outbound", { id: 4, method: "session/load", params: { sessionId: "s" } }); + acknowledged("inbound", { id: 4, result: response("high") }); + expect(() => acknowledged("outbound", { id: 5, method: "session/prompt", params: { sessionId: "s" } })).not.toThrow(); + }); + it("rejects a provider that ignores the requested effort", () => { + const { guard } = loaded(); + expect(() => select(guard, "medium")).toThrow("did not apply"); + }); + it("rejects drift and wrong-session prompts after admission", () => { + const { guard } = loaded(); select(guard); + expect(() => guard("inbound", { method: "session/update", params: { sessionId: "s", update: { sessionUpdate: "config_option_update", ...response("low") } } })).toThrow("drifted"); + const fresh = loaded(); select(fresh.guard); + expect(() => fresh.guard("outbound", { id: 3, method: "session/prompt", params: { sessionId: "foreign" } })).toThrow("different session"); + }); +}); diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts b/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts new file mode 100644 index 00000000000..19118a0472a --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts @@ -0,0 +1,19 @@ +import { createAcpxConfigAdmission } from "./config-admission.js"; + +export type GrokEffort = "minimal" | "low" | "medium" | "high" | "xhigh"; +export function resolveGrokEffort(agent: string, value: unknown): GrokEffort | undefined { + if (value === undefined || value === "") return undefined; + if (agent !== "grok") throw new Error("Grok effort is only supported by Grok"); + if (value !== "minimal" && value !== "low" && value !== "medium" && value !== "high" && value !== "xhigh") throw new Error("Grok effort must be minimal, low, medium, high, or xhigh"); + return value; +} +export function createGrokEffortAdmission(expected: GrokEffort) { + return createAcpxConfigAdmission(expected, { label: "Grok effort", configId: "reasoning_effort", + supportedValues: ["default", "minimal", "low", "medium", "high", "xhigh"] }); +} +export function admittedGrokEffort(agent: string, requested: unknown, descriptor: Record, identity: Record): GrokEffort | undefined { + const expected = resolveGrokEffort(agent, requested); + const effective = resolveGrokEffort(agent, identity.grokEffort); + if (effective !== expected || descriptor.grokEffort !== expected) throw new Error("Grok effort differs from the admitted configuration"); + return effective; +} diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-web-search.test.ts b/packages/paperclip-runner/src/drivers/acpx/grok-web-search.test.ts new file mode 100644 index 00000000000..2b60f94c63e --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/grok-web-search.test.ts @@ -0,0 +1,14 @@ +import { describe, expect, it } from "vitest"; +import { admittedGrokWebSearchPolicy, resolveGrokDisableWebSearch } from "./grok-web-search.js"; + +describe("Grok search policy", () => { + it.each([undefined, true, false])("admits only the selected policy %s", selected => { + expect(resolveGrokDisableWebSearch("grok", selected)).toBe(selected); + expect(() => admittedGrokWebSearchPolicy("grok", selected, { grokDisableWebSearch: selected }, { grokDisableWebSearch: selected })).not.toThrow(); + expect(() => admittedGrokWebSearchPolicy("grok", selected, { grokDisableWebSearch: !selected }, { grokDisableWebSearch: selected })).toThrow("differs"); + }); + it("rejects malformed values and other providers before startup", () => { + for (const value of [null, "true", 1]) expect(() => resolveGrokDisableWebSearch("grok", value)).toThrow("boolean"); + expect(() => resolveGrokDisableWebSearch("claude", true)).toThrow("Grok"); + }); +}); diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-web-search.ts b/packages/paperclip-runner/src/drivers/acpx/grok-web-search.ts new file mode 100644 index 00000000000..a8935920a5e --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/grok-web-search.ts @@ -0,0 +1,13 @@ +/** Preserve the selected Grok search policy across startup and recovery. */ +export function resolveGrokDisableWebSearch(agent: string, value: unknown): boolean | undefined { + if (value === undefined) return undefined; + if (agent !== "grok" || typeof value !== "boolean") throw new Error("Grok web search policy requires a Grok boolean setting"); + return value; +} + +export function admittedGrokWebSearchPolicy(agent: string, expected: unknown, descriptor: Record, identity: Record): void { + const selected = resolveGrokDisableWebSearch(agent, expected); + if (descriptor.grokDisableWebSearch !== selected || identity.grokDisableWebSearch !== selected) { + throw new Error("Grok web search policy differs from the admitted configuration"); + } +} diff --git a/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts index 8d81addb3cf..f57fa913ba7 100644 --- a/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/pi-installation.test.ts @@ -10,6 +10,7 @@ import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "./pi-closure-pins.js"; import { PI_NODE_VERSION } from "./pi-node-pins.js"; import { assertPiInstallationProfile, verifyPiInstallation } from "./pi-installation.js"; import { QUALIFIED_ACPX_PROFILES, type AcpxReleaseProfile } from "./qualified-profiles.js"; +import { isSupportedAcpxProfileVersion } from "./profile-compatibility.js"; const roots: string[] = []; afterEach(async () => { vi.unstubAllEnvs(); await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); }); @@ -164,11 +165,21 @@ describe("Pi installation factory", () => { expect(current.commandDigest).not.toBe(prior.commandDigest); }); + it("retains profile 29 while binding Grok effort recovery identity in profile 30", async () => { + const read = async (version: number) => JSON.parse(await readFile(new URL(`../../../test-fixtures/pi-acp/profile-v${version}-identity.json`, import.meta.url), "utf8")); + const prior = await read(29), current = await read(30); + expect(Object.keys(current.declaration).filter(key => JSON.stringify(current.declaration[key]) !== JSON.stringify(prior.declaration[key])).sort()) + .toEqual(["agentProfileVersion", "recoveryIdentitySourceSha256", "runtimeSandboxSourceSha256"]); + expect(current.declaration.closure).toEqual(prior.declaration.closure); + expect(current.commandDigest).not.toBe(prior.commandDigest); + expect(isSupportedAcpxProfileVersion("pi", 29)).toBe(true); + }); + it("binds the profile declaration to the reviewed patch and platform closure pins", async () => { const hash = (bytes: string | Uint8Array) => createHash("sha256").update(bytes).digest("hex"); const canonical = (value: any): string => Array.isArray(value) ? `[${value.map(canonical).join(",")}]` : value && typeof value === "object" ? `{${Object.keys(value).sort().map(key => `${JSON.stringify(key)}:${canonical(value[key])}`).join(",")}}` : JSON.stringify(value); - const identity = JSON.parse(await readFile(new URL("../../../test-fixtures/pi-acp/profile-v29-identity.json", import.meta.url), "utf8")); + const identity = JSON.parse(await readFile(new URL(`../../../test-fixtures/pi-acp/profile-v${QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion}-identity.json`, import.meta.url), "utf8")); const declaration = identity.declaration; expect(declaration.sharedRuntimeContract).toBe("paperclip.acpx-runtime-contract.v1"); expect(identity.commandDigest).toBe(`sha256:${hash(canonical(declaration))}`); @@ -199,21 +210,21 @@ describe("Pi installation factory", () => { it("rejects legacy profiles and tampered executable identities", () => { expect(() => assertPiInstallationProfile(candidate())).not.toThrow(); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 29"); - expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 29"); - for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 29"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 1 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 2 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 3 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 4 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 5 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 6 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 7 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 8 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 9 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 10 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 11 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 12 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 13 })).toThrow("version 30"); + expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: 14 })).toThrow("version 30"); + for (const priorVersion of [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29]) expect(() => assertPiInstallationProfile({ ...candidate(), agentProfileVersion: priorVersion })).toThrow("version 30"); for (const changed of [{ agentServerVersion: "latest" }, { commandDigest: `sha256:${"0".repeat(64)}` }, { agentRuntimePackage: "ambient-pi" }]) { expect(() => assertPiInstallationProfile({ ...candidate(), ...changed })).toThrow("trusted declaration"); } diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 08db0ba384d..6d397e70882 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { - pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64], + pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/acpx/recovery-identity.test.ts b/packages/paperclip-runner/src/drivers/acpx/recovery-identity.test.ts index 08010a68352..0b0e148d73b 100644 --- a/packages/paperclip-runner/src/drivers/acpx/recovery-identity.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/recovery-identity.test.ts @@ -91,6 +91,33 @@ describe("ACPX recovery identity", () => { expect(() => verifyExpectedAcpxIdentity(expected, high, { ...saved, claudeEffort: undefined })).toThrow("persisted"); expect(() => verifyExpectedAcpxIdentity(expected, low, saved)).toThrow("immutable"); }); + it("binds Grok effort to recovery and rejects changed or omitted effort", async () => { + const fixture = await recoveryFixture(); + const profile = resolveQualifiedAcpxProfile("grok", "grok-4.7"); + const input = { ...fixture.input, profile, requestedModel: profile.qualificationModel, grokEffort: "high" as const }; + const high = await createAcpxRecoveryBinding(input); + const low = await createAcpxRecoveryBinding({ ...input, grokEffort: "low" }); + expect(low.profileSessionKey).not.toBe(high.profileSessionKey); + const expected = { ...fixture.expected, profileDigest: high.commandDigest, requestedModel: high.requestedModel, effectiveModel: high.effectiveModel, grokEffort: "high" as const }; + const saved = createAcpxIdentityRecord(expected, high); + expect(acpxProviderSessionIdentity(saved, high).grokEffort).toBe("high"); + expect(() => verifyExpectedAcpxIdentity({ ...expected, grokEffort: undefined }, high, saved)).toThrow("immutable"); + expect(() => verifyExpectedAcpxIdentity(expected, high, { ...saved, grokEffort: undefined })).toThrow("persisted"); + expect(() => verifyExpectedAcpxIdentity(expected, low, saved)).toThrow("immutable"); + }); + + it("fences changed or omitted Grok search policy across recovery", async () => { + const fixture = await recoveryFixture(); + const profile = resolveQualifiedAcpxProfile("grok", "grok-4.7"); + const input = { ...fixture.input, profile, requestedModel: "grok-4.7", grokDisableWebSearch: true }; + const disabled = await createAcpxRecoveryBinding(input); + const enabled = await createAcpxRecoveryBinding({ ...input, grokDisableWebSearch: false }); + expect(enabled.profileSessionKey).not.toBe(disabled.profileSessionKey); + const expected = { ...fixture.expected, profileDigest: disabled.commandDigest, requestedModel: "grok-4.7", effectiveModel: "grok-4.7", grokDisableWebSearch: true }; + const saved = createAcpxIdentityRecord(expected, disabled); + expect(() => verifyExpectedAcpxIdentity(expected, disabled, { ...saved, grokDisableWebSearch: undefined })).toThrow("persisted"); + expect(() => verifyExpectedAcpxIdentity(expected, enabled, saved)).toThrow("immutable"); + }); it("binds Pi thinking mode and rejects old or mismatched warm identities", async () => { const fixture = await recoveryFixture(); diff --git a/packages/paperclip-runner/src/drivers/acpx/recovery-identity.ts b/packages/paperclip-runner/src/drivers/acpx/recovery-identity.ts index 835a3047831..bb05e194ff6 100644 --- a/packages/paperclip-runner/src/drivers/acpx/recovery-identity.ts +++ b/packages/paperclip-runner/src/drivers/acpx/recovery-identity.ts @@ -1,4 +1,6 @@ +import { resolveGrokDisableWebSearch } from "./grok-web-search.js"; import { resolveClaudeEffort } from "./claude-effort.js"; +import { resolveGrokEffort } from "./grok-effort.js"; import { parseProviderMode } from "../../contracts/provider-mode.js"; import { resolvePiThinkingLevel, type PiThinkingLevel } from "./pi-thinking.js"; import { createHash } from "node:crypto"; @@ -25,6 +27,8 @@ export interface AcpxRecoveryBinding { mode?: string; piThinkingLevel?: PiThinkingLevel; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; profileSessionKey: string; } @@ -42,6 +46,8 @@ export interface AcpxIdentityRecord { mode?: string; piThinkingLevel?: PiThinkingLevel; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; providerLifetimeFenceCandidates: readonly [number, number, number]; } @@ -55,6 +61,8 @@ export async function createAcpxRecoveryBinding(input: { mode?: string; piThinkingLevel?: PiThinkingLevel; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; providerConfigurationDigest?: string; homeBinding?: Readonly>; providerPolicy?: { readOnly: boolean; readRoots?: readonly string[]; protectedPaths?: readonly string[] }; @@ -63,6 +71,8 @@ export async function createAcpxRecoveryBinding(input: { const mode = parseProviderMode(input.mode); const piThinkingLevel = resolvePiThinkingLevel(input.profile.agent, input.piThinkingLevel); const claudeEffort = resolveClaudeEffort(input.profile.agent, input.claudeEffort); + const grokEffort = resolveGrokEffort(input.profile.agent, input.grokEffort); + const grokDisableWebSearch = resolveGrokDisableWebSearch(input.profile.agent, input.grokDisableWebSearch); if (input.providerPolicy !== undefined && typeof input.providerPolicy.readOnly !== "boolean") throw new Error("ACPX recovery requires a valid task execution policy"); if (input.requestedModel !== input.profile.qualificationModel) { throw new Error("ACPX recovery requested model does not match its admitted profile"); @@ -112,6 +122,8 @@ export async function createAcpxRecoveryBinding(input: { ...(mode ? { mode } : {}), ...(piThinkingLevel ? { piThinkingLevel } : {}), ...(claudeEffort ? { claudeEffort } : {}), + ...(grokEffort ? { grokEffort } : {}), + ...(grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch }), }), ).replace("sha256:", "paperclip-"); return { @@ -127,6 +139,8 @@ export async function createAcpxRecoveryBinding(input: { ...(mode ? { mode } : {}), ...(piThinkingLevel ? { piThinkingLevel } : {}), ...(claudeEffort ? { claudeEffort } : {}), + ...(grokEffort ? { grokEffort } : {}), + ...(grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch }), profileSessionKey, }; } @@ -150,6 +164,8 @@ export function createAcpxIdentityRecord( ...(binding.mode ? { mode: binding.mode } : {}), ...(binding.piThinkingLevel ? { piThinkingLevel: binding.piThinkingLevel } : {}), ...(binding.claudeEffort ? { claudeEffort: binding.claudeEffort } : {}), + ...(binding.grokEffort ? { grokEffort: binding.grokEffort } : {}), + ...(binding.grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch: binding.grokDisableWebSearch }), providerLifetimeFenceCandidates: Object.freeze([ ...expected.providerLifetimeFenceCandidates, ]) as readonly [number, number, number], @@ -178,6 +194,8 @@ export function acpxProviderSessionIdentity( ...(record.mode ? { mode: record.mode } : {}), ...(record.piThinkingLevel ? { piThinkingLevel: record.piThinkingLevel } : {}), ...(record.claudeEffort ? { claudeEffort: record.claudeEffort } : {}), + ...(record.grokEffort ? { grokEffort: record.grokEffort } : {}), + ...(record.grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch: record.grokDisableWebSearch }), providerLifetimeFenceCandidates: record.providerLifetimeFenceCandidates, }; verifyExpectedAcpxIdentity(identity, binding, record); @@ -205,7 +223,8 @@ export function verifyExpectedAcpxIdentity( expected.permissionMode !== binding.permissionMode || expected.mode !== binding.mode || expected.piThinkingLevel !== binding.piThinkingLevel || - expected.claudeEffort !== binding.claudeEffort + expected.claudeEffort !== binding.claudeEffort || + expected.grokEffort !== binding.grokEffort || expected.grokDisableWebSearch !== binding.grokDisableWebSearch ) { throw new Error( "ACPX recovery identity conflicts with the immutable session configuration", @@ -227,6 +246,7 @@ export function verifyExpectedAcpxIdentity( record.mode !== binding.mode || record.piThinkingLevel !== binding.piThinkingLevel || record.claudeEffort !== binding.claudeEffort || + record.grokEffort !== binding.grokEffort || record.grokDisableWebSearch !== binding.grokDisableWebSearch || !sameFenceCandidates( record.providerLifetimeFenceCandidates, expected.providerLifetimeFenceCandidates, @@ -253,7 +273,7 @@ function parsePersistedRecord(value: unknown): AcpxIdentityRecord { "permissionMode", "mode", "piThinkingLevel", - "claudeEffort", + "claudeEffort", "grokEffort", "grokDisableWebSearch", "providerLifetimeFenceCandidates", ]); return validatedRecord(record); @@ -282,6 +302,8 @@ function validatedRecord(value: Record): AcpxIdentityRecord { } if (value.mode !== undefined) parseProviderMode(value.mode); resolveClaudeEffort("claude", value.claudeEffort); + resolveGrokEffort("grok", value.grokEffort); + resolveGrokDisableWebSearch("grok", value.grokDisableWebSearch); if (value.piThinkingLevel !== undefined) resolvePiThinkingLevel("pi", value.piThinkingLevel); validateFenceCandidates(value.providerLifetimeFenceCandidates); return value as unknown as AcpxIdentityRecord; @@ -313,6 +335,8 @@ function validateExpected(expected: AcpxExpectedSessionIdentity): void { } if (expected.mode !== undefined) parseProviderMode(expected.mode); resolveClaudeEffort("claude", expected.claudeEffort); + resolveGrokEffort("grok", expected.grokEffort); + resolveGrokDisableWebSearch("grok", expected.grokDisableWebSearch); if (expected.piThinkingLevel !== undefined) resolvePiThinkingLevel("pi", expected.piThinkingLevel); validateFenceCandidates(expected.providerLifetimeFenceCandidates); } diff --git a/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts b/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts index a58bfe2ea75..ddba9de9944 100644 --- a/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts +++ b/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts @@ -1,5 +1,7 @@ +import { resolveGrokDisableWebSearch } from "./grok-web-search.js"; import { composeCopilotSystemInstructions } from "./copilot-profile.js"; import { resolveClaudeEffort } from "./claude-effort.js"; +import { resolveGrokEffort } from "./grok-effort.js"; import { piProviderConfiguration } from "./pi-provider-config.js"; import { withAcpxTurnCancellation } from "./turn-cancellation.js"; import { resolveAcpxProviderMode } from "./provider-mode.js"; @@ -88,6 +90,8 @@ export interface AcpxRuntimePortIdentity { mode?: string; piThinkingLevel?: PiThinkingLevel; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; acpxRecordId: string; backendSessionId: string; agentSessionId: string; @@ -169,6 +173,8 @@ export interface AcpxRuntimePortOpenOptions { mode?: string; piThinkingLevel?: PiThinkingLevel; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; permissionPolicy: ReturnType; launchEnvironment: Readonly; /** Kernel credential-home quorum inherited by the provider sentinel. */ @@ -244,6 +250,8 @@ export interface OpenAcpxRuntimeHostOptions { mode?: string; piThinkingLevel?: PiThinkingLevel; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; systemInstructions?: string; runtimeContext?: NativeRuntimeContextSnapshot | null; environment?: NodeJS.ProcessEnv; @@ -369,6 +377,8 @@ export class AcpxRuntimeHost { mode: resolveAcpxProviderMode(options.agent, options.mode), piThinkingLevel: resolvePiThinkingLevel(options.agent, options.piThinkingLevel), claudeEffort: resolveClaudeEffort(options.agent, options.claudeEffort), + grokEffort: resolveGrokEffort(options.agent, options.grokEffort), + grokDisableWebSearch: resolveGrokDisableWebSearch(options.agent, options.grokDisableWebSearch), ...(preconfigured ? { homeBinding: preconfiguredHomeEnvironment(options.environment) } : {}), ...(["cursor", "copilot", "pi"].includes(options.agent) && options.providerPolicy !== undefined ? { providerPolicy: options.providerPolicy } : {}), @@ -624,6 +634,7 @@ export class AcpxRuntimeHost { mode: binding.mode, piThinkingLevel: binding.piThinkingLevel, claudeEffort: binding.claudeEffort, + grokEffort: binding.grokEffort, grokDisableWebSearch: binding.grokDisableWebSearch, permissionPolicy: acpxRuntimePermissionPolicy( binding.permissionMode, ), @@ -689,6 +700,7 @@ export class AcpxRuntimeHost { throw new Error("ACPX runtime Pi thinking level conflicts with session binding"); } if (runtimeIdentity.claudeEffort !== binding.claudeEffort) throw new Error("ACPX runtime Claude effort conflicts with session binding"); + if (runtimeIdentity.grokEffort !== binding.grokEffort || runtimeIdentity.grokDisableWebSearch !== binding.grokDisableWebSearch) throw new Error("ACPX runtime Grok effort conflicts with session binding"); const observedIdentity: AcpxExpectedSessionIdentity = { kind: "acpx", normalizedSessionId: binding.normalizedSessionId, diff --git a/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.test.ts b/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.test.ts index 7640f681d04..a8852499b24 100644 --- a/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.test.ts @@ -99,6 +99,14 @@ describe("ACPX runtime sandbox", () => { expect((await stat(requirementsPath)).mode & 0o777).toBe(0o600); }); + it.each([true, false])("preserves Grok search policy %s above compatible project settings", async disabled => { + const fixture = await sandboxFixture("grok"); + const sandbox = await prepareAcpxRuntimeSandbox({ binding: { ...fixture.binding, grokDisableWebSearch: disabled }, agent: "grok" }); + expect(await readFile(join(sandbox.agentHomeDirectory, "config.toml"), "utf8")).toContain(`disable_web_search = ${disabled}`); + const requirements = await readFile(join(sandbox.agentHomeDirectory, "requirements.toml"), "utf8"); + expect(requirements.includes("disable_web_search = true")).toBe(disabled); + }); + it("keeps provider credentials outside identity-enabled Codex shell commands", async () => { const fixture = await sandboxFixture("codex"); const sandbox = await prepareAcpxRuntimeSandbox({ diff --git a/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts b/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts index 0a2f679bbaf..6f4543c23ef 100644 --- a/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts +++ b/packages/paperclip-runner/src/drivers/acpx/runtime-sandbox.ts @@ -443,6 +443,7 @@ export async function prepareAcpxRuntimeSandbox(input: { // Native project rules merge with global rules. Ask rules win over project // allows, keeping operation approval at the runner's authenticated boundary. await writePrivateFile(join(agentHomeDirectory, "config.toml"), [ + `disable_web_search = ${input.binding.grokDisableWebSearch === true}`, "[permission]", 'ask = ["Bash", "Read", "Edit", "Grep", "MCPTool", "WebFetch", "WebSearch"]', "[ui]", 'permission_mode = "ask"', @@ -458,7 +459,7 @@ export async function prepareAcpxRuntimeSandbox(input: { // Always-approve may otherwise be enabled by compatible project settings. // Even approve-all runs use one host decision per requested operation. await writePrivateFile(join(agentHomeDirectory, "requirements.toml"), - "[ui]\ndisable_bypass_permissions_mode = true\n"); + `${input.binding.grokDisableWebSearch === true ? "disable_web_search = true\n" : ""}[ui]\ndisable_bypass_permissions_mode = true\n`); } if (input.agent === "pi") { await writePrivateFile( diff --git a/packages/paperclip-runner/src/drivers/acpx/sidecar-protocol.ts b/packages/paperclip-runner/src/drivers/acpx/sidecar-protocol.ts index 717ffc97748..efc7ea9f63b 100644 --- a/packages/paperclip-runner/src/drivers/acpx/sidecar-protocol.ts +++ b/packages/paperclip-runner/src/drivers/acpx/sidecar-protocol.ts @@ -48,6 +48,8 @@ export interface AcpxSidecarOpenParams { mode?: string; piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; permissionModePinned: boolean; providerPolicy?: { readOnly: boolean }; systemInstructions: string; @@ -71,6 +73,8 @@ export interface AcpxExpectedSessionIdentity { mode?: string; piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; providerLifetimeFenceCandidates: readonly [number, number, number]; } diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index 38c34e26872..a4d3769bb4c 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -61,6 +61,7 @@ export { } from "./drivers/acpx/codex-acpx-driver.js"; export * from "./drivers/acpx/sidecar-protocol.js"; export { resolveClaudeEffort } from "./drivers/acpx/claude-effort.js"; +export { resolveGrokEffort } from "./drivers/acpx/grok-effort.js"; export * from "./drivers/runner-tool-bridge.js"; export { createRunnerdCodexTransport, @@ -103,3 +104,5 @@ export type { RunnerExecutionDriver } from "./control-plane/runner-execution-dri export { bindRunnerLaunchResources, createRunnerProviderLaunch, runnerProviderLaunchRequirements, runnerLaunchRequirementsDigest, validatePreparedRunnerLaunch } from "./live/runner-provider-launch.js"; export type { RunnerProviderLaunchRequirements, RunnerProviderLaunchEnvironment, PreparedRunnerProviderLaunch, RunnerLaunchPreparer } from "./live/runner-provider-launch.js"; + +export { resolveGrokDisableWebSearch } from "./drivers/acpx/grok-web-search.js"; diff --git a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts index 57f237bd57d..b924df89d74 100644 --- a/packages/paperclip-runner/src/live/runner-provider-launch.test.ts +++ b/packages/paperclip-runner/src/live/runner-provider-launch.test.ts @@ -19,6 +19,17 @@ const environment: RunnerProviderLaunchEnvironment = { }; describe("built-in provider launch preparation", () => { + it.each([true, false])("binds Grok search policy %s into the prepared descriptor", disabled => { + const selected = runnerProviderLaunchRequirements({ provider: "acpx", acpxAgent: "grok", grokDisableWebSearch: disabled }, + requirements.identity, { model: "grok-4.7" }, null, ""); + const launch = createRunnerProviderLaunch(selected, environment); + expect(launch).toHaveProperty("grokDisableWebSearch", disabled); + const prepared = { configurationId: "grok-policy", requirementsDigest: runnerLaunchRequirementsDigest(selected), provider: launch }; + expect(() => validatePreparedRunnerLaunch(selected, prepared)).not.toThrow(); + expect(() => validatePreparedRunnerLaunch(selected, { ...prepared, provider: { ...launch, grokDisableWebSearch: !disabled } })) + .toThrow("runner_prepared_launch_requirements_mismatch"); + }); + it("binds disabled hosted search to the prepared launch and overrides execution-host arguments", () => { const disabled = runnerProviderLaunchRequirements({ provider: "codex", disableCodexWebSearch: true }, requirements.identity, { model: requirements.model }, null, ""); diff --git a/packages/paperclip-runner/src/live/runner-provider-launch.ts b/packages/paperclip-runner/src/live/runner-provider-launch.ts index 5f2ae436d1e..6f2a815e3d3 100644 --- a/packages/paperclip-runner/src/live/runner-provider-launch.ts +++ b/packages/paperclip-runner/src/live/runner-provider-launch.ts @@ -23,7 +23,7 @@ export interface RunnerProviderLaunchRequirements { includeCollaborationModeInstructions: boolean; settings: Pick; + "piThinkingLevel" | "claudeEffort" | "grokEffort" | "grokDisableWebSearch" | "managedProfile" | "agentCoreProfile" | "opencodePermissionMode" | "disableCodexWebSearch">; instructions: string; runtimeContext: NativeRuntimeContextSnapshot | null; } @@ -56,7 +56,7 @@ export function runnerProviderLaunchRequirements( acpxAgent: options.acpxAgent, acpxMode: options.acpxMode, acpxPermissionMode: options.acpxPermissionMode, acpxPermissionModePinned: options.acpxPermissionModePinned, - piThinkingLevel: options.piThinkingLevel, claudeEffort: options.claudeEffort, + piThinkingLevel: options.piThinkingLevel, claudeEffort: options.claudeEffort, grokEffort: options.grokEffort, grokDisableWebSearch: options.grokDisableWebSearch, managedProfile: options.managedProfile, agentCoreProfile: options.agentCoreProfile, opencodePermissionMode: options.opencodePermissionMode, ...(options.disableCodexWebSearch === true ? { disableCodexWebSearch: true } : {}), @@ -121,6 +121,7 @@ export function validatePreparedRunnerLaunch( if (requirements.provider === "acpx") { const policy = descriptor.providerPolicy as Record | undefined; if (descriptor.agent !== (requirements.settings.acpxAgent ?? "codex") + || descriptor.grokDisableWebSearch !== requirements.settings.grokDisableWebSearch || descriptor.permissionMode !== resolveRunnerdAcpxPermissionMode(requirements.settings.acpxPermissionMode) || policy?.readOnly !== requirements.readOnly || descriptor.runId !== requirements.identity.runId @@ -246,6 +247,8 @@ export function createRunnerProviderLaunch( ...(selectedAcpxMode === undefined ? {} : { mode: selectedAcpxMode }), ...(acpxProfile!.agent === "pi" ? { piThinkingLevel: settings.piThinkingLevel } : {}), ...(acpxProfile!.agent === "claude" && settings.claudeEffort ? { claudeEffort: settings.claudeEffort } : {}), + ...(acpxProfile!.agent === "grok" && settings.grokEffort ? { grokEffort: settings.grokEffort } : {}), + ...(acpxProfile!.agent === "grok" && settings.grokDisableWebSearch !== undefined ? { grokDisableWebSearch: settings.grokDisableWebSearch } : {}), permissionMode: resolveRunnerdAcpxPermissionMode( settings.acpxPermissionMode, ), diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index aa4f781f06f..c05c6a14894 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -1,3 +1,4 @@ +import { resolveGrokDisableWebSearch, admittedGrokWebSearchPolicy } from "../drivers/acpx/grok-web-search.js"; import { PRP_COMPLETION_TOOL_NAME, PRP_COMPLETION_RESULT_BRIDGE_INPUT_SCHEMA } from "../contracts/completion-result.js"; import { RunnerLaunchPreparationStore } from "./runner-launch-preparation-store.js"; import type { PreparedRunnerProviderLaunch, RunnerProviderLaunchRequirements } from "./runner-provider-launch.js"; @@ -8,6 +9,7 @@ import type { RunnerExecutionDriver } from "../control-plane/runner-execution-dr import { resolvePinnedOpenCodeCommand } from "../drivers/opencode/opencode-server-driver.js"; import { ACPX_CAPABILITY_PROFILES } from "../drivers/acpx/capability-profiles.js"; import { resolveClaudeEffort, admittedClaudeEffort } from "../drivers/acpx/claude-effort.js"; +import { resolveGrokEffort, admittedGrokEffort } from "../drivers/acpx/grok-effort.js"; import { admittedPiThinkingLevel, resolvePiThinkingLevel } from "../drivers/acpx/pi-thinking.js"; import { configuredEnvironment } from "../configured-environment.js"; import { githubCredentialEnvironment } from "../github-credential-environment.js"; @@ -1213,6 +1215,8 @@ export interface CapabilityRunnerdProcessEvidence { /** Effective native mode, populated only from the admitted Pi identity. */ piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; providerVersion: string | null; acpxAgent: QualifiedAcpxAgent | null; agentServerVersion: string | null; @@ -1242,6 +1246,8 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD acpxMode?: string; piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; acpxPermissionModePinned?: boolean; acpxSidecarPath?: string; /** SHA-256 verified by the provider-pack authority before runner startup. */ @@ -3601,6 +3607,8 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { throw new Error("runner_launch_preparation_requires_process_owner_and_artifact"); } resolveClaudeEffort(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.claudeEffort); + resolveGrokEffort(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.grokEffort); + resolveGrokDisableWebSearch(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.grokDisableWebSearch); resolvePiThinkingLevel(options.provider === "acpx" ? options.acpxAgent ?? "codex" : "", options.piThinkingLevel); if (options.acpxMode !== undefined && options.provider !== "acpx") { throw new Error("acpxMode requires the ACPX provider"); @@ -6766,7 +6774,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { throw new Error("ACPX capability identity differs from the admitted profile"); } const claudeEffort = admittedClaudeEffort(this.options.acpxAgent ?? "codex", this.options.claudeEffort, descriptor, providerIdentity); + const grokEffort = admittedGrokEffort(this.options.acpxAgent ?? "codex", this.options.grokEffort, descriptor, providerIdentity); + admittedGrokWebSearchPolicy(this.options.acpxAgent ?? "codex", this.options.grokDisableWebSearch, descriptor, providerIdentity); if (claudeEffort !== undefined) this.#evidence.claudeEffort = claudeEffort; + if (grokEffort !== undefined) this.#evidence.grokEffort = grokEffort; + if (this.options.grokDisableWebSearch !== undefined) this.#evidence.grokDisableWebSearch = this.options.grokDisableWebSearch; const piThinkingLevel = admittedPiThinkingLevel(this.options.acpxAgent ?? "codex", this.options.piThinkingLevel, descriptor, providerIdentity); if (piThinkingLevel !== undefined) this.#evidence.piThinkingLevel = piThinkingLevel; this.#turnControls = parseAcpxTurnControlCapabilities(descriptor.turnControls, descriptor.agent); diff --git a/packages/paperclip-runner/src/protocol/generated/schema-bundle.ts b/packages/paperclip-runner/src/protocol/generated/schema-bundle.ts index a8898718e93..f82900041aa 100644 --- a/packages/paperclip-runner/src/protocol/generated/schema-bundle.ts +++ b/packages/paperclip-runner/src/protocol/generated/schema-bundle.ts @@ -1374,6 +1374,18 @@ export const providerDescriptorSchema = { "xhigh", "max" ] + }, + "grokEffort": { + "enum": [ + "minimal", + "low", + "medium", + "high", + "xhigh" + ] + }, + "grokDisableWebSearch": { + "type": "boolean" } }, "allOf": [ @@ -1724,6 +1736,46 @@ export const providerDescriptorSchema = { } } } + }, + { + "if": { + "required": [ + "grokEffort" + ] + }, + "then": { + "properties": { + "provider": { + "const": "acpx" + }, + "agent": { + "const": "grok" + } + }, + "required": [ + "agent" + ] + } + }, + { + "if": { + "required": [ + "grokDisableWebSearch" + ] + }, + "then": { + "properties": { + "provider": { + "const": "acpx" + }, + "agent": { + "const": "grok" + } + }, + "required": [ + "agent" + ] + } } ], "$defs": { diff --git a/packages/paperclip-runner/test-fixtures/pi-acp/profile-v30-identity.json b/packages/paperclip-runner/test-fixtures/pi-acp/profile-v30-identity.json new file mode 100644 index 00000000000..1bd57db0260 --- /dev/null +++ b/packages/paperclip-runner/test-fixtures/pi-acp/profile-v30-identity.json @@ -0,0 +1,51 @@ +{ + "commandDigest": "sha256:4517ab123dd89140ef462fc1905ac7a16a0dfcbaf21d67299a3c62de29c08a4d", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "pi", + "agentProfileVersion": 30, + "acpxVersion": "0.13.1", + "agentServerVersion": "0.0.33", + "agentRuntimeVersion": "1.0.0", + "nodeVersion": "24.21.0", + "closure": { + "darwin-arm64": "282022db10150c6632b3444df421342e7d534bdf5d5fb1097a2e79d0625a2bcf", + "darwin-x64": "64e251e19009f755c0b04f73ce2138246faab71a961b0f13d75ebfcc34bef12e", + "linux-x64": "713b1fdff42fb56a1518bdc084f181d70bee8ebadc3e4b1d76321ed9108c8410" + }, + "wrapperSha256": "9d129b3d38772e93e97080aa6c4e574ac5df4e47ce5bc331a484a9fbf8188b36", + "helperSha256": "41e0490b617da0d60e0c8ec58ef311236f945129d99f816cff6897f78da7f91d", + "extensionSha256": "f5818573a355702388072b28926897db546a5cf346b5171f646582ce5f9102a4", + "agentFilesBinding": "registered-runtime-context-v1", + "nativeQuestions": "select-confirm-input-editor-v4-root-field-types", + "pathPolicy": "sdk-normalization-v1", + "mcpToolNames": "provider-safe-exact-source-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "nativeAssistantMessages": "pi1-rpc-deltas-v3-bound-message-and-tool-receipts", + "nativeNotices": "session-extension-v2-bounded-runtime-failure", + "messageProjectionSha256": "17d1e91b948dfa6a56e1905645add52b502041ad843c8d50a13b07956f22aadd", + "noticeProjectionSha256": "014485adc690998e395d334ea3ae67f72be6060a2fc78a1185df7b8d1da28fbf", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "nativeInputDisposition": "queued-only-v1", + "cacheWarming": "disabled-owned-decision-v1", + "thinkingLevel": { + "supported": [ + "off", + "low", + "high", + "max" + ], + "newConfigurationDefault": "low", + "admission": "explicit-native-effective-before-prompt-v1", + "recovery": "mode-bound-v1" + }, + "modelSelection": "caller-selected-native-verified-v1", + "providerConfigurationSourceSha256": "75a9a13f13f224dc092bd8ebf49e8747162172943761e44476c8eb18d80c52d4", + "credentialEnvironmentSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "runtimeSandboxSourceSha256": "981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec", + "recoveryIdentitySourceSha256": "f74d061cc07b0c7ca04df62ca991d940a733d9ef6594b3362451769fc10030b8", + "nativeCancellation": "acknowledged-cancelled-terminal-preserves-usage-v1", + "dependencySecurityPatchSha256": "5273a195b952f233336122842faaf650193c04f36b9359d20d4d63079b0ce814", + "openRouterAttributionSourceSha256": "1211191529d6073b733b311c3d69cada4973eede0f3e540c46e261f78e0b90f9" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json new file mode 100644 index 00000000000..72cafbda261 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 66, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62", + "semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "4cae579aac836762fdbf92c5447425d8b756e87a5766bc0c7a1885a90541c8d6", + "semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b", + "providerLaunchSourceSha256": "4c1abd77bbed9723e669009a78dab6fe024837718ced9f22c159f9e22ce1f26c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "9584d9801b87f67e60b0007beafa909fc5b05dc6adac71409db1c7f80a2f7916", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297", + "profileCompatibilitySourceSha256": "121214465210f235bebf096a2ff80b093373d84f08b89902e694e6fedbe6ea94", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "976fef21ec9f3ae9dd5eabb133a55ae578f1c0729c657c30e42562a80db237d8", + "runtimeAdapterSourceSha256": "e2dd18de4b134a5492de93c88b4add479f63fb650c655d60a17c04af722c7563", + "runtimeSandboxSourceSha256": "981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/shared/src/agent-runner.ts b/packages/shared/src/agent-runner.ts index 3ae50819a66..413eb0adfb4 100644 --- a/packages/shared/src/agent-runner.ts +++ b/packages/shared/src/agent-runner.ts @@ -6,10 +6,11 @@ export interface AgentRunnerAvailability { defaultRunner: AgentRunner; } -export function paperclipRunnerProfileForHarness(harness: string): { provider: "codex" | "opencode" } | { provider: "acpx"; acpxAgent: "claude" | "pi" } | undefined { +export function paperclipRunnerProfileForHarness(harness: string): { provider: "codex" | "opencode" } | { provider: "acpx"; acpxAgent: "claude" | "pi" | "grok" } | undefined { if (harness === "codex_local") return { provider: "codex" }; if (harness === "claude_local") return { provider: "acpx", acpxAgent: "claude" }; if (harness === "pi_local") return { provider: "acpx", acpxAgent: "pi" }; + if (harness === "grok_local") return { provider: "acpx", acpxAgent: "grok" }; if (harness === "opencode_local") return { provider: "opencode" }; return undefined; } @@ -29,6 +30,7 @@ function supportsLinuxLibc(libc?: "glibc" | "musl" | "unknown", version?: string export function paperclipRunnerSupportsPlatform(harness: string, platform: string, architecture: string, libc?: "glibc" | "musl" | "unknown", glibcVersion?: string): boolean { return !!paperclipRunnerProfileForHarness(harness) + && (harness !== "grok_local" || (platform === "linux" && architecture === "x64") || (platform === "darwin" && architecture === "arm64")) && !(["opencode_local", "pi_local"].includes(harness) && platform === "linux" && architecture === "arm64") && ((platform === "linux" && supportsLinuxLibc(libc, glibcVersion) && (architecture === "x64" || architecture === "arm64")) || (platform === "darwin" && (architecture === "arm64" || architecture === "x64"))); diff --git a/scripts/pi-public-install-probe.mjs b/scripts/pi-public-install-probe.mjs index 945d8c902de..5ca30bbf939 100644 --- a/scripts/pi-public-install-probe.mjs +++ b/scripts/pi-public-install-probe.mjs @@ -52,8 +52,8 @@ assert.equal(JSON.parse(await readFile(join(server, 'package.json'), 'utf8')).na const { resolvePaperclipRunnerBinary } = await import(pathToFileURL(join(server, 'dist/services/native-runtime/native-codex-runner.js'))); const { createCapabilityRunnerdCodexTransport } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/live/runnerd-codex-transport.js'))); const { QUALIFIED_ACPX_PROFILES } = await import(pathToFileURL(join(server, 'dist/vendor/paperclip-runner/drivers/acpx/qualified-profiles.js'))); -assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 29); -assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:e80c6dc7bde12790371345ee5a12f0201d30941d26f1a780aec3d090545b2a06'); +assert.equal(QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, 30); +assert.equal(QUALIFIED_ACPX_PROFILES.pi.commandDigest, 'sha256:4517ab123dd89140ef462fc1905ac7a16a0dfcbaf21d67299a3c62de29c08a4d'); assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'qualificationModel'), false); assert.equal(Object.hasOwn(QUALIFIED_ACPX_PROFILES.pi, 'reportedModelId'), false); const daemon = resolvePaperclipRunnerBinary(); diff --git a/server/src/__tests__/agent-adapter-validation-routes.test.ts b/server/src/__tests__/agent-adapter-validation-routes.test.ts index c1ad79a4cf9..22515f7dc9c 100644 --- a/server/src/__tests__/agent-adapter-validation-routes.test.ts +++ b/server/src/__tests__/agent-adapter-validation-routes.test.ts @@ -363,13 +363,31 @@ describe("agent routes adapter validation", () => { await unregisterTestAdapter(missingAdapterType); }); + it("omits the default marker from the native Grok catalog while retaining Legacy defaults", async () => { + const adapters = await import("../adapters/index.js"); + const models = [{ id: "grok-build", label: "Default" }, { id: "grok-4.7", label: "Grok 4.7" }]; + const list = vi.spyOn(adapters, "listAdapterModels").mockResolvedValue(models); + const refresh = vi.spyOn(adapters, "refreshAdapterModels").mockResolvedValue(models); + try { + const app = await createApp(); + for (const suffix of ["", "&refresh=true"]) { + const native = await requestApp(app, baseUrl => request(baseUrl).get(`/api/companies/company-1/adapters/paperclip_runner/models?provider=acpx&acpxAgent=grok${suffix}`)); + expect(native.status).toBe(200); + expect(native.body).toEqual([models[1]]); + const legacy = await requestApp(app, baseUrl => request(baseUrl).get(`/api/companies/company-1/adapters/grok_local/models?refresh=${Boolean(suffix)}`)); + expect(legacy.status).toBe(200); + expect(legacy.body).toEqual(models); + } + } finally { list.mockRestore(); refresh.mockRestore(); } + }); + it("selects and refreshes the runner provider catalog independently", async () => { const adapters = await import("../adapters/index.js"); const list = vi.spyOn(adapters, "listAdapterModels").mockImplementation(async (type) => [{ id: type, label: type }]); const refresh = vi.spyOn(adapters, "refreshAdapterModels").mockImplementation(async (type) => [{ id: `${type}-fresh`, label: type }]); try { const app = await createApp(); - for (const [provider, adapter] of [["acpx", "claude_local"], ["codex", "codex_local"], ["opencode", "opencode_local"]]) { + for (const [provider, adapter] of [["acpx", "claude_local"], ["acpx&acpxAgent=grok", "grok_local"], ["codex", "codex_local"], ["opencode", "opencode_local"]]) { const res = await requestApp(app, (baseUrl) => request(baseUrl).get(`/api/companies/company-1/adapters/paperclip_runner/models?provider=${provider}`)); expect(res.status).toBe(200); expect(res.body).toEqual([{ id: adapter, label: adapter }]); @@ -746,7 +764,7 @@ describe("agent routes adapter validation", () => { const app = await createApp(); const res = await requestApp(app, baseUrl => request(baseUrl) .post("/api/companies/company-1/adapters/paperclip_runner/test-environment") - .send({ agentId: "11111111-1111-4111-8111-111111111111", adapterConfig: { provider: "acpx", acpxAgent: "grok" } })); + .send({ agentId: "11111111-1111-4111-8111-111111111111", adapterConfig: { provider: "acpx", acpxAgent: "cursor" } })); expect(res.status, JSON.stringify(res.body)).toBe(422); expect(res.body.details).toMatchObject({ code: "paperclip_runner_rollout_disabled" }); expect(probe).not.toHaveBeenCalled(); @@ -875,7 +893,7 @@ describe("agent routes adapter validation", () => { it("keeps other native provider selection behind its existing flag", async () => { const app = await createApp(); - const response = await requestApp(app, baseUrl => request(baseUrl).post("/api/companies/company-1/agents").send({ name: "Native Grok", adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "grok", model: "openrouter/deepseek/deepseek-v4-flash-0731" } })); + const response = await requestApp(app, baseUrl => request(baseUrl).post("/api/companies/company-1/agents").send({ name: "Native Cursor", adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "cursor", model: "provider-model" } })); expect(response.status).toBe(422); expect(response.body.details).toMatchObject({ code: "paperclip_runner_rollout_disabled" }); }); @@ -1021,6 +1039,43 @@ describe("agent routes adapter validation", () => { expect(mockAgentService.update.mock.calls.at(-1)?.[1]).toMatchObject({ adapterConfig: { provider: "codex" } }); }); + it.each([{ maxTurns: 5 }, { disableWebSearch: "invalid" }])("rejects a direct Grok conversion with incompatible saved settings %j", async unsupported => { + const existing = await mockAgentService.getById(); + mockAgentService.getById.mockResolvedValue({ ...existing, adapterType: "grok_local", + adapterConfig: { model: "grok-4.7", reasoningEffort: "high", ...unsupported } }); + const app = await createApp(); + const res = await requestApp(app, baseUrl => request(baseUrl).patch(`/api/agents/${existing.id}`) + .send({ adapterType: "paperclip_runner" })); + expect(res.status, JSON.stringify(res.body)).toBe(422); + expect(mockAgentService.update).not.toHaveBeenCalled(); + }); + + it("translates compatible settings during a direct Grok conversion", async () => { + const existing = await mockAgentService.getById(); + mockAgentService.getById.mockResolvedValue({ ...existing, adapterType: "grok_local", + adapterConfig: { model: "grok-4.7", reasoningEffort: "high", command: "grok", disableWebSearch: true } }); + const app = await createApp(); + const res = await requestApp(app, baseUrl => request(baseUrl).patch(`/api/agents/${existing.id}`) + .send({ adapterType: "paperclip_runner" })); + expect(res.status, JSON.stringify(res.body)).toBe(200); + const saved = mockAgentService.update.mock.calls.at(-1)?.[1]; + expect(saved).toMatchObject({ adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", reasoningEffort: "high", disableWebSearch: true } }); + expect(saved.adapterConfig).not.toHaveProperty("command"); + }); + + it.each([undefined, "acpx"])("preserves an explicitly selected ACPX agent when converting Grok (provider: %s)", async provider => { + const existing = await mockAgentService.getById(); + mockAgentService.getById.mockResolvedValue({ ...existing, adapterType: "grok_local", adapterConfig: { model: "grok-4.7" } }); + const app = await createApp(); + const res = await requestApp(app, baseUrl => request(baseUrl).patch(`/api/agents/${existing.id}`) + .send({ adapterType: "paperclip_runner", replaceAdapterConfig: true, + adapterConfig: { ...(provider ? { provider } : {}), acpxAgent: "claude", model: "claude-opus-5-5" } })); + expect(res.status, JSON.stringify(res.body)).toBe(200); + expect(mockAgentService.update.mock.calls.at(-1)?.[1]).toMatchObject({ + adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "claude", model: "claude-opus-5-5" }, + }); + }); + it.each(["codex_local", "claude_local"])("upgrades an existing %s through runner selection with its saved settings", async adapterType => { const existing = await mockAgentService.getById(); const env = { MIGRATION_MARKER: { type: "plain", value: "keep" } }; diff --git a/server/src/__tests__/agent-runner-qualification-routes.test.ts b/server/src/__tests__/agent-runner-qualification-routes.test.ts index 10c1980d248..1ef12028528 100644 --- a/server/src/__tests__/agent-runner-qualification-routes.test.ts +++ b/server/src/__tests__/agent-runner-qualification-routes.test.ts @@ -58,7 +58,8 @@ describe.skipIf(!support.supported)("agent-scoped runner qualification", () => { }); const qualify = () => request(app).post(`/api/agents/${agentId}/qualify-runner`).send({ expectedEnvironmentId: environmentId }); - it("allows the company owner without instance admin and keeps the generic probe restricted", async () => { + it.each(["claude_local", "grok_local"])("allows the company owner to qualify %s without instance admin and keeps the generic probe restricted", async adapterType => { + await db.update(agents).set({ adapterType }).where(eq(agents.id, agentId)); const before = await db.select().from(agents).where(eq(agents.id, agentId)); const res = await qualify(); expect(res.status).toBe(200); diff --git a/server/src/__tests__/company-portability.test.ts b/server/src/__tests__/company-portability.test.ts index 2a50f43878f..fdf00b828af 100644 --- a/server/src/__tests__/company-portability.test.ts +++ b/server/src/__tests__/company-portability.test.ts @@ -6095,7 +6095,7 @@ describe("company portability", () => { await expect(portability.importBundle(input, "user-1")).rejects.toMatchObject({ status: 422, details: { code: "paperclip_runner_dot_disabled" } }); expect(agentSvc.create).not.toHaveBeenCalled(); instanceSettingsSvc.getExperimental.mockResolvedValue({ enableNativeRunner: false, enableOpenAiDot: true }); - await expect(portability.importBundle({ ...input, adapterOverrides: { claudecoder: { adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "grok", model: "grok-build-1.0.13" } } } }, "user-1")) + await expect(portability.importBundle({ ...input, adapterOverrides: { claudecoder: { adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "copilot", model: "gpt-5.4" } } } }, "user-1")) .rejects.toMatchObject({ status: 422, details: { code: "paperclip_runner_rollout_disabled" } }); expect(agentSvc.create).not.toHaveBeenCalled(); await portability.importBundle(input, "user-1"); @@ -6106,6 +6106,27 @@ describe("company portability", () => { expect(() => resolvePaperclipRunnerProviderProfile(createdConfig)).toThrow(expect.objectContaining({ code: "paperclip_runner_dot_config_invalid" })); }); + it("imports qualified Grok without the experimental flag and preserves its model and effort", async () => { + const portability = companyPortabilityService({} as any); + const exported = await portability.exportBundle("company-1", { include: { company: false, agents: true, projects: false, issues: false } }); + agentSvc.list.mockResolvedValue([]); + agentSvc.create.mockImplementation(async (_companyId: string, input: Record) => ({ id: "agent-created", ...input })); + instanceSettingsSvc.getExperimental.mockResolvedValue({ enableNativeRunner: false }); + const adapterConfig = { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", reasoningEffort: "high" }; + + await portability.importBundle({ + source: { type: "inline", rootPath: exported.rootPath, files: exported.files }, + include: { company: false, agents: true, projects: false, issues: false }, + target: { mode: "existing_company", companyId: "company-1" }, + agents: "all", collisionStrategy: "rename", + adapterOverrides: { claudecoder: { adapterType: "paperclip_runner", adapterConfig } }, + }, "user-1"); + + expect(agentSvc.create).toHaveBeenCalledWith("company-1", expect.objectContaining({ + adapterType: "paperclip_runner", adapterConfig: expect.objectContaining(adapterConfig), + }), { createdByUserId: "user-1" }); + }); + it("accepts native OpenCode imports without the experimental flag", async () => { const portability = companyPortabilityService({} as any); const exported = await portability.exportBundle("company-1", { diff --git a/server/src/redaction.ts b/server/src/redaction.ts index f2fe6f6fa6b..4d3eeb01e23 100644 --- a/server/src/redaction.ts +++ b/server/src/redaction.ts @@ -93,6 +93,7 @@ export const PAPERCLIP_PUBLIC_SCHEMA_IDS = new Set([ "paperclip.native-execution-input.v7", "paperclip.native-execution-input.v8", "paperclip.native-execution-input.v9", + "paperclip.native-execution-input.v10", "paperclip.native-finalization.v1", "paperclip.native-finalization.v2", "paperclip.native-harness-backup-stamp.v1", diff --git a/server/src/routes/agents.ts b/server/src/routes/agents.ts index 3a2d11e574d..e05270bc240 100644 --- a/server/src/routes/agents.ts +++ b/server/src/routes/agents.ts @@ -1881,7 +1881,7 @@ export function agentRoutes( throw unprocessable("Enable Grok Bot, Assistant connections (MCP), and Paperclip Runner before connecting this agent.", { code: "paperclip_runner_grok_disabled" }); } else if (!([undefined, "codex", "opencode"].includes(asRecord(config)?.provider as string | undefined)) && !(asRecord(config)?.provider === "acpx" && asRecord(config)?.acpxAgent === "copilot") && - !["claude_local", "pi_local"].includes(agentHarnessType("paperclip_runner", asRecord(config) ?? {})) && + !["claude_local", "pi_local", "grok_local"].includes(agentHarnessType("paperclip_runner", asRecord(config) ?? {})) && experimental.enableNativeRunner !== true) { throw unprocessable( "Paperclip Runner is experimental and disabled on this instance.", @@ -1952,7 +1952,7 @@ export function agentRoutes( } if (["openai_dot", "muse"].includes(String(input.nextAdapterConfig.provider))) return input.nextAdapterConfig; const defaults = paperclipRunnerTransitionConfig(input.previousAdapterType, input.previousAdapterConfig.model, input.nextAdapterConfig.provider); - if (!["claude_local", "codex_local", "opencode_local", "pi_local"].includes(input.previousAdapterType) + if (!["claude_local", "codex_local", "opencode_local", "pi_local", "grok_local"].includes(input.previousAdapterType) && !isPaperclipRunnerProvider(input.nextAdapterConfig.provider)) { throw unprocessable("Select a Paperclip Runner provider before converting this agent."); } @@ -3096,7 +3096,8 @@ export function agentRoutes( throw unprocessable("Unknown Paperclip Runner provider"); } const modelAdapterType = type === "paperclip_runner" - ? provider === "acpx" || provider === "claude_managed" ? "claude_local" + ? provider === "acpx" && req.query.acpxAgent === "grok" ? "grok_local" + : provider === "acpx" || provider === "claude_managed" ? "claude_local" : provider === "opencode" ? "opencode_local" : provider === "aws_agentcore" ? type : "codex_local" : type; @@ -3107,7 +3108,10 @@ export function agentRoutes( const models = refresh ? await refreshAdapterModels(modelAdapterType) : await listAdapterModels(modelAdapterType); - res.json(models); + // The Legacy default marker is not an explicit native ACP model ID. + res.json(type === "paperclip_runner" && modelAdapterType === "grok_local" + ? models.filter(model => model.id !== "grok-build") + : models); }); router.get("/companies/:companyId/adapters/:type/detect-model", async (req, res) => { @@ -5427,11 +5431,14 @@ export function agentRoutes( const convertingPi = existing.adapterType === "pi_local" && patchData.adapterType === "paperclip_runner" && (requestedRunnerConfig.provider === undefined || (requestedRunnerConfig.provider === "acpx" && [undefined, "pi"].includes(requestedRunnerConfig.acpxAgent as string | undefined))); - if (runner !== undefined || convertingPi) { + const convertingGrok = existing.adapterType === "grok_local" && patchData.adapterType === "paperclip_runner" + && [undefined, "acpx"].includes(requestedRunnerConfig.provider as string | undefined) + && [undefined, "grok"].includes(requestedRunnerConfig.acpxAgent as string | undefined); + if (runner !== undefined || convertingPi || convertingGrok) { await assertCanConfigureAgentSetup(req, existing); const config = hasOwn(patchData, "adapterConfig") ? { ...(replaceAdapterConfig ? {} : existing.adapterConfig), ...asRecord(patchData.adapterConfig) } : existing.adapterConfig; Object.assign(patchData, await resolveNewAgentRunnerForCompany(db, existing.companyId, { - adapterType: convertingPi ? "pi_local" : typeof patchData.adapterType === "string" ? patchData.adapterType : existing.adapterType, + adapterType: convertingPi || convertingGrok ? existing.adapterType : typeof patchData.adapterType === "string" ? patchData.adapterType : existing.adapterType, adapterConfig: config, runner: runner ?? "paperclip", setupOnly: existing.lifecycleHolds?.includes("setup"), onConfigurationWarnings: warnings => { configurationWarnings = warnings; }, defaultEnvironmentId: hasOwn(patchData, "defaultEnvironmentId") ? patchData.defaultEnvironmentId as string | null : existing.defaultEnvironmentId, diff --git a/server/src/services/agent-runner-qualification.ts b/server/src/services/agent-runner-qualification.ts index 8533ed19452..0abe93d01d8 100644 --- a/server/src/services/agent-runner-qualification.ts +++ b/server/src/services/agent-runner-qualification.ts @@ -11,7 +11,7 @@ export async function qualifyAgentRunner( expectedEnvironmentId: string, pluginWorkerManager?: PluginWorkerManager, ): Promise { - if (!["claude_local", "codex_local", "paperclip_runner"].includes(agent.adapterType)) { + if (!["claude_local", "codex_local", "grok_local", "paperclip_runner"].includes(agent.adapterType)) { throw unprocessable("This agent does not support computer qualification."); } const { environment } = await resolveAgentRunnerEnvironmentForCompany(db, agent.companyId, agent.defaultEnvironmentId); diff --git a/server/src/services/agent-runner-selection.test.ts b/server/src/services/agent-runner-selection.test.ts index 042c6084f45..697aa4be978 100644 --- a/server/src/services/agent-runner-selection.test.ts +++ b/server/src/services/agent-runner-selection.test.ts @@ -78,7 +78,7 @@ describe("server-owned Codex runner selection", () => { expect(resolveNewAgentRunner(resolved)).toEqual(resolved); expect(input.adapterConfig).toHaveProperty("search", false); }); - it.each(["grok_local", "cursor", "process", "external"])("keeps %s default unchanged", adapterType => { + it.each(["cursor", "process", "external"])("keeps %s default unchanged", adapterType => { expect(resolveNewAgentRunner({ adapterType, target, adapterConfig: { custom: "preserved" } })).toEqual({ adapterType, adapterConfig: { custom: "preserved" } }); }); it.each(qualifiedTargets.filter(target => !(target.platform === "linux" && target.architecture === "arm64")))("defaults Pi to native on $platform/$architecture and round trips exact thinking", target => { @@ -468,13 +468,60 @@ describe("server-owned Codex runner selection", () => { }); }); +describe("Grok runner migration", () => { + const target = { driver: "sandbox", platform: "linux", architecture: "x64", libc: "glibc" as const, glibcVersion: "2.39" }; + beforeEach(() => { state.disabled = []; state.overridden.clear(); }); + it("selects Grok's native profile and preserves the model and account", () => { + const aiConnection = { mode: "fixed", connectionId: "grok-account" }; + const adapterConfig = { model: "grok-4.7", reasoningEffort: "high", command: "grok", aiConnection, cwd: "/workspace" }; + expect(paperclipRunnerProfileForHarness("grok_local")).toEqual({ provider: "acpx", acpxAgent: "grok" }); + expect(resolveNewAgentRunner({ adapterType: "grok_local", runner: "paperclip", adapterConfig, target })) + .toEqual({ adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", reasoningEffort: "high", disableWebSearch: true, aiConnection, cwd: "/workspace", acpxPermissionMode: "approve-all", lifecycleMode: "per_turn" } }); + }); + it.each([undefined, "", "grok-build"])("resolves legacy default model %s before native admission", model => { + expect(resolveNewAgentRunner({ adapterType: "grok_local", runner: "paperclip", target, adapterConfig: { model } }).adapterConfig) + .toMatchObject({ model: "grok-4.7", disableWebSearch: true }); + }); + it.each([undefined, true, false])("preserves effective legacy web search policy %s", disableWebSearch => { + const result = resolveNewAgentRunner({ adapterType: "grok_local", runner: "paperclip", target, adapterConfig: { disableWebSearch } }); + expect(result.adapterConfig.disableWebSearch).toBe(disableWebSearch ?? true); + }); + it.each([ + { platform: "darwin", architecture: "x64" }, + { platform: "linux", architecture: "arm64" }, + { platform: "win32", architecture: "x64" }, + ])("keeps unqualified Grok $platform/$architecture on Legacy", platform => { + expect(agentRunnerAvailability("grok_local", { ...target, ...platform }).supportedRunners).toEqual(["legacy"]); + }); + it("retains High effort when switching back and rejects restricted native permission loss", () => { + const selected = resolveNewAgentRunner({ adapterType: "grok_local", runner: "paperclip", target, + adapterConfig: { model: "grok-4.7", reasoningEffort: "high" } }); + expect(resolveNewAgentRunner({ ...selected, runner: "legacy", target })).toEqual({ + adapterType: "grok_local", adapterConfig: { model: "grok-4.7", reasoningEffort: "high", disableWebSearch: true }, + }); + expect(() => resolveNewAgentRunner({ ...selected, runner: "legacy", target, + adapterConfig: { ...selected.adapterConfig, acpxPermissionMode: "deny-all" } })).toThrow("cannot honor"); + }); + it.each(["modelReasoningEffort", "effort"])("preserves native %s when switching to Legacy", key => { + const saved = { adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", [key]: "high" } }; + expect(resolveNewAgentRunner({ ...saved, runner: "legacy", target })).toEqual({ + adapterType: "grok_local", adapterConfig: { model: "grok-4.7", reasoningEffort: "high", disableWebSearch: false }, + }); + }); + it.each([{ command: "custom-grok" }, { extraArgs: ["--unsafe"] }, { permissionMode: "dontAsk" }, { alwaysApprove: false }, { maxTurns: 5 }])("rejects incompatible settings %j without mutating them", adapterConfig => { + const before = structuredClone(adapterConfig); + expect(() => resolveNewAgentRunner({ adapterType: "grok_local", runner: "paperclip", adapterConfig, target })).toThrow("cannot honor"); + expect(adapterConfig).toEqual(before); + }); +}); + describe("Codex creation runner contract", () => { it("graduates qualified harnesses and retains saved provider identity", () => { expect(paperclipRunnerProfileForHarness("codex_local")).toEqual({ provider: "codex" }); expect(paperclipRunnerProfileForHarness("claude_local")).toEqual({ provider: "acpx", acpxAgent: "claude" }); expect(paperclipRunnerProfileForHarness("opencode_local")).toEqual({ provider: "opencode" }); expect(paperclipRunnerProfileForHarness("pi_local")).toEqual({ provider: "acpx", acpxAgent: "pi" }); - for (const harness of ["grok_local", "cursor", "process"]) expect(paperclipRunnerProfileForHarness(harness)).toBeUndefined(); + for (const harness of ["cursor", "process"]) expect(paperclipRunnerProfileForHarness(harness)).toBeUndefined(); expect(agentHarnessType("paperclip_runner", { provider: "codex" })).toBe("codex_local"); expect(agentHarnessType("paperclip_runner", { provider: "acpx", acpxAgent: "claude" })).toBe("claude_local"); expect(agentHarnessType("paperclip_runner", { provider: "unknown" })).toBe("unknown"); diff --git a/server/src/services/agent-runner-selection.ts b/server/src/services/agent-runner-selection.ts index 77ecf01cfe7..a2c477ad803 100644 --- a/server/src/services/agent-runner-selection.ts +++ b/server/src/services/agent-runner-selection.ts @@ -1,7 +1,8 @@ import type { Db } from "@paperclipai/db"; -import { agentHarnessType, isEnvironmentDriverSupportedForAdapter, paperclipRunnerSupportsPlatform, type AgentRunnerChoice, type AgentRunnerAvailability } from "@paperclipai/shared"; +import { agentHarnessType, paperclipRunnerProfileForHarness, isEnvironmentDriverSupportedForAdapter, paperclipRunnerSupportsPlatform, type AgentRunnerChoice, type AgentRunnerAvailability } from "@paperclipai/shared"; import { normalizeLegacyRunnerProvider, PAPERCLIP_RUNNER_DEFAULT_MODELS } from "@paperclipai/adapter-utils"; import { resolveClaudeModel } from "@paperclipai/adapter-claude-local"; +import { DEFAULT_GROK_LOCAL_MODEL } from "@paperclipai/adapter-grok-local"; import { findActiveServerAdapter, hasActiveAdapterOverride, waitForExternalAdapters } from "../adapters/registry.js"; import { getDisabledAdapterTypes } from "./adapter-plugin-store.js"; import { claudeRunnerUnsupportedSettings, PaperclipRunnerProviderProfileError, resolvePaperclipRunnerNativeProviderInput } from "./native-runtime/provider-profile.js"; @@ -30,7 +31,7 @@ export interface AgentRunnerSelection { function runnerAdaptersAvailable(harness: string, driver: string): boolean { const disabled = new Set(getDisabledAdapterTypes()); - return ["codex_local", "claude_local", "opencode_local", "pi_local"].includes(harness) + return !!paperclipRunnerProfileForHarness(harness) && !disabled.has(harness) && !disabled.has("paperclip_runner") && !!findActiveServerAdapter(harness) && !!findActiveServerAdapter("paperclip_runner") && !hasActiveAdapterOverride(harness) && !hasActiveAdapterOverride("paperclip_runner") @@ -49,7 +50,7 @@ export function agentRunnerAvailability(harness: string, target: RunnerTarget = const supported = runnerAdaptersAvailable(harness, target.driver) && (target.driver !== "sandbox" || Boolean(target.platform && target.architecture && (target.platform !== "linux" || (target.libc === "glibc" && target.glibcVersion)))) - && (harness === "claude_local" + && (["claude_local", "grok_local"].includes(harness) ? !!target.platform && paperclipRunnerSupportsPlatform(harness, target.platform, target.architecture ?? "", target.libc, target.glibcVersion) : !target.platform || paperclipRunnerSupportsPlatform(harness, target.platform, target.architecture ?? "", target.libc, target.glibcVersion)); const automatic = supported && !(harness === "claude_local" && target.driver === "ssh" && target.platform === "darwin"); @@ -60,6 +61,23 @@ function incompatible(fields: string[]): never { throw unprocessable(`Paperclip Runner cannot honor these settings: ${fields.join(", ")}. Remove them or select Legacy runner in Advanced.`, { code: "agent_runner_config_incompatible", fields }); } +function resolveGrokRunnerConfig(config: Record): Record { + const unsupported = ["extraArgs", "args", "permissionMode", "maxTurns"] + .filter(key => config[key] !== undefined && config[key] !== "" && config[key] !== 0 && !(Array.isArray(config[key]) && config[key].length === 0)); + if (config.command && config.command !== "grok") unsupported.push("command"); + if (config.alwaysApprove !== undefined && config.alwaysApprove !== true) unsupported.push("alwaysApprove"); + if (config.disableWebSearch !== undefined && typeof config.disableWebSearch !== "boolean") unsupported.push("disableWebSearch"); + if (config.model !== undefined && typeof config.model !== "string") unsupported.push("model"); + if (unsupported.length) incompatible(unsupported); + const translated = { ...config }; + const effort = config.reasoningEffort ?? config.modelReasoningEffort ?? config.effort; + for (const key of ["command", "extraArgs", "args", "alwaysApprove", "maxTurns", "permissionMode", "reasoningEffort", "modelReasoningEffort", "effort"]) delete translated[key]; + return { ...translated, provider: "acpx", acpxAgent: "grok", disableWebSearch: config.disableWebSearch ?? true, + model: typeof config.model === "string" && config.model.trim() && config.model.trim() !== DEFAULT_GROK_LOCAL_MODEL ? config.model.trim() : "grok-4.7", + ...(effort === undefined || effort === "" ? {} : { reasoningEffort: effort }), + acpxPermissionMode: config.acpxPermissionMode ?? "approve-all", lifecycleMode: config.lifecycleMode ?? "per_turn" }; +} + function resolvePiRunnerConfig(config: Record): Record { const unsupported = ["extraArgs", "args", "modelReasoningEffort", "reasoningEffort", "effort"] .filter(key => config[key] !== undefined && config[key] !== "" && !(Array.isArray(config[key]) && config[key].length === 0)); @@ -150,12 +168,20 @@ function resolveAgentRunner(input: AgentRunnerSelection): { adapterType: string; let config = { ...input.adapterConfig }; const harness = agentHarnessType(adapterType, config); const native = adapterType === "paperclip_runner"; - if (["codex_local", "claude_local", "opencode_local", "pi_local"].includes(adapterType) && getDisabledAdapterTypes().includes(adapterType)) throw unprocessable(`${adapterType === "pi_local" ? "Pi" : adapterType === "opencode_local" ? "OpenCode" : adapterType === "claude_local" ? "Claude" : "Codex"} is disabled on this instance. Enable its adapter or choose another harness.`, { code: "agent_runner_unavailable" }); + if (paperclipRunnerProfileForHarness(adapterType) && getDisabledAdapterTypes().includes(adapterType)) throw unprocessable(`${adapterType === "grok_local" ? "Grok Build" : adapterType === "pi_local" ? "Pi" : adapterType === "opencode_local" ? "OpenCode" : adapterType === "claude_local" ? "Claude" : "Codex"} is disabled on this instance. Enable its adapter or choose another harness.`, { code: "agent_runner_unavailable" }); if (native && (input.runner === undefined || input.runner === "paperclip" || (harness !== "codex_local" && input.runner !== "legacy"))) return { adapterType, adapterConfig: normalizeLegacyRunnerProvider(config) }; const availability = agentRunnerAvailability(harness, input.target); const choice = input.runner ?? "auto"; const useNative = choice === "paperclip" || (choice === "auto" && availability.defaultRunner === "paperclip"); if (!useNative) { + if (harness === "grok_local") { + if (!native) return { adapterType, adapterConfig: config }; + if (config.acpxPermissionMode !== undefined && config.acpxPermissionMode !== "approve-all") incompatible(["acpxPermissionMode"]); + if (config.lifecycleMode !== undefined && config.lifecycleMode !== "per_turn") incompatible(["lifecycleMode"]); + const effort = config.modelReasoningEffort ?? config.reasoningEffort ?? config.effort; + for (const key of ["provider", "acpxAgent", "acpxPermissionMode", "lifecycleMode", "idleTimeoutMs", "modelReasoningEffort", "effort"]) delete config[key]; + return { adapterType: "grok_local", adapterConfig: { ...config, disableWebSearch: config.disableWebSearch ?? false, ...(effort === undefined ? {} : { reasoningEffort: effort }) } }; + } if (harness === "pi_local") { if (!native) return { adapterType, adapterConfig: config }; const thinking = config.thinking ?? config.piThinkingLevel ?? "low"; @@ -198,6 +224,15 @@ function resolveAgentRunner(input: AgentRunnerSelection): { adapterType: string; ? "Paperclip Runner is unavailable for this sandbox. Choose an immutable OCI image digest and run its environment connection probe to qualify the OS, architecture and libc, or select Legacy runner in Advanced." : "Paperclip Runner is unavailable for this harness and environment. Select Legacy runner in Advanced.", { code: "agent_runner_unavailable" }); if (native) return { adapterType, adapterConfig: normalizeLegacyRunnerProvider(config) }; + if (harness === "grok_local") { + config = resolveGrokRunnerConfig(config); + try { resolvePaperclipRunnerNativeProviderInput({ backend: "acpx_runtime", adapterConfig: config }); } + catch (error) { + if (error instanceof PaperclipRunnerProviderProfileError) throw unprocessable(error.message, { code: error.code }); + throw error; + } + return { adapterType: "paperclip_runner", adapterConfig: config }; + } if (harness === "pi_local") { config = resolvePiRunnerConfig(config); if (input.setupOnly && !config.model) return { adapterType: "paperclip_runner", adapterConfig: config }; @@ -283,7 +318,7 @@ function resolveAgentRunner(input: AgentRunnerSelection): { adapterType: string; } export async function resolveNewAgentRunnerForCompany(db: Db, companyId: string, input: AgentRunnerSelection & { defaultEnvironmentId?: string | null }) { - if (!["codex_local", "claude_local", "opencode_local", "pi_local"].includes(agentHarnessType(input.adapterType ?? "process", input.adapterConfig)) + if (!paperclipRunnerProfileForHarness(agentHarnessType(input.adapterType ?? "process", input.adapterConfig)) || input.runner === "legacy" || (input.adapterType === "paperclip_runner" && (input.runner === undefined || input.runner === "paperclip"))) return resolveNewAgentRunner(input); const target = await resolveAgentRunnerTargetForCompany(db, companyId, input.defaultEnvironmentId); return resolveNewAgentRunner({ ...input, target }); @@ -303,7 +338,7 @@ export async function resolveAgentRunnerTargetForCompany(db: Db, companyId: stri const qualified = await readSandboxPlatformQualification(db, environment, companyId); if (qualified) target = { driver, ...qualified }; } - if (driver === "ssh" && environment && ["codex_local", "claude_local", "opencode_local", "pi_local"].some(harness => runnerAdaptersAvailable(harness, driver))) { + if (driver === "ssh" && environment && ["codex_local", "claude_local", "opencode_local", "pi_local", "grok_local"].some(harness => runnerAdaptersAvailable(harness, driver))) { const parsed = await resolveEnvironmentDriverConfigForRuntime(db, companyId, environment); if (parsed.driver !== "ssh") throw unprocessable("The selected SSH environment is unavailable."); let output: string; diff --git a/server/src/services/built-in-agents.ts b/server/src/services/built-in-agents.ts index bdadbe6c1c5..c248c555580 100644 --- a/server/src/services/built-in-agents.ts +++ b/server/src/services/built-in-agents.ts @@ -8,7 +8,7 @@ import { readPaperclipSkillSyncPreference, writePaperclipSkillSyncPreference } f import { and, desc, eq, ne } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { activityLog, agents, builtInManagedResources, companies, issueThreadInteractions, issues, routines, routineTriggers } from "@paperclipai/db"; -import { agentHarnessType, agentRunner, syncRoutineVariablesWithTemplate, type AgentRunnerChoice } from "@paperclipai/shared"; +import { agentHarnessType, paperclipRunnerProfileForHarness, agentRunner, syncRoutineVariablesWithTemplate, type AgentRunnerChoice } from "@paperclipai/shared"; import type { Agent, Approval, CompanySkill, PermissionKey, Routine, RoutineTrigger, RoutineVariable } from "@paperclipai/shared"; import { conflict, HttpError, notFound, unprocessable } from "../errors.js"; import { logActivity } from "./activity-log.js"; @@ -1762,7 +1762,7 @@ export function builtInAgentService(db: Db) { const adapterConfig = resolvedInput.adapterConfig ?? existing.adapterConfig; assertAdapterAllowed(definition, adapterType, adapterConfig); const selectedHarness = agentHarnessType(adapterType, adapterConfig); - const sameQualifiedHarness = ["codex_local", "claude_local", "opencode_local", "pi_local"].includes(selectedHarness) + const sameQualifiedHarness = Boolean(paperclipRunnerProfileForHarness(selectedHarness)) && agentHarnessType(existing.adapterType, existing.adapterConfig) === selectedHarness; const runner = resolvedInput.runner ?? (sameQualifiedHarness && existing.metadata?.paperclipBuiltInSetupRequired !== true ? agentRunner(existing.adapterType) : undefined); diff --git a/server/src/services/company-portability.ts b/server/src/services/company-portability.ts index e6868da3827..dc7dca311d6 100644 --- a/server/src/services/company-portability.ts +++ b/server/src/services/company-portability.ts @@ -1,5 +1,5 @@ import { createAgentLifecycle } from "./agent-lifecycle.js"; -import { agentAppearanceSchema, agentHarnessType, agentRunner, type AgentRunnerChoice } from "@paperclipai/shared"; +import { agentAppearanceSchema, agentHarnessType, paperclipRunnerProfileForHarness, agentRunner, type AgentRunnerChoice } from "@paperclipai/shared"; import { resolveNewAgentRunnerForCompany } from "./agent-runner-selection.js"; import { createHash, randomUUID } from "node:crypto"; import { promises as fs } from "node:fs"; @@ -5307,7 +5307,7 @@ export function companyPortabilityService(db: Db, storage?: StorageService) { "OpenAI Dot is experimental and disabled on this instance.", { code: "paperclip_runner_dot_disabled" }, ); - } else if (config.provider !== "codex" && config.provider !== "opencode" && config.provider !== undefined && !["claude_local", "pi_local"].includes(agentHarnessType("paperclip_runner", config)) && experimental.enableNativeRunner !== true) { + } else if (config.provider !== "codex" && config.provider !== "opencode" && config.provider !== undefined && !["claude_local", "pi_local", "grok_local"].includes(agentHarnessType("paperclip_runner", config)) && experimental.enableNativeRunner !== true) { throw unprocessable( "Paperclip Runner is experimental and disabled on this instance.", { code: "paperclip_runner_rollout_disabled" }, @@ -5651,7 +5651,7 @@ export function companyPortabilityService(db: Db, storage?: StorageService) { runner = agentRunner(existingImportAgent.adapterType); } } - const retainedNative = runner === "paperclip" && ["codex_local", "claude_local", "opencode_local", "pi_local"].includes(selectedHarness) + const retainedNative = runner === "paperclip" && Boolean(paperclipRunnerProfileForHarness(selectedHarness)) ? [manifestAgent, existingImportAgent].find((agent) => agent?.adapterType === "paperclip_runner" && agentHarnessType(agent.adapterType, agent.adapterConfig) === selectedHarness) : undefined; diff --git a/server/src/services/heartbeat-runner-provider-config.test.ts b/server/src/services/heartbeat-runner-provider-config.test.ts index 8090eb3cc04..b209a0ff706 100644 --- a/server/src/services/heartbeat-runner-provider-config.test.ts +++ b/server/src/services/heartbeat-runner-provider-config.test.ts @@ -153,6 +153,21 @@ describe("Paperclip Runner native provider configuration", () => { expect(() => resolvePaperclipRunnerNativeProviderInput({ backend: "acpx_runtime", adapterConfig: { ...base, networkScope: "unrestricted" } })).toThrow("Legacy runner"); }); + it("projects Grok task model and effort while retaining the search policy and account", () => { + const base = { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", reasoningEffort: "high", disableWebSearch: true, aiConnection: "saved" }; + const projected = projectPaperclipRunnerTaskConfig("acpx_runtime", base, { model: "grok-next", effort: "medium", provider: "codex", disableWebSearch: false, aiConnection: "other" }); + expect(projected).toEqual({ ...base, model: "grok-next", reasoningEffort: "medium" }); + expect(resolvePaperclipRunnerNativeProviderInput({ backend: "acpx_runtime", adapterConfig: projected })) + .toMatchObject({ acpxAgent: "grok", model: "grok-next", grokEffort: "medium", grokDisableWebSearch: true }); + }); + + it("resolves the Grok task default marker before native model admission", () => { + const base = { provider: "acpx", acpxAgent: "grok", model: "grok-4.7", reasoningEffort: "high", disableWebSearch: true }; + const config = projectPaperclipRunnerTaskConfig("acpx_runtime", base, { model: "grok-build", reasoningEffort: "high" }); + expect(resolvePaperclipRunnerNativeProviderInput({ backend: "acpx_runtime", adapterConfig: config })) + .toMatchObject({ model: "grok-4.7", grokEffort: "high", grokDisableWebSearch: true }); + }); + it("uses an OpenCode task model without changing the Runner provider", () => { const projected = projectPaperclipRunnerTaskConfig( "opencode_server", diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index 032f3875580..72e3ec01264 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -717,7 +717,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele }); getNativeFreshSessionHandoff = nativeReviewRequest ? undefined : getFreshSessionHandoff; const nativeProviderConfig = nativeRuntimeResolution.profile.backend === "codex_app_server" - || (nativeRuntimeResolution.profile.backend === "acpx_runtime" && agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig)) === "claude_local") + || (nativeRuntimeResolution.profile.backend === "acpx_runtime" && ["claude_local", "grok_local"].includes(agentHarnessType(agent.adapterType, parseObject(agent.adapterConfig)))) || nativeRuntimeResolution.profile.backend === "opencode_server" || (nativeRuntimeResolution.profile.backend === "acpx_runtime" && parseObject(agent.adapterConfig).acpxAgent === "copilot") diff --git a/server/src/services/native-runtime/native-execution-input.test.ts b/server/src/services/native-runtime/native-execution-input.test.ts index f10000ce8c7..588b8fbe663 100644 --- a/server/src/services/native-runtime/native-execution-input.test.ts +++ b/server/src/services/native-runtime/native-execution-input.test.ts @@ -688,4 +688,24 @@ describe("Claude effort and master home-mode compatibility", () => { if (claudeEffort) expect(value.provider).toHaveProperty("claudeEffort", claudeEffort); expect(JSON.stringify(buildNativeModelEnvelope(value))).not.toContain("claudeEffort"); }); + it.each([ + ["isolated", undefined, undefined, "v5"], ["isolated", "high", true, "v10"], ["isolated", undefined, false, "v10"], + ] as const)("retains %s home mode with effort %s and search policy %s in %s", (homeMode, grokEffort, grokDisableWebSearch, version) => { + const value = buildNativeExecutionInput({ + companyId: "company", runId: "run", agentId: "agent", + issue: { id: "issue", identifier: "CLA-1", title: "Useful output", description: null, workMode: "standard" }, + taskPrompt: "Create the requested file.", + workspace: { id: "workspace", cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null }, + normalizedSessionId: null, provider: "acpx", acpxAgent: "grok", model: "grok-4.7", + acpxPermissionMode: "approve-all", homeMode, grokEffort, grokDisableWebSearch, + completionContract: { id: "contract", sha256: `sha256:${"a".repeat(64)}`, schemaVersion: "paperclip.run-result.v1", + contract: { revision: "1", objective: "Create a file", criteria: [{ id: "output", requirement: "Create the file" }] } }, + runtimeContext: nativeRuntimeContextFixture(), + }); + expect(value.schema).toBe(`paperclip.native-execution-input.${version}`); + if (version !== "v5") expect(value).toHaveProperty("homeMode", homeMode); + if (grokEffort) expect(value.provider).toHaveProperty("grokEffort", grokEffort); + if (grokDisableWebSearch !== undefined) expect(value.provider).toHaveProperty("grokDisableWebSearch", grokDisableWebSearch); + expect(JSON.stringify(buildNativeModelEnvelope(value))).not.toContain("grokEffort"); + }); }); diff --git a/server/src/services/native-runtime/native-execution-input.ts b/server/src/services/native-runtime/native-execution-input.ts index 974ea876de6..3457b5a6cf6 100644 --- a/server/src/services/native-runtime/native-execution-input.ts +++ b/server/src/services/native-runtime/native-execution-input.ts @@ -18,6 +18,7 @@ import type { NativeExecutionInputV6, NativeExecutionInputV8, NativeExecutionInputV9, + NativeExecutionInputV10, MuseBindingSnapshot, DotBindingSnapshot, NativeExecutionInputV7, @@ -108,6 +109,8 @@ export interface BuildNativeExecutionInput { acpxSessionMode?: "agent" | "plan" | "ask"; piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; model?: string | null; managedProfile?: Extract< NativeExecutionInputV5["provider"], @@ -138,9 +141,9 @@ export interface BuildNativeExecutionInput { } export function buildNativeExecutionInput(input: BuildNativeExecutionInput & { provider: "muse"; museBinding: MuseBindingSnapshot }): NativeExecutionInputV8; export function buildNativeExecutionInput(input: BuildNativeExecutionInput & { provider: "openai_dot" | "grok_bot"; dotBinding: DotBindingSnapshot }): NativeExecutionInputV6; -export function buildNativeExecutionInput(input: BuildNativeExecutionInput & { provider?: Exclude }): NativeExecutionInputV5 | NativeExecutionInputV7 | NativeExecutionInputV9; -export function buildNativeExecutionInput(input: BuildNativeExecutionInput): NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9; -export function buildNativeExecutionInput(input: BuildNativeExecutionInput): NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9 { +export function buildNativeExecutionInput(input: BuildNativeExecutionInput & { provider?: Exclude }): NativeExecutionInputV5 | NativeExecutionInputV7 | NativeExecutionInputV9 | NativeExecutionInputV10; +export function buildNativeExecutionInput(input: BuildNativeExecutionInput): NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9 | NativeExecutionInputV10; +export function buildNativeExecutionInput(input: BuildNativeExecutionInput): NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9 | NativeExecutionInputV10 { if (input.issue.workMode !== "standard" && input.issue.workMode !== "planning" && input.issue.workMode !== "ask") { throw new Error("native_execution_input_invalid: issue work mode must be standard, planning, or ask"); } @@ -254,9 +257,10 @@ export function buildNativeExecutionInput(input: BuildNativeExecutionInput): Nat const workflowCase = input.caseContext && previousCaseProjection ? projectCaseContinuation(input.caseContext, previousCaseProjection) : null; const prepared = { - schema: input.claudeEffort !== undefined ? "paperclip.native-execution-input.v9" + schema: input.grokEffort !== undefined || input.grokDisableWebSearch !== undefined ? "paperclip.native-execution-input.v10" + : input.claudeEffort !== undefined ? "paperclip.native-execution-input.v9" : input.homeMode === "preconfigured" ? "paperclip.native-execution-input.v7" : "paperclip.native-execution-input.v5", - ...(input.claudeEffort !== undefined || input.homeMode === "preconfigured" ? { homeMode: input.homeMode ?? "isolated" } : {}), + ...(input.grokEffort !== undefined || input.grokDisableWebSearch !== undefined || input.claudeEffort !== undefined || input.homeMode === "preconfigured" ? { homeMode: input.homeMode ?? "isolated" } : {}), ...((input.initialCommunicationGuidance || input.freshSessionHandoff) ? { initialCommunicationGuidance: [input.initialCommunicationGuidance, input.freshSessionHandoff].filter(Boolean).join("\n\n"), } : {}), @@ -341,6 +345,8 @@ export function buildNativeExecutionInput(input: BuildNativeExecutionInput): Nat ...(mode === undefined ? {} : { mode }), ...(piThinkingLevel === undefined ? {} : { piThinkingLevel }), ...(input.claudeEffort === undefined ? {} : { claudeEffort: input.claudeEffort }), + ...(input.grokEffort === undefined ? {} : { grokEffort: input.grokEffort }), + ...(input.grokDisableWebSearch === undefined ? {} : { grokDisableWebSearch: input.grokDisableWebSearch }), profile: { driverKind: acpxProfile!.driverKind, protocolVersion: acpxProfile!.protocolVersion, @@ -390,7 +396,7 @@ export function buildNativeExecutionInput(input: BuildNativeExecutionInput): Nat schema: "paperclip.native-execution-input.v6", provider: { kind: input.provider, model: null, binding: input.dotBinding }, workspace: { access: "none", cwd: null, repoUrl: null, repoRef: null, branchName: null }, session: { ...prepared.session, driverKind: input.provider === "grok_bot" ? "grok_bot_mcp" : "openai_dot_mcp", lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } - } : prepared) as NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9; + } : prepared) as NativeExecutionInputV5 | NativeExecutionInputV6 | NativeExecutionInputV7 | NativeExecutionInputV8 | NativeExecutionInputV9 | NativeExecutionInputV10; } diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 3ebaa15f773..d78b215d16c 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -11426,6 +11426,36 @@ describe("runnerd provider runtime wiring", () => { }); }); + it.each(["isolated", "preconfigured"].flatMap(homeMode => + [true, false].map(disableWebSearch => ({ homeMode, disableWebSearch })), + ))("pins Grok v10 permissions, effort and search policy at daemon startup: $homeMode $disableWebSearch", async ({ homeMode, disableWebSearch }) => { + const current = { ...execution, schema: "paperclip.native-execution-input.v10", homeMode, + provider: { kind: "acpx", agent: "grok", model: "grok-4.7", permissionMode: "approve-all", + grokEffort: "high", grokDisableWebSearch: disableWebSearch }, + session: { ...execution.session, driverKind: "acpx_runtime" }, + } as unknown as NativeExecutionInput; + const startup = { db: leaseDb(current), execution: current, + runnerInstanceId: `grok-v10-pinned-${homeMode}-${disableWebSearch}`, + managedAiCredentialHome: join(isolatedStateDirectory, "absent-grok-login") }; + await createRunnerdBackend({ ...startup, runnerEnvironment: { XAI_API_KEY: "synthetic-grok-api-key" } }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + expect(state.createTransport.mock.calls.at(-1)![0]).toMatchObject({ + provider: "acpx", acpxAgent: "grok", acpxPermissionMode: "approve-all", + acpxPermissionModePinned: true, grokEffort: "high", grokDisableWebSearch: disableWebSearch, homeMode, + }); + }); + + it("requires explicit Grok credentials before v10 daemon startup", async () => { + const current = { ...execution, schema: "paperclip.native-execution-input.v10", homeMode: "isolated", + provider: { kind: "acpx", agent: "grok", model: "grok-4.7", permissionMode: "approve-all", grokEffort: "high" }, + session: { ...execution.session, driverKind: "acpx_runtime" }, + } as unknown as NativeExecutionInput; + await expect(createRunnerdBackend({ db: leaseDb(current), execution: current, + runnerInstanceId: "grok-v10-missing-credential", runnerEnvironment: {}, + managedAiCredentialHome: join(isolatedStateDirectory, "absent-grok-login"), + })).rejects.toThrow("Grok subscription login is unavailable"); + }); + it("copies back and removes Grok credentials from the exact ACPX launch home", async () => { const priorHome = process.env.PAPERCLIP_HOME; const privateRoot = await realpath(isolatedStateDirectory); diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index cea0ac147cb..b1cb4ccdade 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -5871,6 +5871,8 @@ export function providerSessionIdentityFromDurableProviderState(input: { !acpxRecoveryModeMatches(input.execution.provider, descriptor.mode, identity.mode) || !acpxRecoveryPiThinkingMatches(input.execution.provider, descriptor.piThinkingLevel, identity.piThinkingLevel) || !acpxRecoveryClaudeEffortMatches(input.execution.provider, descriptor.claudeEffort, identity.claudeEffort) || + !acpxRecoveryGrokEffortMatches(input.execution.provider, descriptor.grokEffort, identity.grokEffort) || + !acpxRecoveryGrokWebSearchMatches(input.execution.provider, descriptor.grokDisableWebSearch, identity.grokDisableWebSearch) || !["approve-all", "approve-paperclip", "approve-reads", "deny-all"].includes( String(identity.permissionMode), ) || @@ -5985,6 +5987,19 @@ function acpxRecoveryClaudeEffortMatches(provider: unknown, projected: unknown, return expected === projected && expected === observed && (expected === undefined || ["low", "medium", "high", "xhigh", "max"].includes(String(expected))); } +function acpxRecoveryGrokEffortMatches(provider: unknown, projected: unknown, observed: unknown): boolean { + const config = record(provider); + const expected = config.grokEffort; + if (config.kind !== "acpx" || config.agent !== "grok") return expected === undefined && projected === undefined && observed === undefined; + return expected === projected && expected === observed && (expected === undefined || ["minimal", "low", "medium", "high", "xhigh"].includes(String(expected))); +} + +function acpxRecoveryGrokWebSearchMatches(provider: unknown, ...observed: unknown[]): boolean { + const config = record(provider), expected = config.grokDisableWebSearch; + return (expected === undefined || (config.kind === "acpx" && config.agent === "grok" && typeof expected === "boolean")) + && observed.every(value => value === expected); +} + function acpxRecoveryPiThinkingMatches(provider: NativeExecutionInput["provider"], ...observed: unknown[]): boolean { const expected = record(provider).piThinkingLevel; if (provider.kind === "acpx" && provider.agent === "pi") return ["off", "low", "high", "max"].includes(String(expected)) && observed.every(value => value === expected); @@ -6003,6 +6018,8 @@ export function providerSessionIdentityTransitionIsAllowed(input: { )) return false; if (!acpxRecoveryPiThinkingMatches(input.execution.provider, record(record(input.previous).providerSessionIdentity).piThinkingLevel, record(record(input.current).providerSessionIdentity).piThinkingLevel)) return false; if (!acpxRecoveryClaudeEffortMatches(input.execution.provider, record(record(input.previous).providerSessionIdentity).claudeEffort, record(record(input.current).providerSessionIdentity).claudeEffort)) return false; + if (!acpxRecoveryGrokEffortMatches(input.execution.provider, record(record(input.previous).providerSessionIdentity).grokEffort, record(record(input.current).providerSessionIdentity).grokEffort)) return false; + if (!acpxRecoveryGrokWebSearchMatches(input.execution.provider, record(record(input.previous).providerSessionIdentity).grokDisableWebSearch, record(record(input.current).providerSessionIdentity).grokDisableWebSearch)) return false; if (canonicalJson(input.previous) === canonicalJson(input.current)) { return true; } @@ -12124,6 +12141,8 @@ async function createRunnerdBackendWithinSessionClaim( acpxCandidateProfile: provider.agent === "pi" ? undefined : resolveAcpxQualification(provider, process.env), acpxPermissionMode: provider.permissionMode, acpxMode: provider.mode, piThinkingLevel: provider.piThinkingLevel, claudeEffort: "claudeEffort" in provider ? provider.claudeEffort : undefined, + grokEffort: "grokEffort" in provider ? provider.grokEffort : undefined, + grokDisableWebSearch: "grokDisableWebSearch" in provider ? provider.grokDisableWebSearch : undefined, acpxPermissionModePinned: true } : {}), ...(provider.kind === "opencode" ? { opencodePermissionMode: provider.permissionMode } : {}), runnerReconnectGraceMs: 120_000, runnerProcessReconnectGraceMs: null, @@ -13583,11 +13602,13 @@ async function createRunnerdBackendWithinSessionClaim( acpxMode: input.execution.provider.mode, piThinkingLevel: input.execution.provider.piThinkingLevel, claudeEffort: "claudeEffort" in input.execution.provider ? input.execution.provider.claudeEffort : undefined, + grokEffort: "grokEffort" in input.execution.provider ? input.execution.provider.grokEffort : undefined, + grokDisableWebSearch: "grokDisableWebSearch" in input.execution.provider ? input.execution.provider.grokDisableWebSearch : undefined, acpxPermissionModePinned: input.execution.schema === "paperclip.native-execution-input.v4" || input.execution.schema === "paperclip.native-execution-input.v5" || input.execution.schema === "paperclip.native-execution-input.v7" || - input.execution.schema === "paperclip.native-execution-input.v9", + input.execution.schema === "paperclip.native-execution-input.v9" || input.execution.schema === "paperclip.native-execution-input.v10", acpxRuntimeDirectory: remoteRunnerFilesystemRoot ? posix.join(remoteRunnerFilesystemRoot, "acpx") : resolve( diff --git a/server/src/services/native-runtime/plugin-runner-probe.ts b/server/src/services/native-runtime/plugin-runner-probe.ts index 62c3772d272..3b3a83e79e3 100644 --- a/server/src/services/native-runtime/plugin-runner-probe.ts +++ b/server/src/services/native-runtime/plugin-runner-probe.ts @@ -1,3 +1,4 @@ +import { resolveGrokDisableWebSearch } from "../../vendor/paperclip-runner/index.js"; import { randomUUID } from "node:crypto"; import { mkdtempSync, realpathSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; @@ -5,7 +6,7 @@ import { join } from "node:path"; import type { AdapterEnvironmentTestContext, AdapterEnvironmentTestResult } from "@paperclipai/adapter-utils"; import { resolvePaperclipRunnerPiThinkingLevel } from "@paperclipai/adapter-utils"; import { environmentTaskResultSchema } from "@paperclipai/plugin-sdk"; -import { probeNativeRunnerEnvironment, resolveClaudeEffort } from "../../vendor/paperclip-runner/index.js"; +import { probeNativeRunnerEnvironment, resolveClaudeEffort, resolveGrokEffort } from "../../vendor/paperclip-runner/index.js"; import { preparePluginRunnerCredentials } from "./plugin-runner-credentials.js"; import { createPluginRunnerTaskOwner } from "./plugin-runner-task.js"; import { resolvePaperclipRunnerProviderProfile } from "./provider-profile.js"; @@ -58,6 +59,7 @@ export async function probePluginRunnerEnvironment(context: AdapterEnvironmentTe ...(profile.provider === "acpx" ? { acpxAgent: profile.acpxAgent, acpxPermissionMode: "deny-all", acpxPermissionModePinned: true, ...(profile.acpxAgent === "pi" ? { piThinkingLevel: resolvePaperclipRunnerPiThinkingLevel("acpx", "pi", context.config.piThinkingLevel) } : {}), ...(profile.acpxAgent === "claude" ? { claudeEffort: resolveClaudeEffort("claude", context.config.modelReasoningEffort ?? context.config.reasoningEffort ?? context.config.effort) } : {}), + ...(profile.acpxAgent === "grok" ? { grokEffort: resolveGrokEffort("grok", context.config.modelReasoningEffort ?? context.config.reasoningEffort ?? context.config.effort), grokDisableWebSearch: resolveGrokDisableWebSearch("grok", context.config.disableWebSearch) } : {}), acpxCandidateProfile: profile.acpxAgent === "pi" ? undefined : resolveAcpxQualification({ kind: "acpx", agent: profile.acpxAgent, model: profile.model }, process.env) } : {}), prpIdentity: { runnerInstanceId: runnerId, environmentLeaseId: target.leaseId, runId: binding.runId, normalizedSessionId: binding.sessionId, turnId: `turn-${binding.runId}`, itemId: `item-${binding.runId}` }, diff --git a/server/src/services/native-runtime/provider-profile.ts b/server/src/services/native-runtime/provider-profile.ts index d217c3a05ca..046c737e5bd 100644 --- a/server/src/services/native-runtime/provider-profile.ts +++ b/server/src/services/native-runtime/provider-profile.ts @@ -1,5 +1,6 @@ +import { resolveGrokDisableWebSearch } from "../../vendor/paperclip-runner/index.js"; import { agentHarnessType, normalizeClaudeRunnerOptions } from "@paperclipai/shared"; -import { resolveClaudeEffort, resolveQualifiedAcpxProfile } from "../../vendor/paperclip-runner/index.js"; +import { resolveClaudeEffort, resolveGrokEffort, resolveQualifiedAcpxProfile } from "../../vendor/paperclip-runner/index.js"; import { isPaperclipRunnerProvider, PAPERCLIP_RUNNER_PERMISSION_CAPABILITIES, @@ -140,6 +141,8 @@ export type PaperclipRunnerNativeProviderInput = acpxSessionMode?: "agent" | "plan" | "ask"; piThinkingLevel?: "off" | "low" | "high" | "max"; claudeEffort?: "low" | "medium" | "high" | "xhigh" | "max"; + grokEffort?: "minimal" | "low" | "medium" | "high" | "xhigh"; + grokDisableWebSearch?: boolean; }; export class PaperclipRunnerProviderProfileError extends Error { @@ -210,7 +213,8 @@ export function projectPaperclipRunnerTaskConfig( // The server resolves a connection's model namespace after task overrides. // Carry that model into durable input without allowing routing to change the harness. const model = optionalString(managedModel) ?? optionalString(task.model); - const supportsEffort = backend === "codex_app_server" || (backend === "acpx_runtime" && agentHarnessType("paperclip_runner", base) === "claude_local"); + const grok = backend === "acpx_runtime" && agentHarnessType("paperclip_runner", base) === "grok_local"; + const supportsEffort = backend === "codex_app_server" || grok || (backend === "acpx_runtime" && agentHarnessType("paperclip_runner", base) === "claude_local"); const effortKey = supportsEffort ? ["modelReasoningEffort", "reasoningEffort", "effort"].find((key) => key in task) : undefined; @@ -220,7 +224,7 @@ export function projectPaperclipRunnerTaskConfig( delete config.effort; } if (model !== null) config.model = model; - if (effortKey !== undefined) config.modelReasoningEffort = task[effortKey]; + if (effortKey !== undefined) config[grok ? "reasoningEffort" : "modelReasoningEffort"] = task[effortKey]; return config; } @@ -445,6 +449,12 @@ export function resolvePaperclipRunnerProviderProfile( try { if (candidate === "acpx" && agentHarnessType("paperclip_runner", config) === "claude_local") resolveClaudeEffort("claude", config.modelReasoningEffort ?? config.reasoningEffort ?? config.effort); } catch (error) { throw new PaperclipRunnerProviderProfileError("paperclip_runner_claude_effort_invalid", error instanceof Error ? error.message : "Invalid Claude effort"); } + try { if (candidate === "acpx" && config.acpxAgent === "grok") resolveGrokEffort("grok", config.modelReasoningEffort ?? config.reasoningEffort ?? config.effort); } catch (error) { + throw new PaperclipRunnerProviderProfileError("paperclip_runner_grok_effort_invalid", error instanceof Error ? error.message : "Invalid Grok effort"); + } + try { if (candidate === "acpx" && config.acpxAgent === "grok") resolveGrokDisableWebSearch("grok", config.disableWebSearch); } catch (error) { + throw new PaperclipRunnerProviderProfileError("paperclip_runner_grok_search_invalid", error instanceof Error ? error.message : "Invalid Grok search policy"); + } try { resolvePaperclipRunnerPiThinkingLevel(candidate, config.acpxAgent, config.piThinkingLevel); } catch (error) { throw new PaperclipRunnerProviderProfileError("paperclip_runner_pi_thinking_invalid", error instanceof Error ? error.message : "Invalid Pi thinking level"); } @@ -566,7 +576,9 @@ export function resolvePaperclipRunnerProviderProfile( "Paperclip Runner ACPX requires a qualified agent profile.", ); } - const selectedModel = model || DEFAULT_ACPX_RUNNER_MODELS[acpxAgent] || ""; + const selectedModel = acpxAgent === "grok" && model === "grok-build" + ? DEFAULT_ACPX_RUNNER_MODELS.grok + : model || DEFAULT_ACPX_RUNNER_MODELS[acpxAgent] || ""; try { resolveQualifiedAcpxProfile(acpxAgent, selectedModel); } catch (error) { @@ -661,6 +673,7 @@ export function resolvePaperclipRunnerNativeProviderInput(input: { model: profile.model, acpxAgent: profile.acpxAgent, ...(profile.acpxAgent === "claude" ? { claudeEffort: resolveClaudeEffort("claude", config.modelReasoningEffort ?? config.reasoningEffort ?? config.effort) } : {}), + ...(profile.acpxAgent === "grok" ? { grokEffort: resolveGrokEffort("grok", config.modelReasoningEffort ?? config.reasoningEffort ?? config.effort), grokDisableWebSearch: resolveGrokDisableWebSearch("grok", config.disableWebSearch) } : {}), ...(profile.acpxAgent === "pi" ? { piThinkingLevel: resolvePaperclipRunnerPiThinkingLevel("acpx", "pi", config.piThinkingLevel) } : {}), ...(profile.acpxAgent === "cursor" ? { acpxSessionMode: resolvePaperclipRunnerCursorMode("acpx", "cursor", config.acpxSessionMode), diff --git a/server/src/vendor/paperclip-runner/index.ts b/server/src/vendor/paperclip-runner/index.ts index 9e6f34b2f1e..e12786d3d4a 100644 --- a/server/src/vendor/paperclip-runner/index.ts +++ b/server/src/vendor/paperclip-runner/index.ts @@ -38,6 +38,7 @@ export type { NativeExecutionInputV6, NativeExecutionInputV8, NativeExecutionInputV9, + NativeExecutionInputV10, MuseBindingSnapshot, DotBindingSnapshot, ExternalProviderPort, @@ -147,6 +148,7 @@ export const parsePaperclipQuestionResponse = runner.parsePaperclipQuestionResponse; export const resolveQualifiedAcpxProfile = runner.resolveQualifiedAcpxProfile; export const resolveClaudeEffort = runner.resolveClaudeEffort; +export const resolveGrokEffort = runner.resolveGrokEffort; export const QUALIFIED_ACPX_PROFILES = runner.QUALIFIED_ACPX_PROFILES; export const QUALIFIED_ACPX_VERSION = runner.QUALIFIED_ACPX_VERSION; export const CURSOR_DISTRIBUTION_PINS = runner.CURSOR_DISTRIBUTION_PINS; @@ -179,3 +181,5 @@ export const configuredEnvironmentProjection = runner.configuredEnvironmentProje export const configuredEnvironment = runner.configuredEnvironment; export const probeNativeRunnerEnvironment = runner.probeNativeRunnerEnvironment; + +export const resolveGrokDisableWebSearch = runner.resolveGrokDisableWebSearch; diff --git a/ui/src/components/AgentConfigForm.render.test.tsx b/ui/src/components/AgentConfigForm.render.test.tsx index 7b8037b275b..e6b2164137f 100644 --- a/ui/src/components/AgentConfigForm.render.test.tsx +++ b/ui/src/components/AgentConfigForm.render.test.tsx @@ -764,7 +764,7 @@ describe("AgentConfigForm environment selector", () => { expect(result.onSave).toHaveBeenCalledWith({ name: "Renamed saved agent" }); }); - it.each(["codex_local", "claude_local"])("upgrades a saved %s in one click with pending edits", async adapterType => { + it.each(["codex_local", "claude_local", "grok_local"])("upgrades a saved %s in one click with pending edits", async adapterType => { mockAdaptersApi.list.mockResolvedValueOnce([{ type: adapterType, supportedRunners: ["legacy", "paperclip"], defaultRunner: "paperclip" }]); const result = await renderForm([], { adapterType, adapterConfig: { model: "existing-model", effort: "high" } }); roots.push(result.root); @@ -976,8 +976,8 @@ describe("AgentConfigForm environment selector", () => { expect(result.container.textContent).toContain("is ignored. The agent can still run."); }); - it.each(["codex", "claude"].flatMap(harness => ["modelReasoningEffort", "reasoningEffort", "effort"].map(key => ({ harness, key }))))("hydrates saved effort and clears explicit Auto for $harness ($key)", async ({ harness, key }) => { - const profile = harness === "codex" ? { provider: "codex", model: "gpt-5.6-sol" } : { provider: "acpx", acpxAgent: "claude", model: "claude-opus-5-5" }; + it.each(["codex", "claude", "grok"].flatMap(harness => ["modelReasoningEffort", "reasoningEffort", "effort"].map(key => ({ harness, key }))))("hydrates saved effort and clears explicit Auto for $harness ($key)", async ({ harness, key }) => { + const profile = harness === "codex" ? { provider: "codex", model: "gpt-5.6-sol" } : { provider: "acpx", acpxAgent: harness, model: harness === "grok" ? "grok-4.7" : "claude-opus-5-5" }; const result = await renderForm([], { adapterType: "paperclip_runner", adapterConfig: { ...profile, [key]: "high" } }); roots.push(result.root); const effort = result.container.querySelector('[aria-label="Thinking effort"]'); @@ -1279,8 +1279,8 @@ describe("AgentConfigForm environment selector", () => { } }); - it("saves Grok 4.7 reasoning effort using the runtime key", async () => { - const result = await renderForm([], { adapterType: "grok_local", adapterConfig: { model: "grok-4.7", reasoningEffort: "high" } }); + it.each(["grok_local", "paperclip_runner"])("saves Grok 4.7 reasoning effort using the runtime key on %s", async adapterType => { + const result = await renderForm([], { adapterType, adapterConfig: { ...(adapterType === "paperclip_runner" ? { provider: "acpx", acpxAgent: "grok" } : {}), model: "grok-4.7", reasoningEffort: "high" } }); roots.push(result.root); const effort = [...result.container.querySelectorAll("button")].find(button => button.textContent?.trim() === "High")!; expect(effort).toBeTruthy(); diff --git a/ui/src/components/AgentConfigForm.tsx b/ui/src/components/AgentConfigForm.tsx index 1d7357b6f5d..4966526432c 100644 --- a/ui/src/components/AgentConfigForm.tsx +++ b/ui/src/components/AgentConfigForm.tsx @@ -918,7 +918,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { runnerSettingsTargetRef.current = runnerSettingsTarget; useEffect(() => () => { runnerSettingsTargetRef.current = ""; }, []); const showRunnerUpgrade = !isCreate - && ["codex_local", "claude_local"].includes(props.agent.adapterType) + && ["codex_local", "claude_local", "grok_local"].includes(props.agent.adapterType) && adapterType === props.agent.adapterType && !runnerDiscovery.isError && (runnerAvailability?.defaultRunner === "paperclip" || canQualifyRunnerUpgrade); @@ -1425,7 +1425,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { ? "mode" : adapterType === "opencode_local" ? "variant" - : adapterType === "grok_local" ? "reasoningEffort" + : harnessType === "grok_local" ? "reasoningEffort" : adapterType === "pi_local" ? "thinking" : "effort"; const thinkingEffortOptions = isCodexHarness @@ -1441,7 +1441,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) { ? kimiThinkingEffortOptions : adapterType === "pi_local" ? [{ id: "", label: "Auto" }, ...["off", "minimal", "low", "medium", "high", "xhigh"].map(id => ({ id, label: id }))] - : harnessType === "claude_local" || adapterType === "grok_local" + : harnessType === "claude_local" || harnessType === "grok_local" ? [{ id: "", label: "Auto" }, ...setupEfforts(harnessType, currentModelId).map((id) => ({ id, label: id === "xhigh" ? "X-High" : id[0].toUpperCase() + id.slice(1), @@ -1449,11 +1449,11 @@ export function AgentConfigForm(props: AgentConfigFormProps) { : claudeThinkingEffortOptions; const currentThinkingEffort = isCreate ? val!.thinkingEffort - : isCodexHarness || (adapterType === "paperclip_runner" && harnessType === "claude_local") + : isCodexHarness || (adapterType === "paperclip_runner" && ["claude_local", "grok_local"].includes(harnessType)) ? eff( "adapterConfig", thinkingEffortKey, - String(config[thinkingEffortKey] ?? config.reasoningEffort ?? config.modelReasoningEffort ?? config.effort ?? ""), + String(config.modelReasoningEffort ?? config.reasoningEffort ?? config.effort ?? ""), ) : adapterType === "cursor" ? eff("adapterConfig", "mode", String(config.mode ?? "")) diff --git a/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx b/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx index d7be4eb5099..1dc387d5036 100644 --- a/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx +++ b/ui/src/components/task-chat/ComposerRunSettingsPicker.tsx @@ -114,14 +114,15 @@ export function ComposerRunSettingsPicker({ const binding = aiRuntimeConnectionBindingSchema.safeParse(agent?.runtimeConfig?.aiConnection).data; const poolId = binding?.mode === "router" ? binding.connectionId : undefined; const copilotCatalog = copilotTaskModelCatalogOptions(agent); + const acpxAgent = agent?.adapterType === "paperclip_runner" && provider === "acpx" && typeof agent.adapterConfig.acpxAgent === "string" ? agent.adapterConfig.acpxAgent : undefined; const copilotCatalogScope = JSON.stringify(copilotCatalog ?? null); const requestCopilotModels = Boolean(copilotCatalog && open && view !== "agents" && !modelOptionsOverride); const { data: fetchedModels = [], isPending: modelsPending, refetch: refreshModels } = useQuery({ queryKey: agent && modelSupported - ? [...queryKeys.agents.adapterModels(companyId, agent.adapterType, agent.defaultEnvironmentId ?? null, provider), poolId ?? null, copilotCatalog ?? null] + ? [...queryKeys.agents.adapterModels(companyId, agent.adapterType, agent.defaultEnvironmentId ?? null, provider), poolId ?? null, copilotCatalog ?? null, acpxAgent ?? null] : ["agents", "composer-models", "none"], queryFn: () => agentsApi.adapterModels(companyId, agent!.adapterType, { - environmentId: agent!.defaultEnvironmentId ?? null, provider, poolId, ...copilotCatalog, + environmentId: agent!.defaultEnvironmentId ?? null, provider, poolId, acpxAgent, ...copilotCatalog, }), ...(copilotCatalog ? COPILOT_MODEL_DISCOVERY_QUERY_OPTIONS : {}), enabled: Boolean(agent && modelSupported && !modelOptionsOverride && (!copilotCatalog || requestCopilotModels)), diff --git a/ui/src/components/task-chat/composer-run-settings.test.ts b/ui/src/components/task-chat/composer-run-settings.test.ts index 9df5c74917b..a3a6d2e930f 100644 --- a/ui/src/components/task-chat/composer-run-settings.test.ts +++ b/ui/src/components/task-chat/composer-run-settings.test.ts @@ -23,6 +23,12 @@ describe("composer run settings", () => { expect(composerDefaultModel(agent("grok_local"))).toBe("grok-build"); expect(composerDefaultModel(undefined)).toBe(""); }); + it("offers native Grok effort and keeps its default model", () => { + const native = { ...agent("paperclip_runner", "acpx"), adapterConfig: { provider: "acpx", acpxAgent: "grok", reasoningEffort: "high" } }; + expect(composerDefaultModel(native)).toBe("grok-4.7"); + expect(composerEfforts(native, "grok-4.7", ["grok-4.7"])).toEqual(["low", "medium", "high", "xhigh"]); + expect(readComposerRunSettings({ adapterConfig: native.adapterConfig }, native.adapterType)).toMatchObject({ effort: "high" }); + }); it("shows only effort levels known for the selected harness and model", () => { expect(composerEfforts(agent("codex_local"), "gpt-6-astra", [])).toContain("ultra"); expect(composerEfforts(agent("paperclip_runner", "codex"), "gpt-6-astra", [])).toContain("ultra"); diff --git a/ui/src/components/task-chat/composer-run-settings.ts b/ui/src/components/task-chat/composer-run-settings.ts index a33b338a42a..43d2fcaed7b 100644 --- a/ui/src/components/task-chat/composer-run-settings.ts +++ b/ui/src/components/task-chat/composer-run-settings.ts @@ -45,6 +45,7 @@ export function composerDefaultModel(agent: Agent | undefined): string { if (agent.adapterType === "paperclip_runner") { const provider = composerCatalogProvider(agent); if (provider === "codex" || provider === "opencode") return resolvePaperclipRunnerModel(provider, config.model); + if (provider === "acpx" && config.acpxAgent === "grok") return "grok-4.7"; if (provider === "acpx" && config.acpxAgent !== "grok") return resolvePaperclipRunnerModel(provider, config.model); } return ""; @@ -72,7 +73,7 @@ export function composerEfforts(agent: Agent | undefined, model: string, catalog } if (!catalogIds.includes(effectiveModel)) return []; if (agentHarnessType(agent.adapterType, agent.adapterConfig) === "claude_local") return claudeLocalReasoningEffortsForModel(effectiveModel); - if (agent.adapterType === "grok_local") return grokLocalReasoningEffortsForModel(effectiveModel); + if (agentHarnessType(agent.adapterType, agent.adapterConfig) === "grok_local") return grokLocalReasoningEffortsForModel(effectiveModel); if (agent.adapterType === "pi_local") return ["off", "minimal", "low", "medium", "high", "xhigh"]; if (agent.adapterType === "kimi_local" && typeof agent.adapterConfig.engine === "string" && agent.adapterConfig.engine.trim().toLowerCase() === "cli" && modelSupportsEffort(effectiveModel)) return KIMI_SUPPORTED_EFFORTS; From 35820cf398d6241cd2baaf5b397bcc51cd002e78 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 15:41:07 -0500 Subject: [PATCH 08/41] fix(evals): qualify Runner model receipts with separate operating reconciliation (#17929) ## Thinking Path > - Paperclip must account for paid eval work. > - The maintained Runner workflow distributes one immutable controller manifest. > - Its cost gate currently requires vendor proofs for every operating charge. > - Model-spend qualification must also work under an explicit policy that keeps operating charges separate. > - This PR applies the shared model-readiness policy to the hosted controller. > - It retains every operating hold and still rejects missing model receipts and incomplete workers. ## Linked Issues or Issue Description Refs #17928, #17889. **What happened?** Runner qualification requires full-lifecycle vendor operating proofs even when the operator selects model-request readiness. That blocks the model-token qualification path. **Expected behavior** An explicit `overhead.accountingReadiness: "model_requests"` policy permits absent operating cost proofs. Supplied proofs still validate. Complete final model receipts are required. **Steps to reproduce** Prepare a two-worker hosted manifest with the explicit policy and no vendor cost proofs. Preserve missing-worker allocations, then complete both workers and inspect the private archive summary. A second test returns a provider refusal without terminal usage and requires failure with the full hold. ## What Changed - Apply the shared explicit policy to hosted capacity and observability cost-proof admission. - Preserve pricing windows, model bounds, immutable source/request/runtime identities, trace quotas and every allocation. - Reject zero or unresolved model requests in this policy, even if a request has a provisional bound. - Retain operating amounts and pending reconciliation through campaign and private archive summaries. - Document the separate model-readiness contract. Preserve strict behavior when the field is absent. ## Verification - Regression first failed at the original vendor-bound gate. - All 80 controlled-eval Node checks pass, including source and copied portable runtime checks, archived partial/missing workers, credential exclusion and model receipt failure. - Product readiness prerequisite #17928 has its own eval unit/typecheck and full CI checks. - No paid workflow, credential change or recurring activation is part of this source PR. ## Risks This opt-in policy does not prove a full-lifecycle operating-cost ceiling. The report retains operating holds and pending reconciliation. Default policies retain the existing vendor-proof gate. Source and worker identity checks remain required. A model refusal without final usage now fails readiness even if its reservation has a reviewed bound. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../docs/runner-protocol-live-evals.md | 22 +++++++++ .../scripts/controlled-evals/archive.mjs | 3 ++ .../controlled-evals/controller.test.mjs | 39 +++++++++++++++ .../scripts/controlled-evals/hosted.mjs | 6 ++- .../scripts/controlled-evals/hosted.test.mjs | 49 +++++++++++++++++++ .../scripts/controlled-evals/manifest.mjs | 8 ++- .../controlled-evals/portable.test.mjs | 2 +- .../scripts/controlled-evals/report.mjs | 6 ++- .../scripts/controlled-evals/run.mjs | 4 +- 9 files changed, 132 insertions(+), 7 deletions(-) diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index d6190a93b7e..478eaf869cc 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -930,3 +930,25 @@ portable runtime isolation, private archive replay, and complete/partial/empty campaign accounting. They do not establish current vendor prices, future retention charges, live Braintrust attribution or invoice settlement. Keep paid recurrence disabled until these independent gates are qualified. + + +### Token-spend qualification with separate operating reconciliation + +A policy may explicitly set `overhead.accountingReadiness: "model_requests"` +when model-token qualification is the operator's priority. In that mode an +absent campaign capacity proof and absent observability cost proof do not block +admission. Any supplied proofs must still validate. Current allowance/pricing +windows, trace quotas, immutable source/request/runtime bindings, complete +selection, per-cell allocations, request limits, no retries and the shared +campaign admission cap remain required. Model reservation proofs remain required. + +The final report retains the once-only operating allowance and labels operating +reconciliation pending. Missing, failed or canceled model evidence still holds +the full cell allocation and fails completion. This policy does not certify a +vendor operating-cost bound. Omit the field to retain full-lifecycle proof +requirements. Existing manifests and held amounts are not rewritten. + +For this policy version, `operatingReconciliation` is always `pending`; the +immutable controller summary has no operating invoice settlement step. Mixed +histories retain that warning when any campaign uses the policy. A shared +`accountingReadiness` label appears only when every campaign uses it. diff --git a/packages/paperclip-runner/scripts/controlled-evals/archive.mjs b/packages/paperclip-runner/scripts/controlled-evals/archive.mjs index 6e6276bf0bc..5c3fb90ca6d 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/archive.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/archive.mjs @@ -170,6 +170,9 @@ export async function summarizeControlledEvidence(bundles, options) { function history(campaigns) { const sum = field => usd(campaigns.reduce((total, campaign) => total + units(campaign[field]), 0n)); return { schema: 'paperclip.runner-eval.control-history/v1', costStatus: campaigns.some(c => c.costStatus === 'incomplete') ? 'incomplete' : 'provisional', + ...(campaigns.length && campaigns.every(c => c.accountingReadiness === 'model_requests') + ? { accountingReadiness: 'model_requests' } : {}), + ...(campaigns.some(c => c.operatingReconciliation === 'pending') ? { operatingReconciliation: 'pending' } : {}), knownUsd: sum('knownUsd'), heldUsd: sum('heldUsd'), committedUsd: sum('committedUsd'), operatingHeldUsd: sum('operatingHeldUsd'), incompleteAttempts: campaigns.reduce((total, c) => total + c.incompleteAttempts, 0), overBudget: campaigns.some(c => c.overBudget), campaigns }; } diff --git a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs index 8f4fa1cd46f..a5ac4e731af 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/controller.test.mjs @@ -787,3 +787,42 @@ test('retained failure codes distinguish native startup, child exit and invalid const artifact = await fixture(t, { missingRoute: true, requests: 0 }); assert.equal((await artifact.run()).report.failure, 'artifact_invalid'); }); + + +test('model readiness rejects a missing terminal model receipt even when its reservation has a reviewed bound', async t => { + const f = await fixture(t, { requests: 1 }); + f.manifest.overhead.accountingReadiness = 'model_requests'; + for (const component of f.manifest.overhead.components) delete component.provisionalBound; + await f.save(); + const result = await f.run(0, async () => Response.json({ error: 'synthetic refusal' }, { status: 429 })); + assert.equal(result.report.failure, 'request_accounting_unbounded'); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); + assert.equal(summary.heldUsd, '0.014000000'); + assert.equal(summary.operatingReconciliation, 'pending'); +}); + + +test('model readiness rejects zero model requests and keeps the full allocation', async t => { + const f = await fixture(t, { requests: 0 }); + f.manifest.overhead.accountingReadiness = 'model_requests'; + for (const component of f.manifest.overhead.components) delete component.provisionalBound; + await f.save(); + const result = await f.run(); + assert.equal(result.report.failure, 'request_accounting_unbounded'); + assert.equal(result.report.accounting.requests.length, 0); + const summary = await summarizeControlledCampaign(f.manifest); + assert.equal(summary.incompleteAttempts, 1); assert.equal(summary.heldUsd, '0.014000000'); +}); + +test('mixed-policy private histories keep the pending operating reconciliation warning', async t => { + const model = await fixture(t, { requests: 1 }), strict = await fixture(t, { requests: 1 }); + model.manifest.campaignId = 'model-readiness-history'; + model.manifest.overhead.accountingReadiness = 'model_requests'; + for (const component of model.manifest.overhead.components) delete component.provisionalBound; + await model.save(); await model.run(); await strict.run(); + const bundles = await Promise.all([model, strict].map(f => packControlledAttempt({ manifest: f.manifest, attemptId: 'attempt-0', environment: f.environment }))); + const history = await summarizeControlledEvidence(bundles); + assert.equal(history.incompleteAttempts, 0); assert.equal(history.operatingHeldUsd, '0.008000000'); + assert.equal(history.accountingReadiness, undefined); assert.equal(history.operatingReconciliation, 'pending'); +}); diff --git a/packages/paperclip-runner/scripts/controlled-evals/hosted.mjs b/packages/paperclip-runner/scripts/controlled-evals/hosted.mjs index 6b0f77a87b5..4a9f42cb95a 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/hosted.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/hosted.mjs @@ -23,7 +23,9 @@ export function prepareHostedManifest({ catalog, catalogSha256, prepared, policy || !Number.isSafeInteger(policy.maxCells) || policy.maxCells < 1 || policy.maxCells > 512 || !Array.isArray(policy.allocations) || !policy.allocations.length) throw new Error('A bounded hosted policy with tracing is required'); if (!/^[a-f0-9]{64}$/.test(policy.controllerLockSha256 ?? '')) throw new Error('Invalid hosted controller lock binding'); - costRuntime.reservationBound('operating_allowance', policy.capacityBound, hostedPolicyConfiguration(policy)); + if (policy.overhead?.accountingReadiness !== 'model_requests' || policy.capacityBound !== undefined) { + costRuntime.reservationBound('operating_allowance', policy.capacityBound, hostedPolicyConfiguration(policy)); + } fields(policy.observability, ['projectId', 'capture', 'nativeLimits']); const runId = environment.GITHUB_RUN_ID, paths = hostedPaths(runId); if (prepared.schema !== 'paperclip.runner-eval.prepared-requests/v1' || prepared.githubRunId !== runId || prepared.githubRunAttempt !== 1 @@ -65,7 +67,7 @@ export function prepareHostedManifest({ catalog, catalogSha256, prepared, policy models: policy.models, overhead: policy.overhead, observability: { ...policy.observability }, sourceRevision: prepared.sourceRevision, definitionsRevision: prepared.definitionsRevision, ...artifacts, outputDirectory: paths.evidence, hosted: { repository: environment.GITHUB_REPOSITORY, runId, runAttempt: 1, workflowRevision: environment.GITHUB_SHA, controllerLockSha256: policy.controllerLockSha256, policySha256: sha256(canonical(policy)) }, attempts }; - manifest.observability.provisionalBound = { ...policy.capacityBound, configurationSha256: sha256(canonical(observabilityBoundConfiguration(manifest))) }; + if (policy.capacityBound !== undefined) manifest.observability.provisionalBound = { ...policy.capacityBound, configurationSha256: sha256(canonical(observabilityBoundConfiguration(manifest))) }; const validated = validateManifest(manifest, now); assertHostedAdmission(manifest, attempts[0], { ...environment, PAPERCLIP_RUNNER_EVAL_EXECUTION_ID: attempts[0].executionId }); return validated; diff --git a/packages/paperclip-runner/scripts/controlled-evals/hosted.test.mjs b/packages/paperclip-runner/scripts/controlled-evals/hosted.test.mjs index d6d8742fd96..ba64159479d 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/hosted.test.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/hosted.test.mjs @@ -119,3 +119,52 @@ test('hosted CLI reports safe admission reasons without echoing private paths or assert.match(result.stderr, /A required hosted input file is missing/); assert.doesNotMatch(result.stderr + result.stdout, /SENSITIVE_SENTINEL/); }); + + +test('model-readiness policy retains operating holds without requiring vendor cost proofs', async t => { + const c = await campaign(t); + c.input.policy.overhead.accountingReadiness = 'model_requests'; + for (const component of c.input.policy.overhead.components) delete component.provisionalBound; + delete c.input.policy.capacityBound; + const result = c.build(); + assert.equal(result.manifest.overhead.accountingReadiness, 'model_requests'); + assert.equal(result.manifest.observability.provisionalBound, undefined); + Object.assign(c.f.manifest, result.manifest); await c.f.save(); + const missing = await summarizeControlledManifest(result.manifest, []); + assert.equal(missing.incompleteAttempts, 2); + assert.equal(missing.operatingHeldUsd, '0.004000000'); + Object.assign(c.f.environment, c.environment, { PAPERCLIP_RUNNER_EVAL_EXECUTION_ID: 'execution-0' }); + assert.equal((await c.f.run(0, () => success())).code, 0); + const first = await packControlledAttempt({ manifest: result.manifest, attemptId: 'attempt-0', environment: c.f.environment }); + Object.assign(c.f.environment, { PAPERCLIP_RUNNER_EVAL_EXECUTION_ID: 'execution-1' }); + assert.equal((await c.f.run(1, () => success())).code, 0); + const second = await packControlledAttempt({ manifest: result.manifest, attemptId: 'attempt-1', environment: c.f.environment }); + const complete = await summarizeControlledManifest(result.manifest, [first, second]); + assert.equal(complete.incompleteAttempts, 0); + assert.equal(complete.operatingHeldUsd, '0.004000000'); + assert.equal(complete.accountingReadiness, 'model_requests'); + assert.equal(complete.operatingReconciliation, 'pending'); + delete c.input.policy.overhead.accountingReadiness; + assert.throws(() => c.build(), /cost bound|operating bound/); +}); + + +test('model readiness still validates supplied operating and capacity proofs', async t => { + const c = await campaign(t), policy = c.input.policy; + policy.overhead.accountingReadiness = 'model_requests'; + for (const component of policy.overhead.components) { + component.provisionalBound.configurationSha256 = sha256(canonical({ accountingReadiness: 'model_requests', + reference: policy.overhead.reference, runner: policy.overhead.runner, verifiedAt: policy.overhead.verifiedAt, + expiresAt: policy.overhead.expiresAt, component: { component: component.component, maxUsd: component.maxUsd, reference: component.reference } })); + } + policy.capacityBound.configurationSha256 = sha256(canonical(hostedPolicyConfiguration(policy))); + assert.equal(c.build().manifest.overhead.accountingReadiness, 'model_requests'); + const mismatch = structuredClone(c.input); mismatch.policy.capacityBound.evidenceSha256 = '0'.repeat(64); + assert.throws(() => prepareHostedManifest(mismatch), /Observability capacity requires its operating bound evidence/); + const stale = structuredClone(c.input); stale.policy.capacityBound.configurationSha256 = '0'.repeat(64); + assert.throws(() => prepareHostedManifest(stale), /Reviewed cost bound configuration changed/); + const absent = structuredClone(c.input); + for (const component of absent.policy.overhead.components) delete component.provisionalBound; + absent.policy.capacityBound.configurationSha256 = sha256(canonical(hostedPolicyConfiguration(absent.policy))); + assert.throws(() => prepareHostedManifest(absent), /Observability capacity requires its operating bound evidence/); +}); diff --git a/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs index 825aab6fb2b..4149afe37da 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/manifest.mjs @@ -49,7 +49,7 @@ export function validateManifest(value, now = Date.now(), { admission = true } = validateHostedIdentity(value); const overhead = costRuntime.validateOperatingOverhead(value.overhead, value.maxUsd); if (admission && Date.parse(value.overhead.verifiedAt) > now || Date.parse(value.overhead.expiresAt) <= deadline - || value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds'); + || value.overhead.accountingReadiness !== 'model_requests' && value.overhead.components.some(component => !component.provisionalBound)) throw new Error('Controller requires current full-lifecycle operating bounds'); if (!Array.isArray(value.models) || !value.models.length || value.models.length > 16 || !Array.isArray(value.attempts) || !value.attempts.length || value.attempts.length > 1000) throw new Error('Invalid controlled campaign selection'); const prices = new Map(); @@ -141,7 +141,13 @@ function validateObservability(manifest) { || Object.keys(policy.capture).length !== 4 || Object.values(policy.capture).some(value => typeof value !== 'boolean') || canonical(Object.keys(policy.nativeLimits ?? {}).sort()) !== canonical(Object.keys(costRuntime.EVAL_OTLP_LIMITS).sort()) || Object.entries(costRuntime.EVAL_OTLP_LIMITS).some(([key, max]) => !Number.isSafeInteger(policy.nativeLimits[key]) || policy.nativeLimits[key] < 1 || policy.nativeLimits[key] > max)) throw new Error('Invalid controller observability allocation'); + if (manifest.overhead.accountingReadiness === 'model_requests' && policy.provisionalBound === undefined) return; const operating = manifest.overhead.components.find(item => item.component === 'observability'); if (!operating || policy.provisionalBound?.evidenceSha256 !== operating.provisionalBound?.evidenceSha256) throw new Error('Observability capacity requires its operating bound evidence'); costRuntime.reservationBound('operating_allowance', policy.provisionalBound, observabilityBoundConfiguration(manifest)); } + + +export function modelReadinessDefect(manifest, snapshot) { + return manifest.overhead.accountingReadiness === 'model_requests' && (!snapshot.requests.length || snapshot.unknownRequests > 0); +} diff --git a/packages/paperclip-runner/scripts/controlled-evals/portable.test.mjs b/packages/paperclip-runner/scripts/controlled-evals/portable.test.mjs index a4a648b4104..334b6a0a7ff 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/portable.test.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/portable.test.mjs @@ -92,7 +92,7 @@ test('copied controller runs without source dependencies, retains native evidenc const hosted = await exec(process.execPath, ['--test', '--test-reporter=tap', join(source, 'hosted.test.mjs')], { cwd: root, env: { ...environment, PAPERCLIP_CONTROLLER_TEST_RUNTIME: join(directory, 'controlled-evals') }, timeout: 30000, maxBuffer: 1_048_576, }); - assert.match(hosted.stdout, /# pass 4\b/); + assert.match(hosted.stdout, /# pass 6\b/); const cli = await exec(process.execPath, [join(directory, 'controlled-evals/archive.mjs'), 'report-attempts', join(root, 'empty-attempts')], { diff --git a/packages/paperclip-runner/scripts/controlled-evals/report.mjs b/packages/paperclip-runner/scripts/controlled-evals/report.mjs index cef7a45e583..877ce7d473f 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/report.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/report.mjs @@ -1,6 +1,6 @@ import { isEntrypoint } from './entrypoint.mjs'; import { join } from 'node:path'; -import { canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs'; +import { modelReadinessDefect, canonical, costRuntime, readJson, requestBindingSha256, sha256, units, usd, validateManifest } from './manifest.mjs'; import { inspectControlledObservability } from './observability.mjs'; import { attemptDirectory } from './run.mjs'; import { limitFor, readRegularFile } from './evidence-files.mjs'; @@ -79,7 +79,8 @@ export async function summarizeControlledCampaign(value, { filesByAttempt } = {} campaignId: manifest.campaignId, executionId: allocation.executionId, attemptId: allocation.attemptId, descriptorSha256: sha256(descriptorBytes) }) || artifact.attemptId !== allocation.attemptId || artifact.requestedModel !== allocation.model || snapshot.inflight || snapshot.conflictingRequests || snapshot.overBudget - || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) throw new Error('Incomplete controller evidence'); + || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests + || modelReadinessDefect(manifest, snapshot)) throw new Error('Incomplete controller evidence'); verified = true; } catch {} const cost = snapshot ? units(snapshot.knownUsd) : 0n; @@ -92,6 +93,7 @@ export async function summarizeControlledCampaign(value, { filesByAttempt } = {} unknownRequests: snapshot?.unknownRequests ?? null, boundedUnresolvedRequests: snapshot?.boundedUnresolvedRequests ?? null }); } return { schema: 'paperclip.runner-eval.control-campaign/v1', campaignId: manifest.campaignId, manifestSha256, + ...(manifest.overhead.accountingReadiness ? { accountingReadiness: manifest.overhead.accountingReadiness, operatingReconciliation: 'pending' } : {}), limitUsd: manifest.maxUsd, allocatedUsd, operatingHeldUsd: overheadUsd, knownUsd: usd(known), heldUsd: usd(committed - known), committedUsd: usd(committed), overBudget: committed > units(manifest.maxUsd), incompleteAttempts: attempts.filter(attempt => attempt.status === 'incomplete').length, costStatus: attempts.some(attempt => attempt.status === 'incomplete') || committed > units(manifest.maxUsd) ? 'incomplete' : 'provisional', attempts }; diff --git a/packages/paperclip-runner/scripts/controlled-evals/run.mjs b/packages/paperclip-runner/scripts/controlled-evals/run.mjs index 8125e6594b8..27659b161bf 100644 --- a/packages/paperclip-runner/scripts/controlled-evals/run.mjs +++ b/packages/paperclip-runner/scripts/controlled-evals/run.mjs @@ -1,3 +1,4 @@ +import { modelReadinessDefect } from './manifest.mjs'; import { isEntrypoint } from './entrypoint.mjs'; import { spawn } from 'node:child_process'; import { once } from 'node:events'; @@ -185,7 +186,8 @@ registerHooks({ resolve(specifier, context, nextResolve) { catch { cleanupVerified = false; failure ??= 'executable_cleanup_unverified'; } } const snapshot = ledger.snapshot(); - if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests) failure ??= 'request_accounting_unbounded'; + if (snapshot.overBudget || snapshot.conflictingRequests || snapshot.inflight || snapshot.unknownRequests !== snapshot.boundedUnresolvedRequests + || modelReadinessDefect(manifest, snapshot)) failure ??= 'request_accounting_unbounded'; const receiptHashes = {}; try { for (const request of snapshot.requests) receiptHashes[request.requestId] = sha256(await readRegularFile(join(root, 'receipts', `${request.requestId}.json`), limitFor(`receipts/${request.requestId}.json`))); From 77295faebc8fa0c3aa455523f2eb084974d8358d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:10:44 -0500 Subject: [PATCH 09/41] fix(server): recover approved tool actions from committed changes (#17454) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages agent work and its governed tool calls. > - A human approval can persist before the approving process dispatches the tool. > - Recovery currently scans approved requests on every maintenance tick. > - Completed and deliberately inert approvals do not owe another execution. > - Test-tab approval also needs a shared execution claim before recovery can safely race the foreground path. > - This PR wakes recovery from committed changes and retries only unresolved approvals. > - Idle history becomes quiet while existing approval and execution checks remain with the gateway. ## Linked Issues or Issue Description Refs #17453. This is the fifth slice of the recurring-server sleep migration, stacked on `codex/sleep-04-tool-review-expiry`. **Problem** Approved tool requests are scanned periodically even when no executable work remains. The Test-tab foreground path lacks the request claim needed to admit concurrent recovery safely. **Proposed Solution** Use the existing transaction-aware review notification boundary and shared deadline worker. Keep signed-payload interpretation, current policy and target checks, and provider dispatch in the gateway. Share a claim between Test-tab foreground and recovery, then persist both authoritative outcome rows atomically. **Alternatives Considered** A longer interval still scans idle history. A new durable queue would duplicate existing request/invocation state. Dispatching Test-tab approvals without a claim risks duplicate external calls. ## What Changed - Recheck both principals for delegated Test-tab approvals. Save their rate receipts in the same claim transaction, share only the same bucket, and roll back new charges on denial. - Preserve delegated gateway authorization and both issuer/target rate receipts inside the existing claimed-invocation transaction; retain the durable actor during approved replay. - Expire stale managed connector payloads before consuming approved-tool admission. - Persist current rate-policy charges and their invocation receipts atomically before dispatch, preserving exact approval claim fencing and read-only preflight. - Recover approved requests at startup and after committed review changes; retry only unresolved approvals. - Leave legacy agent approvals without execute-on-approve intent inert. Repair completed Test-tab history without another provider call. - Fence Test-tab execution and outcome settlement with an atomic claim shared by both callers. Revalidate the approved target and current policy before dispatch. - Keep provider calls outside database transactions. Retain quarantine for interrupted execution and unknown outcomes. - Add a concurrent, idempotent partial index for approved-request pagination. Reuse the parent's company-scoped review signals and shared scoped deadline worker. - Remove approved scanning from the aggregate tick. Preserve the public combined sweep and scheduler-disabled startup behavior. - Document producer ownership, quiet states, lifecycle behavior, and remaining recovery duties. ## Verification Latest published head: `23868c1751c100a27f58087f7ce16b93c52d4eba`, targeting master. The review-found delegated gateway pool deadlock is repaired by passing the admission transaction through token and issuer-authority reads. Foreground and recovery each reproduced the deadlock with a one-connection pool; all three constrained-pool cases, including revoked authority, now pass. All 81 HTTP integration cases pass. The historical foreground rate-slot regression also reproduced before its repair; verified preview hashes are accepted only by read-only preflight, while consuming admission still requires the complete identity under the exact claim. Five full suites passed 329 cases; the gateway service then passed all 141 cases after two additional mixed historical hash controls. Production types/build passed before the final normal integration of master `35820cf398` (including independent Grok/Runner changes); integration validation is running. Initial fixture/assertion errors and the prior E2E restoration/cancellation deadlock remain recorded in [the owning audit](doc/idle-approved-tool-recovery.md#gateway-admission-uses-its-transaction-connection). Fresh-head CI, Greptile 5/5, and passing Ponytail are still required before merge. - Master landing reconciliation at `8dfe3958db6e4e8a0b7499f3f5be0a396ca77361` after #17453 merged: production and server test files are byte-identical to the accepted 320-case repair. The landed migration journal prefix is unchanged, and #17928's independent eval reporting changes are preserved. Fresh CI and reviews on this exact head are required before merge. - Historical-identity repair published at `096b7c7198c3ca864aca9377e261cbaa0a8aad6f` (production/test source identical to integrated acceptance `81231824d773a79fc50c835eccf7d9b3977e79fc`): all 320 accepted gateway, policy, startup, preparation-timing and authenticated HTTP management cases pass. Server types and the production server build pass. Four real historical sensitive-argument regressions failed before repair; invalid-signature, foreign-hash and stale-claim controls remain denied without dispatch or identity changes. Both agent and Test-tab calls deliver the original signed arguments once, and new rate policies still apply. - The first integrated combined run passed 319 cases and hit a Slack bot ownership collision left in a reused HTTP test database. The guard correctly returned 409. The full 78-case HTTP suite passed on a fresh dedicated fixture without product or assertion changes. Earlier fixture-construction errors are distinguished from the four real regressions in the audit document. Fresh remote CI and reviews remain required. - Parent reconciliation at `736e7f56097055e21ce13b4e45d9033391483af8`: 299 cases pass (221 full gateway/policy/startup cases and 78 assembled-server connection management cases), with no skipped cases. Server types pass. This includes real delegated Test-tab approval admission for newly added shared and caller rate rules. Fresh remote CI and Greptile review remain required. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/bcca5b006976a471f4afd1f70c4031aa40a2b3f7/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `bcca5b006976a471f4afd1f70c4031aa40a2b3f7`. This remains a single-duty PR stacked on #17453. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/a021467f04d22be42ee5b0c87b6583b2300a62a9/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Notifications cover the existing database-owner, single-active-writer contract. Independent SQL writers and other replicas need an explicit wake path. - Unknown provider outcomes cannot be replayed safely. Failed result persistence retains an executing claim and the existing stale-execution quarantine. - Interrupted execution and terminal interaction projection repair remain periodic until their separate slices. This PR changes only approved recovery. - A pending-only review write can trigger one empty approved-recovery pass. Settled idle work schedules no further scan. - The partial index uses the existing top-level concurrent migration executor. Snapshot pruning follows repository policy. - Cloud sleep eligibility and external alarm coverage remain unchanged. Worker quiet is not proof of deployed instance suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-approved-tool-recovery.md | 381 ++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 115 ++- packages/db/src/client.ts | 1 + .../migrations/0384_funny_adam_warlock.sql | 1 + ...{0379_snapshot.json => 0384_snapshot.json} | 85 +- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/tool_access.ts | 1 + .../connection-management.integration.test.ts | 91 +- .../__tests__/helpers/runner-api-server.ts | 1 + .../server-startup-feedback-export.test.ts | 2 + .../tool-access-policy-service.test.ts | 120 ++- .../src/__tests__/tool-access-service.test.ts | 8 +- .../__tests__/tool-gateway-service.test.ts | 733 ++++++++++++++- server/src/index.ts | 11 +- server/src/services/approvals.ts | 17 +- .../services/delivery-work-notifications.ts | 1 + server/src/services/tool-access-policy.ts | 115 ++- .../src/services/tool-approval-execution.ts | 119 +++ server/src/services/tool-gateway.ts | 849 +++++++++--------- .../src/services/tool-review-recovery-work.ts | 33 + server/src/services/tool-review-recovery.ts | 119 +++ .../tool-review-work-notifications.ts | 5 +- server/src/services/tool-runtime-metrics.ts | 4 +- 23 files changed, 2313 insertions(+), 506 deletions(-) create mode 100644 doc/idle-approved-tool-recovery.md create mode 100644 packages/db/src/migrations/0384_funny_adam_warlock.sql rename packages/db/src/migrations/meta/{0379_snapshot.json => 0384_snapshot.json} (99%) create mode 100644 server/src/services/tool-approval-execution.ts create mode 100644 server/src/services/tool-review-recovery-work.ts create mode 100644 server/src/services/tool-review-recovery.ts diff --git a/doc/idle-approved-tool-recovery.md b/doc/idle-approved-tool-recovery.md new file mode 100644 index 00000000000..dbb7a8682c7 --- /dev/null +++ b/doc/idle-approved-tool-recovery.md @@ -0,0 +1,381 @@ +# Approved tool-action recovery + +Approved tool actions recover at startup and after committed review changes. +The aggregate maintenance tick no longer scans approved requests. The existing +request and invocation rows remain authoritative; this change adds no queue table. + +## Work and quiet states + +The gateway owns the approved-work predicate. A signed agent action with +`executionOnApprove: true` owes execution through the normal approval path. +A legacy signed action without that flag remains deliberately inert. A Test-tab +approval can owe execution even when no agent run exists. Recovery uses the saved +origin and current authorization/target checks; the scheduler sees only whether +another attempt is needed. + +Startup discovers companies with saved approved requests. Committed review +signals carry the affected company ID; subsequent discovery and recovery reads +filter by that company. The shared scoped worker coalesces changes and dispatches +one company per pass with a one-second minimum spacing. A company writing reviews +does not scan or lock another company's approved backlog. + +Recovery pages that company's approved requests in ID order using a partial index. It does not +let a full batch of unavailable approvals starve later work. A failed read or +unresolved executable approval retains a 30-second retry. New writes preserve +that company's retry deadline, while other companies can run independently. +A company pass still drains all its pages; this change does not cap provider +calls within that pass or alter the provider's existing execution timeout. Once a pass leaves no +such obligation, there is no recurring approved-recovery query. An empty system, +configured idle tools, completed history, and deliberately inert approvals are +quiet after reconciliation. + +## Producer and transaction ownership + +The existing tool-review notification boundary now signals both pending expiry +and approved recovery. This covers human review transactions, gateway approval +and accepted-interaction inference, administrative interaction resolution, review +queue invalidation, and the gateway's approval claims and managed-argument +invalidation. The notification is awaited before SQL in the existing transaction; +caller-owned outer transactions retain commit/rollback ownership. + +A pending-only change can cause one empty company-scoped discovery read. It does not +create a periodic scan. No per-route recovery bookkeeping or database listener +is added. Signals use the existing database owner's single-active-writer model. +Independent SQL writers or another process need a separate notification contract. + +## Single execution and outcome ownership + +Agent actions retain their existing approved-to-executing claim. Test-tab +foreground approval and recovery now share an equivalent claim before provider +dispatch. The invocation then starts only if the request still holds that claim. +Provider calls and target discovery remain outside SQL transactions. Current rate-policy admission and its receipt commit together in a short database transaction. + +The Test-tab invocation outcome and terminal request status commit together. +Claim timestamps fence late settlement. A later audit failure cannot overwrite +an already committed outcome. Recovery repairs legacy approved Test-tab requests +from a saved terminal invocation without decoding old arguments or calling the +provider. A legacy executing invocation moves to the existing interrupted-work +path; it is not dispatched again. + +If the provider may have run but its result cannot be persisted, the executing +claim remains authoritative. The existing stale-execution recovery quarantines +unknown outcomes. This slice does not promise exactly-once external effects or +replay an uncertain provider call. + +## Lifecycle and remaining duties + +The shared deadline worker coalesces notifications, preserves changes during a +pass, and retains skipped work across idle drain/resume. Startup reconstructs +saved approvals. Company deletion invalidates cached recovery and retries through +the same transactional notification boundary. Shutdown stops its coordinator registration. With heartbeat +scheduling disabled, startup keeps the existing one-off recovery behavior. + +Pending expiry is owned by the preceding slice. Interrupted execution and +interaction projection repair remain in the old sweep until their own migrations. +The public combined sweep retains all duties for manual/recovery callers. +No Cloud eligibility or external-wake proof changes here; worker quiet does not +establish deployed database or host suspension. + +## Verification + +Real PostgreSQL service/startup tests cover producer commit and rollback, +startup reconstruction, empty/configured/history quiet states, concrete retries, +first-write drain/resume, legacy inert approvals, and multi-page recovery. +Controlled Test-tab provider tests cover both foreground/recovery race winners, +completed legacy outcomes, interrupted legacy calls, target changes, and a failed +atomic outcome write followed by quarantine without another provider call. + +The service/startup selection passes 123 tests. The existing gateway acceptance +suite passes 92 tests. The strengthened paging and outcome-failure cases pass +again after their final edits. Workspace typecheck and build pass. Full CI and +review remain separate PR gates; no real external provider was invoked by these +fixtures. + +Review found that a one-shot success-write failure could fall into the provider +error handler and save a false failure on its second attempt. The added sequence- +based PostgreSQL fault reproduces that bug before the fix. Provider execution, +pre-dispatch validation, and outcome persistence now have separate error +boundaries. Both persistent and one-shot result-write failures retain the claim +for quarantine without repeating the call. The complete 123-test selection and +92 gateway acceptance cases pass after the repair, as do typecheck and build. + +The existing tool-access service regression now expects a completed Test-tab +request to be executed and verifies exactly one provider call before asking for +a new approval. Its full 469-test suite passes. The initial local command skipped +its database cases; the suite now honors its supplied disposable database through +the standard test-database guard independently of embedded Postgres support. + +Company-scoping regression: a real signed/reviewed action in company A previously +caused an unfiltered approved-request SELECT over company B's inert legacy action. +The captured-SQL test fails before this repair and passes with scoped reads. Added +lifecycle cases check retry preservation across committed writes, independent +company progress, and disarming after the real company-deletion producer. These +checks supplement the earlier execution, transaction, and paging evidence above. + +After company scoping, the complete gateway-service and startup selection passes +149 tests against a disposable PostgreSQL database, and server typecheck passes. +Stack integration, workspace checks, and fresh remote CI remain separate gates. + +## Preparation failures retain an approved Test-tab action + +A real PostgreSQL policy-read timeout reproduced a permanent-failure bug after +claiming a Test-tab action: the provider had never been called, but both request +and invocation were saved as failed, so approved recovery could not retry them. + +The preparation boundary now distinguishes explicit validation/access rejections +from unexpected dependency errors and transient HTTP errors. The existing +`tool-approval-execution` owner releases only its exact claim: invocation and request +are locked in the established order, their link/company/claim timestamp must +match, and the invocation must still await approval with no recorded start. +The request returns to `approved` with a transaction-aware notification. Signed +arguments and the human approval remain unchanged; the error propagates so the +existing recovery worker retains its retry obligation. + +This boundary ends before `runTestToolInvocation`. Provider dispatch and outcome +persistence keep their existing conservative no-replay behavior. A changed claim, +started invocation or prior-dispatch evidence cannot be released by this path. +If the requeue transaction itself fails, the error propagates and no terminal +failure is fabricated; existing interrupted-execution reconciliation remains the +fallback. This change does not infer replay safety from a failed requeue. + +Before the fix, both foreground and recovery timeout regressions failed. They +now observe the committed claim under a real PostgreSQL table lock, wait for +`55P03`, verify unchanged approval and no provider call, then prove automatic +recovery invokes the provider once and becomes quiet. Additional real-row cases +cover changed ownership, mismatched invocation, a started call, prior-dispatch +history, and outer-transaction rollback. A released claim cannot start execution. +All 117 gateway-service cases pass on the owning slice, including existing target +invalidation, provider-outcome persistence failures and unknown-outcome no-replay +cases. Integrated consumer checks, workspace verification and remote PR gates +must still be recorded after propagation. + + +## Agent-origin preparation and dispatch ownership + +Agent-origin approvals use the same pre-dispatch retry contract as Test-tab +approvals. Previously, a policy or identity dependency failure after claiming an +approved action permanently failed it; earlier issue/signing failures could +leave the claim executing. Preparation now has one owning error boundary around +issue, signature, identity, target, policy, managed-argument, and content checks. +Validation and nontransient access rejections retain their terminal handling. +Unexpected dependency failures and transient HTTP errors release only the exact, +still-undispatched claim to the existing approved queue and publish its existing +transactional work signal. No table, maintenance scan, or retry wrapper per +producer is added. + +`tool-approval-execution.ts` owns the shared claim token, release, and dispatch +transition. Its Test-tab claim/outcome functions remain specific to that consumer. +Both origins use the same invocation-then-request locks and verify invocation, +company, request linkage, claim timestamp, awaiting-approval state, and absence +of an earlier start before dispatch. An agent that loses its claim during +preparation cannot call the provider or overwrite its successor. The Test-tab +unapproved direct-call path retains its original behavior. + +Only preparation is retryable here. Recording the invocation start, provider +execution, and outcome persistence remain outside that catch. A provider error, +unknown outcome, or failed durable release does not establish replay safety. +Existing interruption handling remains the conservative fallback. Closed-issue +expiry, signature/target invalidation, identity authority, and policy rejection +remain enforced, including the final issue check before recording dispatch. + +Real PostgreSQL lock-timeout cases cover foreground approval and recovery: +observe the committed executing claim, block the policy read, then require the +same signed request to remain approved with no invocation start and zero provider +calls. Recovery after releasing the table lock executes once and becomes quiet. +A separate contention case replaces the claim while preparation is blocked and +proves the old owner cannot dispatch; the surviving approval later executes once. +A provider-error case proves that an already-started call is not requeued as a +preparation failure. The existing Test-tab timeout and ownership cases exercise +the other consumer of the shared boundary. + + +## New or edited rate rules during approval + +A reviewed invocation already has durable `consumedRateAdmissions` in its policy +explanation. The original read-only recheck retained unchanged rules correctly, +but new or edited rules only inspected their counters. Two approved actions could +therefore both execute against one available slot. The real recovery regression +failed for both a newly added limit and an increased limit before this repair. + +The policy service now owns `admitInvocation`: lock the company-scoped invocation, +validate its exact actor/task/tool/argument binding, recheck current policy, spend +only admissions not already recorded for that policy generation, and save the +receipt atomically. The gateway supplies its execution claim; that claim is locked +and checked by ID, invocation, company, executing status and timestamp. This also +supports historical approved requests whose invocation still says pending, without +accepting a stale claimant. Existing claim/start/settlement owners still control +provider execution. No queue table, periodic query, or per-route wrapper is added. + +Counter updates preserve the number of already spent slots when the limit changes. +A rejection by a later policy or failure to save the receipt rolls back new charges. +A retry of a committed admission retains the saved receipt, including concurrent +rechecks. Read-only approval preflight and browser authority probes stay read-only; +trust-rule usage is not counted again during execution revalidation. The policy +service and its metric helper accept only the database capabilities they use, so +transaction clients do not require a cast to a root connection. + +The owning gateway and policy suites pass all 181 cases against disposable real +PostgreSQL databases. Coverage includes two competing invocations, concurrent +retries of one invocation, changed limits, failed receipt writes, rejection after +an earlier rate charge, unchanged admissions, historical approvals, scope mismatch, +claim loss, quiet recovery and unknown provider outcomes. Server typechecking +passes after preparing current runner dependencies. + +Earlier verification failures remain part of this evidence: the first full run +was stopped because an old claim-loss fixture held the policy table while trying +to revoke the newly locked invocation. It now pauses discovery before admission +and retains the no-provider-dispatch assertion. A new receipt assertion was fixed +to permit the intended additional receipt while preserving original fields. The +next full run exposed an overly strict historical-approval check; exact execution- +claim validation fixed it. Typechecking then exposed root-client-only signatures, +which were narrowed to their actual database capabilities. The final complete +suites and TypeScript check pass; stack integration and current-head CI remain +separate gates. This proves worker and admission behavior, not deployed host sleep. + + +## Managed-argument expiry ordering + +Latest-head CI exposed a preparation-order regression in +`tool-gateway.test.ts`: a valid historical Shopify approval whose managed UCP +arguments were obsolete became `failed` instead of `expired`. The new consuming +admission compared the current invocation identity before the existing managed +argument expiry owner ran. It prevented dispatch, but bypassed the correct saved +expiry outcome. + +Approved execution now verifies target/signed/managed arguments and content +before consuming admission. Exact claim fencing, fresh policy checks, atomic +rate receipts, and the final issue-open check remain before dispatch. No check +was removed and stale payloads are not permitted to execute. + +The existing regression reproduced the CI failure on this owner in +`/tmp/sleep-approved-managed-expiry-red.log`. All 92 gateway acceptance cases +pass after the ordering change (`/tmp/sleep-approved-managed-expiry-full.log`), +including the historical approval's expired request, failed invocation with +`approvalState: expired`, and zero provider requests. Server typecheck passes +in `/tmp/sleep-approved-managed-expiry-types.log`. Latest-head remote CI and +combined-stack acceptance still need their own results. + +## Delegated Test-tab admission after policy changes + +A delegated Test-tab call has an originating agent but no gateway token. The +named-gateway admission path therefore did not evaluate its caller policy. If a +rate rule was added while approval was pending, approval charged only the target. +A shared company/tool bucket then rejected the caller against that same charge; +a caller-specific rule instead allowed execution without recording its charge. + +The Test-tab owner now supplies its caller/target policy evaluation to the existing +claim-transaction admission API. Both decisions use the claimed invocation's +receipts. The target can share the caller's charge only when the rule and bucket +match. Both receipts commit together. A denial rolls back new charges and receipts. +Management reauthorization and credential identity checks remain with their owners, +and provider execution stays outside the transaction. This adds no queue or timer. + +Three actual HTTP/PostgreSQL cases cover a newly added shared rate rule, separate +caller and target rules, and a target denial after caller admission. The unchanged +implementation failed the first two cases: the shared rule rejected execution, +and separate rules left only one charge. The denial case already passed. After +the repair, all three pass, including duplicate approval, both saved receipts, +one provider execution, and rejection of a subsequent authorized call. + +The full connection-management suite passes all 78 cases against a fresh database. +All 92 existing gateway tests and the server production typecheck also pass. +Intermediate fixture failures remain accounted for: the new assertion initially +used a nonexistent counter count field instead of remaining; the subsequent-call +fixture first reused an ended run and then lacked responsible-user authority. +The final fixture uses a new live run with the original responsible user. A reused +full-suite database also retained a conflicting Slack bot identity; final acceptance +uses a fresh isolated database. These are local checks, not current-head CI or a +host-suspension qualification. + + +## Historical argument identity compatibility repair + +Earlier invocations stored the hash of their redacted argument preview. Modern +admission correctly requires the complete request identity, including a keyed hash +for sensitive arguments. The gateway already verified signed historical arguments, +but admission rejected their saved preview hash even with no rate policy. That +permanently failed an otherwise valid approval before the provider call. + +Both agent-origin and Test-tab preparation now bind the verified complete identity +for a historical invocation before current policy admission. Signature, reviewed +target and managed-argument checks still run first. The approval execution owner +performs the identity transition in an invocation-then-request lock transaction. +It requires the exact current claim, company and invocation relationship, an +undispatched awaiting-approval invocation, and the expected reviewed/preview hash. +Unrelated saved hashes are not accepted. Modern matching identities do not add +this transaction. Current policies and rate charges still apply afterward. + +Four real-PostgreSQL regressions reproduced the defect: valid sensitive historical +arguments, with and without a newly configured rate rule, through each of the two +origins. Four controls keep invalid signatures and unrelated invocation hashes +rejected without dispatch or identity changes. The provider receives the original +verified arguments exactly once, rather than their redacted preview. A separate +real-transaction case proves a replaced claim cannot change the saved identity. + +The first draft fixtures used a nonexistent request-level tool name and then +non-sensitive arguments whose preview and complete hashes were equal. Those +fixture failures are not credited as product regressions. The corrected four +positive cases fail before the repair and all eight parameter cases pass after. +The complete gateway, policy and startup selection passes 230 cases with no skips; +server types pass. The complete assembled-server connection-management suite also +passes all 78 cases, including delegated approval and rate-policy behavior. Fresh +remote CI and reviews remain required after publication. + + +The repair was then integrated with the master-based parent, including #17918's +tracing changes, at `81231824d773a79fc50c835eccf7d9b3977e79fc`. Server types and the +production server build pass. The gateway, policy, startup and preparation-timing +suites pass all 242 cases. The complete HTTP management suite passes all 78 cases +on a fresh dedicated database, for 320 passing cases across the accepted runs. + +The first combined integrated run passed 319 cases and failed the Slack setup +case because its reused HTTP database still held the previous run's bot identity. +The existing ownership guard correctly rejected the duplicate connection. A fresh +dedicated fixture resolved that environment collision; no product behavior or +assertion changed for the rerun. That failed run remains part of the evidence. + + +## Gateway admission uses its transaction connection + +Review of the master-targeted branch exposed a delegated gateway deadlock: the +admission transaction held the sole pool connection while issuer/token authority +reads used the root client. Foreground approval and approved recovery each failed +the real one-connection regression; a revoked-issuer control already passed. +The admission callback now receives its transaction and passes it through the +existing issuer authority checks. Both connection authorization checks and the +token read use that connection. Ordinary checks retain their root client, and +current authority, claim, company and rate enforcement remain in place. +The three constrained-pool cases pass, as do all 81 HTTP integration cases and +190 gateway, policy, startup and deadline consumer cases. The initial consumer +command selected an incorrect policy filename and omitted the gateway database +override; it was interrupted and is not counted as passing evidence. + +## Pending historical approvals retain their original admission + +A second review finding concerned the normal foreground approval gate, before +execution has claimed and rebound historical arguments. A real PostgreSQL +regression with signed sensitive arguments and an already-consumed limit of one +failed with `rate_limited`; the Test-tab and invalid-signature controls passed. +The gateway now supplies the preview digest only after verifying the signed +payload and matching either supported form of the review's canonical digest. Policy may match that digest +only in a read-only preflight, with every other invocation-context field and +stored rate-rule receipt still checked. The consuming admission continues to +require the complete keyed identity, rebound under the exact execution claim. +New or changed rate rules still apply. Historical recovery, invalid signatures, +foreign invocation hashes, stale claims, and provider argument preservation stay +covered. This adds no schema, recurring query or additional connection. + +The preceding CI head also failed E2E shard 3/8. Its failure is retained in the +validation record; local gateway checks do not erase it. Fresh-head CI and both +required automated reviews remain merge gates. + +The five full gateway/policy/startup/deadline suites passed 329 cases before the +additional mixed historical identity control. The first combined attempt is +retained: a Test-tab failure-result assertion incorrectly expected a thrown +error, and reusing the earlier HTTP database left formal-approval foreign keys +that prevented gateway acceptance setup. The assertion now checks the existing +failed-result contract, with unchanged no-provider, hash and counter checks; +the complete acceptance rerun used an independently migrated gateway database. +Server production typechecking and build passed. The earlier E2E failure was a +500 during the agent-chat project-plan fixture restoration, with a PostgreSQL +deadlock in cancellation; it remains a fresh-head CI gate. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 5bdacbcd130..b699f1b242d 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -18,7 +18,7 @@ Do not mark an item done from a timer removal or a green check alone. | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | -| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry implemented on slice 04; other review duties pending | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), historical argument identity repair verified and awaiting fresh CI/review; other review duties pending | | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | @@ -117,3 +117,116 @@ index, applies/replays its migration, and preserves approval-time rejection and separate approved/executing/projection recovery. Workspace typecheck/build and all 92 existing gateway acceptance tests also passed. The first acceptance command matched no files; the corrected `tool-gateway.test.ts` selection passed. Fresh PR CI/review remains required. + + +## Approved-review follow-up repairs (2026-10-26) + +Slice 05 preserves the rate slot admitted before a human review. Agent and +Test-tab foreground/recovery paths reuse only the exact invocation/context and +unchanged rate-policy admission while rechecking current policy. A one-call +limit must not reject the same approved operation against its own occupied slot. + +Recovery now locks and rereads invocation/request history before dispatch. +Finished records repair the request without provider replay; executing records +retain unknown-outcome recovery. Invalid signatures, unusable formal approvals, +and inconsistent invocation states settle once. Missing signing-key configuration +and temporary dependencies retain retries. Company mismatches never mutate the +foreign invocation. No new table or generic HTTP-error discard rule is introduced. + +Formal approvals awaiting human input have no recovery cadence. The existing +approval service publishes commit-aware review notifications for approve/reject +and cancellation. Its shared record owner retains the decision validation/CAS +contract; hire decisions remain with their lifecycle owner. Revision/resubmission +keeps the review parked, and startup rediscovers saved work. Budget approval writes +belong to budget incidents, not tool-gateway formal reviews. Company deletion +removes the dependent work. These exclusions come from the producer audit. + +Local evidence: the four rate-admission regressions, seven permanent-record +regressions, and four formal-approval commit/rollback regressions failed before +their respective repairs. The complete gateway, policy, approval-service and +signature suites pass (208 tests), and server typechecking passes. Configuration +failure, company boundaries, concurrent claims, provider-outcome persistence +failure, restart and paging remain covered. The paging fixture now has independent +invocations instead of 101 approvals sharing an already-expired invocation. +Full remote checks and downstream integration verification remain required; +this is not an all-stack acceptance or deployed-sleep claim. + +## Next owner audit: approved tool-action recovery + +The next branch is `codex/sleep-05-approved-tool-recovery`, based on pending +expiry PR #17453 at `ce6a260a83`. Keep this separate from interrupted execution +and terminal projection repair. The public combined sweep remains compatible. + +- `commitToolActionReview` and accepted-interaction inference in the gateway are + approved-state producers. Existing notification ownership from slice 04 can + carry the new duty without adding another wrapper around callers. +- Agent execute-on-approve uses an atomic approved-to-executing claim before + provider execution. Keep that claim and all current authorization, signed + argument, current-target, identity, and issue-state checks in the gateway. +- Approved history is not uniformly executable. Legacy signed arguments without + `executionOnApprove` are deliberately inert. The payload can be encrypted; + shared scheduling must not decode it or inspect private provider fields. +- Board Test-tab invocations are a separate execution origin. The normal fresh + approval calls `runApprovedTestInvocation`, but the already-approved branch + currently skips test-origin execution. Test execution can leave the request + approved while the invocation holds the completed result. Do not treat those + completed rows as perpetual unfinished work. +- Before driving test-origin recovery, establish the same single-execution + claim on both foreground and recovery paths. The existing test helper does + not claim the request before provider dispatch. Simply invoking it from a + new worker would risk duplicate effects during foreground/recovery races. +- Distinguish saved completed outcomes, never-dispatched approved work, and + interrupted execution with unknown outcomes. Known completed provider results + must not be converted to unknown-outcome failures because another projection + write was interrupted. Retain quarantine for actually uncertain calls. +- Concrete unresolved approval work may retain a retry; configured idle, + terminal invocation history, and deliberately inert legacy approvals must + not cause empty recurring scans. Preserve restart reconstruction and all + supported producer paths, including transitions that finish an approved row. + +This is an audit, not an implementation or readiness claim. Tests must exercise +real approval producers, foreground/recovery contention, provider failure, +commit/rollback, legacy inert approvals, completed Test-tab history, and actual +unfinished Test-tab recovery before the next PR is ready. + +Current parent CI note: #17450 at `c5c53fc43f` has Greptile 5/5. Its server shard +10/12 failed the unchanged login-cleanup consume test waiting for an idle hold. +All 16 login-cleanup tests passed locally on recheck. The run remains active; +an attempted single-job rerun was rejected as not rerunnable. Recheck after the +run finishes before requesting its one justified retry. Do not report it green. + + +Slice 05 implementation now separates approved recovery from the aggregate tick. +The existing review notification boundary wakes it after outer commit. Legacy +inert approvals and completed Test-tab history become quiet; unresolved work +retains a retry. Test-tab foreground and recovery share a claim and atomic outcome +settlement. No provider call occurs inside that transaction. See +[approved recovery](../idle-approved-tool-recovery.md). Local service/startup +verification passes 122 tests, gateway acceptance passes 92, and workspace +typecheck/build pass. Paging and outcome-failure tests passed again after their +final strengthening. Full CI/review is not yet complete. + +Parent CI follow-up: #17450's completed run was retried once for its unchanged +login-cleanup timeout after all 16 local tests passed. #17453 has a Runner +prepared-launch close failure. Seven focused local Runner cases also fail, so +this is under investigation rather than assumed flaky. No retry of that run has +been requested. Neither parent is reported green without fresh evidence. + + +Slice 05 is [#17454](https://github.com/paperclipai/paperclip/pull/17454). +Review identified a real one-shot outcome-write error path. A new regression +failed before the repair; result persistence now runs outside provider-error +handling. The repaired head passes 123 service/startup tests, 92 acceptance +tests, workspace typecheck and build. Fresh CI/review remains required. +#17450 now has passing checks after its single retry. The local Runner +reproduction for #17453 was missing the fake-provider binary: its durable state +showed ENOENT hidden by close failure. After building the fixture all seven +cases pass; one CI retry is running. + +#17453 now has passing checks after its single Runner retry. #17454's initial +server shard found an old expected approved state for a completed Test-tab call; +the updated regression requires executed and one provider call. All 469 +tool-access service tests pass after enabling the existing external database +path independently of embedded support. Its initial browser shard failed during +unrelated plan-task fixture company deletion with a concurrent activity-log FK +violation. Inspect fresh-head CI before deciding whether that needs a repair. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index df20aa973ab..b3d66e14855 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -503,6 +503,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0369_late_logan.sql", "0381_mean_scream.sql", "0382_narrow_the_executioner.sql", + "0384_funny_adam_warlock.sql", "0383_tool_review_company_expiry.sql", ]); diff --git a/packages/db/src/migrations/0384_funny_adam_warlock.sql b/packages/db/src/migrations/0384_funny_adam_warlock.sql new file mode 100644 index 00000000000..c4f62d14e82 --- /dev/null +++ b/packages/db/src/migrations/0384_funny_adam_warlock.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "tool_action_requests_approved_idx" ON "tool_action_requests" USING btree ("id") WHERE "tool_action_requests"."status" = 'approved'; diff --git a/packages/db/src/migrations/meta/0379_snapshot.json b/packages/db/src/migrations/meta/0384_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0379_snapshot.json rename to packages/db/src/migrations/meta/0384_snapshot.json index 98067609cb0..bb2ed3c9595 100644 --- a/packages/db/src/migrations/meta/0379_snapshot.json +++ b/packages/db/src/migrations/meta/0384_snapshot.json @@ -1,6 +1,6 @@ { - "id": "2a9439e3-4a2b-4a20-ab2b-2aee3db8cb03", - "prevId": "a0fcd9b8-f125-4985-8795-f567466b9aef", + "id": "4f96f70a-fd19-4d35-ad8a-8d5559b83cbd", + "prevId": "410457fc-266b-4bd0-8bc0-2b14b0634d07", "version": "7", "dialect": "postgresql", "tables": { @@ -52812,6 +52812,66 @@ "method": "btree", "with": {} }, + "tool_action_requests_pending_expiry_idx": { + "name": "tool_action_requests_pending_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_company_pending_expiry_idx": { + "name": "tool_action_requests_company_pending_expiry_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_approved_idx": { + "name": "tool_action_requests_approved_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'approved'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_invocation_idx": { "name": "tool_action_requests_invocation_idx", "columns": [ @@ -55222,6 +55282,21 @@ "method": "btree", "with": {} }, + "tool_gateway_sessions_expiry_idx": { + "name": "tool_gateway_sessions_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_gateway_sessions_run_idx": { "name": "tool_gateway_sessions_run_idx", "columns": [ @@ -55967,6 +56042,12 @@ "notNull": true, "default": "''" }, + "responsible_user_id": { + "name": "responsible_user_id", + "type": "text", + "primaryKey": false, + "notNull": false + }, "owner_note": { "name": "owner_note", "type": "text", diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 02023247485..6b79f74c167 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2675,6 +2675,13 @@ "when": 1793060791727, "tag": "0383_tool_review_company_expiry", "breakpoints": true + }, + { + "idx": 384, + "version": "7", + "when": 1793060791728, + "tag": "0384_funny_adam_warlock", + "breakpoints": true } ] } diff --git a/packages/db/src/schema/tool_access.ts b/packages/db/src/schema/tool_access.ts index e970c445b6c..f1e788626e3 100644 --- a/packages/db/src/schema/tool_access.ts +++ b/packages/db/src/schema/tool_access.ts @@ -836,6 +836,7 @@ export const toolActionRequests = pgTable( .where(sql`${table.status} = 'pending'`), index("tool_action_requests_company_pending_expiry_idx").on(table.companyId, table.expiresAt) .where(sql`${table.status} = 'pending'`), + index("tool_action_requests_approved_idx").on(table.id).where(sql`${table.status} = 'approved'`), index("tool_action_requests_invocation_idx").on(table.invocationId), index("tool_action_requests_issue_idx").on(table.companyId, table.issueId), ], diff --git a/server/src/__tests__/connection-management.integration.test.ts b/server/src/__tests__/connection-management.integration.test.ts index 8f091fa49d6..2a91b321306 100644 --- a/server/src/__tests__/connection-management.integration.test.ts +++ b/server/src/__tests__/connection-management.integration.test.ts @@ -1,10 +1,12 @@ +import { createToolGatewayService } from "../services/tool-gateway.js"; +import { commitToolActionReview } from "../services/tool-action-review.js"; import { googleWorkspaceSelectedScopes, defaultGitHubReviewPolicy, SLACK_CHAT_BOT_SCOPES, SLACK_BOT_TOOL_SCOPES } from "@paperclipai/shared"; import { randomUUID, generateKeyPairSync } from "node:crypto"; import { join } from "node:path"; import { createServer } from "node:http"; -import { and, eq } from "drizzle-orm"; +import { and, eq, sql } from "drizzle-orm"; import { beforeAll, afterAll, describe, expect, it, vi } from "vitest"; -import { activityLog, agentApiKeys, companies, companySecrets, companySecretVersions, chatConversations, chatDeliveries, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, toolPolicies, connectionGrants, connectionGrantDelegations, toolAccessAuditEvents, toolOauthStates, issueThreadInteractions, connectionIntentDeliveries, agents, companyMemberships, heartbeatRuns, issues, principalPermissionGrants, projects, toolConnections, toolMcpGateways, toolMcpGatewayTokens, toolActionRequests, toolCatalogEntries, toolInvocations, toolCallEvents, toolProfileBindings, plugins, adapterAuthSessions, environments, routines, chatGitHubReviews } from "@paperclipai/db"; +import { createDb, activityLog, agentApiKeys, companies, companySecrets, companySecretVersions, chatConversations, chatDeliveries, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, toolPolicies, toolRateLimitCounters, connectionGrants, connectionGrantDelegations, toolAccessAuditEvents, toolOauthStates, issueThreadInteractions, connectionIntentDeliveries, agents, companyMemberships, heartbeatRuns, issues, principalPermissionGrants, projects, toolConnections, toolMcpGateways, toolMcpGatewayTokens, toolActionRequests, toolCatalogEntries, toolInvocations, toolCallEvents, toolProfileBindings, plugins, adapterAuthSessions, environments, routines, chatGitHubReviews } from "@paperclipai/db"; import { findSatisfiedToolConnection, satisfiedConnectionIntentService } from "../services/satisfied-connection-intents.js"; import { connectionManagementService, continuingConnectionManagement } from "../services/connection-management.js"; import { redactRunnerApiPayloads } from "../services/native-runtime/runner-api-log-redaction.js"; @@ -234,7 +236,7 @@ describe("connection management with production default agents", () => { expect(bindings.map(binding => binding.targetId)).toEqual([f.agentId]); }); - it.each(["none", "other-agent", "other-agent-no-access", "other-agent-allow-policy", "other-agent-explicit-grant", "other-agent-caller-ask", "other-agent-caller-rate", "other-agent-shared-rate", "other-agent-credential", "other-agent-credential-none", "other-agent-caller-revoked", "private-status", "agent", "user", "cancelled", "project-block", "rate-limit", "agent-rate-approval", "authorized-policy-edit"] as const)("keeps delegated test policy and approval continuation bounded (%s)", async revocation => { + it.each(["none", "other-agent", "other-agent-no-access", "other-agent-allow-policy", "other-agent-explicit-grant", "other-agent-caller-ask", "other-agent-new-rate", "other-agent-new-shared-rate", "other-agent-new-rate-target-block", "other-agent-caller-rate", "other-agent-shared-rate", "other-agent-credential", "other-agent-credential-none", "other-agent-caller-revoked", "private-status", "agent", "user", "cancelled", "project-block", "rate-limit", "agent-rate-approval", "authorized-policy-edit"] as const)("keeps delegated test policy and approval continuation bounded (%s)", async revocation => { const f = await fixture(); let targetAgentId = f.agentId; if (revocation.startsWith("other-agent")) { @@ -247,7 +249,7 @@ describe("connection management with production default agents", () => { await lifecycle.process(targetAgentId); await vi.waitFor(async () => expect((await agentService(server.db).getById(targetAgentId))?.status).toBe("idle")); } - const allowed = revocation === "none" || revocation === "other-agent" || revocation === "other-agent-caller-ask"; + const allowed = revocation === "none" || revocation === "other-agent" || revocation === "other-agent-caller-ask" || ["other-agent-new-rate", "other-agent-new-shared-rate"].includes(revocation); let calls = 0; const remote = createServer(async (req, res) => { const chunks: Buffer[] = []; @@ -264,7 +266,7 @@ describe("connection management with production default agents", () => { const address = remote.address(); if (!address || typeof address === "string") throw new Error("Fixture did not listen"); const mcpUrl = `http://127.0.0.1:${address.port}/mcp`; - const freezeRateClock = ["other-agent-caller-rate", "other-agent-shared-rate", "rate-limit", "agent-rate-approval"].includes(revocation); + const freezeRateClock = revocation.startsWith("other-agent-new-") || ["other-agent-caller-rate", "other-agent-shared-rate", "rate-limit", "agent-rate-approval"].includes(revocation); if (freezeRateClock) vi.useFakeTimers({ toFake: ["Date"] }); try { const created = await f.request("POST", `/companies/${f.companyId}/tools/connections`, { name: "Governed test", transport: "mcp_remote", authKind: "none", transportConfig: { url: mcpUrl } }); @@ -381,6 +383,16 @@ describe("connection management with production default agents", () => { if (revocation === "cancelled") await server.db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, f.issueId)); if (revocation === "other-agent-caller-revoked") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "Caller access revoked", policyType: "block", selectors: { connectionId: created.body.id, agentId: f.agentId }, priority: 0 }); if (revocation === "project-block") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New project block", policyType: "block", selectors: { connectionId: created.body.id, projectId: f.projectId }, priority: 0 }); + if (revocation.startsWith("other-agent-new-")) { + await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New delegated test admission", policyType: "rate_limit", + selectors: { connectionId: created.body.id, ...(revocation === "other-agent-new-shared-rate" ? {} : { agentId: f.agentId }) }, + priority: -2, config: { limit: 1, windowSeconds: 60, keyBy: revocation === "other-agent-new-shared-rate" ? ["company", "tool"] : ["agent", "tool"] } }); + if (revocation === "other-agent-new-rate") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New test target admission", policyType: "rate_limit", + selectors: { connectionId: created.body.id, agentId: targetAgentId }, priority: -2, + config: { limit: 1, windowSeconds: 60, keyBy: ["agent", "tool"] } }); + if (revocation === "other-agent-new-rate-target-block") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New test target block", policyType: "block", + selectors: { connectionId: created.body.id, agentId: targetAgentId }, priority: -1 }); + } const cookie = await server.humanSession(f.userId); const approve = () => fetch(server.apiUrl + "/api" + approvalPath, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" }, body: JSON.stringify({ companyId: f.companyId }) }); const response = await approve(); @@ -390,6 +402,20 @@ describe("connection management with production default agents", () => { else expect(settled, JSON.stringify(settled)).toMatchObject({ status: allowed ? "succeeded" : "failed" }); expect(calls).toBe(allowed ? 1 : 0); if (allowed) { expect((await approve()).status).toBe(200); expect(calls).toBe(1); } + if (revocation.startsWith("other-agent-new-")) { + const counters = await server.db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.companyId, f.companyId)); + expect(counters).toHaveLength(allowed ? revocation === "other-agent-new-shared-rate" ? 1 : 2 : 0); + expect(counters.every(counter => counter.remaining === 0)).toBe(true); + if (allowed) { + expect(settled.policyExplanation?.consumedRateAdmissions).toHaveLength(1); + expect(settled.policyExplanation?.delegatedCallerConsumedRateAdmissions).toHaveLength(1); + const [nextRun] = await server.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, + status: "running", responsibleUserId: f.userId, contextSnapshot: { issueId: f.issueId } }).returning(); + const nextJwt = createLocalAgentJwt(f.agentId, f.companyId, "paperclip_runner", nextRun.id, f.userId); + expect(await f.request("POST", path, body, nextJwt)).toMatchObject({ status: 200, body: { decision: "off", error: { reasonCode: "rate_limited" } } }); + expect(calls).toBe(1); + } + } } finally { if (freezeRateClock) vi.useRealTimers(); remote.closeAllConnections(); await new Promise((resolve, reject) => remote.close(error => error ? reject(error) : resolve())); } }); @@ -1229,7 +1255,7 @@ describe("connection management with production default agents", () => { expect((await use()).status).toBeGreaterThanOrEqual(400); }); - it.each(["block", "ask", "ask-board", "ask-revoked", "ask-rate", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked", "rate", "shared-rate", "target-block", "credential"] as const)("intersects gateway issuer and target rules (%s)", async mode => { + it.each(["block", "ask", "ask-board", "ask-revoked", "ask-rate", "ask-new-rate", "ask-new-shared-rate", "ask-new-rate-target-block", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked", "ask-one-connection", "ask-one-connection-recovery", "ask-one-connection-revoked", "rate", "shared-rate", "target-block", "credential"] as const)("intersects gateway issuer and target rules (%s)", async mode => { const f = await fixture(); const cookie = await server.humanSession(f.userId); const createdTarget = await fetch(`${server.apiUrl}/api/companies/${f.companyId}/agents`, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" }, @@ -1302,16 +1328,59 @@ describe("connection management with production default agents", () => { if (!mode.startsWith("ask")) return; const [action] = await server.db.select().from(toolActionRequests).where(eq(toolActionRequests.invocationId, invocation.id)); expect(action.requestedByAgentId).toBe(mode === "ask-board" ? null : f.agentId); + if (mode.startsWith("ask-new-")) { + await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New admission after review", policyType: "rate_limit", + selectors: { connectionId: connection.body.id, ...(mode === "ask-new-shared-rate" ? {} : { agentId: f.agentId }) }, + priority: -2, config: { limit: 1, windowSeconds: 60, keyBy: mode === "ask-new-shared-rate" ? ["company", "tool"] : ["agent", "tool"] } }); + if (mode === "ask-new-rate") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New target admission", policyType: "rate_limit", + selectors: { connectionId: connection.body.id, agentId: target.id }, priority: -1, + config: { limit: 1, windowSeconds: 60, keyBy: ["agent", "tool"] } }); + if (mode === "ask-new-rate-target-block") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "New target block", policyType: "block", + selectors: { connectionId: connection.body.id, agentId: target.id }, priority: -1 }); + } if (mode === "ask-revoked") await server.db.insert(toolPolicies).values({ companyId: f.companyId, name: "Issuer blocked after review", policyType: "block", selectors: { connectionId: connection.body.id, agentId: f.agentId }, priority: -1 }); if (mode === "ask-token-revoked") await server.db.update(toolMcpGatewayTokens).set({ revokedAt: new Date() }).where(eq(toolMcpGatewayTokens.id, token.body.id)); if (mode === "ask-token-deleted") await server.db.delete(toolMcpGatewayTokens).where(eq(toolMcpGatewayTokens.id, token.body.id)); if (mode === "ask-token-scope") await server.db.update(toolMcpGatewayTokens).set({ allowedActions: ["tools/list"] }).where(eq(toolMcpGatewayTokens.id, token.body.id)); - if (mode === "ask-grant-revoked") await server.db.delete(principalPermissionGrants).where(and(eq(principalPermissionGrants.principalId, f.agentId), eq(principalPermissionGrants.permissionKey, "tools:manage_connections"))); - const approvalDenied = ["ask-revoked", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked"].includes(mode); - const response = await fetch(`${server.apiUrl}/api/tool-gateway/action-requests/${action.id}/approve`, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" }, body: JSON.stringify({ companyId: f.companyId }) }); - expect(response.status, JSON.stringify(await response.json())).toBe(approvalDenied ? 403 : 200); + if (mode === "ask-grant-revoked" || mode === "ask-one-connection-revoked") await server.db.delete(principalPermissionGrants).where(and(eq(principalPermissionGrants.principalId, f.agentId), eq(principalPermissionGrants.permissionKey, "tools:manage_connections"))); + const approvalDenied = ["ask-revoked", "ask-new-rate-target-block", "ask-token-revoked", "ask-token-deleted", "ask-token-scope", "ask-grant-revoked", "ask-one-connection-revoked"].includes(mode); + if (mode.startsWith("ask-one-connection")) { + const limited = createDb(server.databaseUrl, { maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + const service = createToolGatewayService(limited); + let timer: ReturnType | undefined; + let attempt: Promise | undefined; + try { + if (mode === "ask-one-connection-recovery") await commitToolActionReview(server.db, { + companyId: f.companyId, actionRequestId: action.id, decision: "approved", actor: { userId: f.userId }, + }); + attempt = mode === "ask-one-connection-recovery" ? service.recoverApprovedActionReviews(f.companyId) + : service.approveActionRequest({ companyId: f.companyId, actionRequestId: action.id, actor: { userId: f.userId } }); + void attempt.catch(() => undefined); + const completed = Promise.race([attempt, new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error("Single-connection gateway approval did not settle")), 5000); + })]); + if (approvalDenied) await expect(completed).rejects.toMatchObject({ status: 403 }); + else await completed; + expect((await limited.execute(sql`select 1 as available`))[0].available).toBe(1); + } finally { + clearTimeout(timer); await limited.$client.end({ timeout: 1 }); await attempt?.catch(() => undefined); + } + } else { + const response = await fetch(`${server.apiUrl}/api/tool-gateway/action-requests/${action.id}/approve`, { method: "POST", headers: { cookie, origin: server.apiUrl, "Content-Type": "application/json" }, body: JSON.stringify({ companyId: f.companyId }) }); + expect(response.status, JSON.stringify(await response.json())).toBe(approvalDenied ? 403 : 200); + } expect(calls).toBe(approvalDenied ? 0 : 1); - if (mode === "ask-rate") { + if (mode.startsWith("ask-new-")) { + const counters = await server.db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.companyId, f.companyId)); + expect(counters).toHaveLength(approvalDenied ? 0 : mode === "ask-new-shared-rate" ? 1 : 2); + expect(counters.every(counter => counter.remaining === 0)).toBe(true); + if (!approvalDenied) { + const [settled] = await server.db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id)); + expect(settled.policyExplanation?.consumedRateAdmissions).toHaveLength(1); + expect(settled.policyExplanation?.delegatedCallerConsumedRateAdmissions).toHaveLength(1); + } + } + if (["ask-rate", "ask-new-rate", "ask-new-shared-rate"].includes(mode)) { await rpc("tools/call", { name: tool.name, arguments: { another: true } }); expect(calls).toBe(1); } diff --git a/server/src/__tests__/helpers/runner-api-server.ts b/server/src/__tests__/helpers/runner-api-server.ts index e59e1bb61fe..40cbabf896e 100644 --- a/server/src/__tests__/helpers/runner-api-server.ts +++ b/server/src/__tests__/helpers/runner-api-server.ts @@ -60,6 +60,7 @@ export async function startRunnerApiTestServer(options: { const apiUrl = `http://127.0.0.1:${address.port}`; setupRunnerPrpWebSocketServer(http, { apiUrl }); return { + databaseUrl: temporary.connectionString, db, root, apiUrl, storage, async humanSession(userId: string) { if (!auth) throw new Error("Enable humanSessions for this fixture"); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 2ede1c6c85f..658f5c418ff 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -69,6 +69,8 @@ const { stop: vi.fn(async () => {}), }, toolGateway: { + recoverApprovedActionReviews: vi.fn(async () => ({ scanned: 0, retry: false })), + approvedActionReviewCompanies: vi.fn(async () => []), sweepActionReviews: vi.fn(async () => ({ scanned: 0 })), sweepExpiredActionReviews: vi.fn(async () => ({ scanned: 0 })), nextActionReviewExpiryAt: vi.fn(async () => null), diff --git a/server/src/__tests__/tool-access-policy-service.test.ts b/server/src/__tests__/tool-access-policy-service.test.ts index e74fa60b9c9..adf2fd5a374 100644 --- a/server/src/__tests__/tool-access-policy-service.test.ts +++ b/server/src/__tests__/tool-access-policy-service.test.ts @@ -1,5 +1,5 @@ import { createHash, randomUUID } from "node:crypto"; -import { eq } from "drizzle-orm"; +import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; import { activityLog, @@ -34,8 +34,9 @@ import { toolAccessService } from "../services/tool-access.js"; import { createToolGatewayService, ToolGatewayHttpError } from "../services/tool-gateway.js"; import { canonicalToolArguments, signToolArguments } from "../services/tool-content-guards.js"; -const externalTestDatabaseUrl = process.env.PAPERCLIP_TOOL_POLICY_TEST_DATABASE_URL; -const embeddedPostgresSupport = externalTestDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TOOL_POLICY_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; function legacySideEffectKey(invocation: typeof toolInvocations.$inferSelect, argumentsHash: string) { @@ -216,8 +217,11 @@ describeEmbeddedPostgres("tool access policy service", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - if (!externalTestDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-access-policy-"); - db = createDb(externalTestDatabaseUrl ?? tempDb!.connectionString); + if (externalDatabaseUrl) db = createDb(externalDatabaseUrl); + else { + tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-access-policy-"); + db = createDb(tempDb.connectionString); + } }, 20_000); afterEach(async () => { @@ -1115,11 +1119,103 @@ describeEmbeddedPostgres("tool access policy service", () => { expect(await svc.decide(input, { admittedInvocationId: invocation.id })).toMatchObject({ allowed: true }); expect(await db.select().from(toolRateLimitCounters)).toEqual(counters); expect((await db.select().from(toolRateLimitCounters))[0].remaining).toBe(0); - // Neither a new invocation nor an accidental consuming recheck can borrow - // another invocation's already-spent slot. + // A fresh invocation cannot borrow another invocation's already-spent slot. + // Execution revalidation retains its own durable admission. expect(await svc.decide(input, { admittedInvocationId: randomUUID() })).toMatchObject({ decision: "rate_limited" }); - expect(await svc.decide({ ...input, consumeRateLimit: true }, { admittedInvocationId: invocation.id })) - .toMatchObject({ decision: "rate_limited" }); + expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ allowed: true }); + expect(await db.select().from(toolRateLimitCounters)).toEqual(counters); + }); + + it("atomically charges a new rule once across concurrent retries of the same invocation", async () => { + const { company, input, invocation, svc } = await admittedRateLimitedInvocation(); + const added = await svc.createPolicy(company.id, { name: "New admission", policyType: "rate_limit", priority: 0, + selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } }); + const oldCounters = await db.select().from(toolRateLimitCounters); + expect(await svc.decide(input, { admittedInvocationId: invocation.id })).toMatchObject({ allowed: true }); + expect(await db.select().from(toolRateLimitCounters)).toEqual(oldCounters); + const results = await Promise.all(Array.from({ length: 4 }, () => svc.admitInvocation(input, invocation.id))); + expect(results.every(result => result.allowed)).toBe(true); + expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id))) + .toMatchObject([{ limit: 1, remaining: 0 }]); + const [saved] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id)); + expect(saved.policyExplanation?.consumedRateAdmissions).toHaveLength(2); + const { consumedRateAdmissions: oldAdmissions, ...oldExplanation } = invocation.policyExplanation!; + expect(saved.policyExplanation).toMatchObject(oldExplanation); + expect(saved.policyExplanation?.consumedRateAdmissions).toEqual(expect.arrayContaining(oldAdmissions as unknown[])); + }); + + it("allows only one of two admitted invocations to spend a newly added final slot", async () => { + const { company, input, invocation, svc } = await admittedRateLimitedInvocation(2); + const initial = await svc.decide({ ...input, consumeRateLimit: true }); + expect(initial.allowed).toBe(true); + const { invocation: second } = await svc.recordInvocation(input, initial); + await db.update(toolInvocations).set({ status: "executing" }).where(eq(toolInvocations.id, second.id)); + const added = await svc.createPolicy(company.id, { name: "One competing slot", policyType: "rate_limit", priority: 0, + selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } }); + const results = await Promise.all([invocation.id, second.id].map(id => svc.admitInvocation(input, id))); + expect(results.filter(result => result.allowed)).toHaveLength(1); + expect(results.filter(result => result.decision === "rate_limited")).toHaveLength(1); + expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id))) + .toMatchObject([{ limit: 1, remaining: 0 }]); + }); + + it("rolls back new rate charges when a later policy rejects the invocation", async () => { + const { company, input, invocation, svc } = await admittedRateLimitedInvocation(); + await svc.createPolicy(company.id, { name: "Before rejection", policyType: "rate_limit", priority: 0, + selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } }); + await svc.createPolicy(company.id, { name: "Later rejection", policyType: "block", priority: 10, + selectors: { toolName: "send_email" } }); + const counters = await db.select().from(toolRateLimitCounters); + expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ decision: "deny" }); + expect(await db.select().from(toolRateLimitCounters)).toEqual(counters); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id)))[0].policyExplanation) + .toEqual(invocation.policyExplanation); + }); + + it("rolls back rate charges when saving the admission receipt fails and admits a retry once", async () => { + const { company, input, invocation, svc } = await admittedRateLimitedInvocation(); + const added = await svc.createPolicy(company.id, { name: "Receipt failure", policyType: "rate_limit", priority: 0, + selectors: { toolName: "send_email" }, config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } }); + await db.execute(sql`create function sleep_admission_fail() returns trigger language plpgsql as $$ begin + raise exception 'receipt write unavailable'; end $$`); + await db.execute(sql`create trigger sleep_admission_fail before update of policy_explanation on tool_invocations + for each row execute function sleep_admission_fail()`); + try { + await expect(svc.admitInvocation(input, invocation.id)).rejects.toThrow(); + expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id))).toEqual([]); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, invocation.id)))[0].policyExplanation) + .toEqual(invocation.policyExplanation); + } finally { + await db.execute(sql`drop trigger sleep_admission_fail on tool_invocations`); + await db.execute(sql`drop function sleep_admission_fail()`); + } + expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ allowed: true }); + expect(await svc.admitInvocation(input, invocation.id)).toMatchObject({ allowed: true }); + expect(await db.select().from(toolRateLimitCounters).where(eq(toolRateLimitCounters.policyId, added.id))) + .toMatchObject([{ limit: 1, remaining: 0 }]); + }); + + it.each([2, 4])("preserves spent slots when a rate limit changes from three to %s", async limit => { + const { company, input, ratePolicy, svc } = await admittedRateLimitedInvocation(3); + await svc.updatePolicy({ companyId: company.id, policyId: ratePolicy.id, + body: { config: { limit, windowSeconds: 3600, keyBy: ["agent", "tool"] } } }); + const results = await Promise.all(Array.from({ length: limit }, () => svc.decide({ ...input, consumeRateLimit: true }))); + expect(results.filter(result => result.allowed)).toHaveLength(limit - 1); + expect(results.filter(result => result.decision === "rate_limited")).toHaveLength(1); + expect(await db.select().from(toolRateLimitCounters)).toMatchObject([{ limit, remaining: 0 }]); + }); + + it.each(["pending", "approved"] as const)("retains a waiting invocation's consumed slot without bypassing %s approval policy", async approvalState => { + const { company, input, invocation, svc } = await admittedRateLimitedInvocation(); + await db.update(toolInvocations).set({ status: "awaiting_approval", policyDecision: "require_approval", approvalState }) + .where(eq(toolInvocations.id, invocation.id)); + await db.insert(toolPolicies).values({ companyId: company.id, name: "Review admitted sends", + policyType: "require_approval", priority: 2, selectors: { toolName: "send_email" } }); + const counters = await db.select().from(toolRateLimitCounters); + expect(await svc.decide(input, { admittedInvocationId: invocation.id })) + .toMatchObject({ allowed: false, decision: "require_approval" }); + expect(await db.select().from(toolRateLimitCounters)).toEqual(counters); + expect(await svc.decide(input)).toMatchObject({ decision: "rate_limited" }); }); it("rechecks a newly added rate policy rather than borrowing an older policy's consumed slot", async () => { @@ -1172,15 +1268,19 @@ describeEmbeddedPostgres("tool access policy service", () => { { actorType: "system" }, { actorId: randomUUID() }, { agentId: null }, { runId: null }, { issueId: null }, { gatewayId: null }, { applicationId: null }, { connectionId: null }, { catalogEntryId: null }, { toolName: "different_tool" }, { providerType: null }, { applicationKey: null }, { upstreamToolName: null }, { argumentsHash: "different" }, - { status: "authorized" }, { policyDecision: "deny" }, { policyDecision: "require_approval", approvalState: "pending" }, + { status: "authorized" }, { status: "succeeded" }, { status: "awaiting_approval" }, + { status: "awaiting_approval", policyDecision: "require_approval", approvalState: "rejected" }, { policyDecision: "deny" }, { policyDecision: "require_approval", approvalState: "pending" }, ]; for (const mutation of mutations) { await db.update(toolInvocations).set(mutation).where(eq(toolInvocations.id, invocation.id)); expect(await svc.decide(input, options), JSON.stringify(mutation)).toMatchObject({ decision: "rate_limited" }); + expect(await svc.admitInvocation(input, invocation.id), JSON.stringify(mutation)).toMatchObject({ decision: "deny" }); await db.update(toolInvocations).set({ ...invocation, status: "executing" }).where(eq(toolInvocations.id, invocation.id)); } expect(await svc.decide({ ...input, request: { ...input.request, arguments: { to: "another@example.test" } } }, options)) .toMatchObject({ decision: "rate_limited" }); + expect(await svc.admitInvocation({ ...input, request: { ...input.request, arguments: { to: "another@example.test" } } }, invocation.id)) + .toMatchObject({ decision: "deny" }); }); it("rejects unsupported policy semantics at create and update time", async () => { diff --git a/server/src/__tests__/tool-access-service.test.ts b/server/src/__tests__/tool-access-service.test.ts index 6caafb2cb03..e5629ead55f 100644 --- a/server/src/__tests__/tool-access-service.test.ts +++ b/server/src/__tests__/tool-access-service.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import * as fs from "node:fs/promises"; import { createHash, generateKeyPairSync, randomUUID } from "node:crypto"; import { createServer } from "node:http"; @@ -97,8 +98,8 @@ import { HttpError } from "../errors.js"; import type { VercelConnectClient } from "../services/vercel-connect.js"; import { invalidatePaperclipCloudConnectorCapabilities, PaperclipCloudConnectorError, type PaperclipCloudConnector } from "../services/paperclip-cloud-connector.js"; -const externalDatabaseUrl = process.env.PAPERCLIP_TOOL_ACCESS_TEST_DATABASE_URL?.trim(); -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalDatabaseUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TOOL_ACCESS_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; @@ -4119,8 +4120,9 @@ describeEmbeddedPostgres("tool access service", () => { .from(toolActionRequests) .where(eq(toolActionRequests.companyId, company.id)); expect(requests).toHaveLength(2); + expect(globalThis.fetch).toHaveBeenCalledTimes(1); expect(requests.map((row) => row.status).sort()).toEqual([ - "approved", + "executed", "pending", ]); }); diff --git a/server/src/__tests__/tool-gateway-service.test.ts b/server/src/__tests__/tool-gateway-service.test.ts index 3e9829670c6..4a014b22ab0 100644 --- a/server/src/__tests__/tool-gateway-service.test.ts +++ b/server/src/__tests__/tool-gateway-service.test.ts @@ -35,12 +35,15 @@ import { toolGatewaySessions, toolInvocations, toolPolicies, + toolRateLimitCounters, } from "@paperclipai/db"; import type { PluginToolDispatcher } from "../services/plugin-tool-dispatcher.js"; import type { VercelConnectClient } from "../services/vercel-connect.js"; import { initializeRunIdentity, reserveSteeredIdentity, reconcileSteeredIdentity } from "../services/run-identity.js"; import { issueService } from "../services/issues.js"; +import { approvalService } from "../services/approvals.js"; import { commitToolActionReview } from "../services/tool-action-review.js"; +import { bindVerifiedToolApprovalArguments, claimToolTestApproval, releaseToolApproval, startToolInvocation } from "../services/tool-approval-execution.js"; import { materializeNativeInteractionResponses } from "../services/native-runtime/native-interaction-bridge.js"; import { toolActionDeliveryService } from "../services/tool-action-delivery.js"; import { secretService } from "../services/secrets.js"; @@ -49,7 +52,7 @@ import { createToolGatewayService, ToolGatewayHttpError, } from "../services/tool-gateway.js"; -import { canonicalToolArguments, signToolArguments } from "../services/tool-content-guards.js"; +import { canonicalToolArguments, readSignedToolArgumentsPayload, signToolArguments, summarizeToolValue, toolRequestIdentityHash } from "../services/tool-content-guards.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, @@ -57,6 +60,7 @@ import { import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { registerToolReviewRecoveryWork } from "../services/tool-review-recovery-work.js"; import { registerToolReviewExpiryWork } from "../services/tool-review-expiry-work.js"; import { idleWorkSnapshot } from "../services/task-admission.js"; @@ -210,7 +214,7 @@ function runSessionInput(company: IdRef, agent: IdRef, run: IdRef): Parameters { let db!: ReturnType; let tempDb: Awaited> | null = null; - let stopExpiryWorker: (() => Promise) | undefined; + let stopReviewWorker: (() => Promise) | undefined; beforeAll(async () => { if (!externalDatabaseUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-tool-gateway-"); @@ -221,7 +225,7 @@ describeEmbeddedPostgres("tool gateway service", () => { }, 20_000); afterEach(async () => { - await stopExpiryWorker?.(); stopExpiryWorker = undefined; + await stopReviewWorker?.(); stopReviewWorker = undefined; await clearFixtures(); }); @@ -268,7 +272,7 @@ describeEmbeddedPostgres("tool gateway service", () => { await db.insert(toolPolicies).values(reviewNotePolicyRow(fixture.company)); gateway = createTestToolGatewayService(db); draining = false; errors.length = 0; - stopExpiryWorker = stopWorker; + stopReviewWorker = stopWorker; vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); }); async function stopWorker() { @@ -276,6 +280,7 @@ describeEmbeddedPostgres("tool gateway service", () => { vi.useRealTimers(); vi.restoreAllMocks(); await db.execute(sql`drop trigger if exists sleep_review_fail on tool_invocations`); await db.execute(sql`drop function if exists sleep_review_fail()`); + await db.execute(sql`drop sequence if exists sleep_review_failure_attempt`); expect(errors).toEqual([]); } async function start() { @@ -313,6 +318,710 @@ describeEmbeddedPostgres("tool gateway service", () => { expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); } + describe("approved review recovery worker", () => { + async function startRecovery() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerToolReviewRecoveryWork(coordinator, gateway, () => !draining).ready; + } + const approve = (pending: Awaited>, source = db) => commitToolActionReview(source, { + companyId: fixture.company.id, actionRequestId: pending.id, decision: "approved", actor: { userId: "board" }, + }); + async function executed(pending: Awaited>) { + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executed"); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0].status).toBe("succeeded"); + } + + it("is quiet while configured idle and executes a committed approval once", async () => { + await startRecovery(); await quiet(); + const pending = await request(); await approve(pending); await settle(); + await executed(pending); await quiet(); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1); + }); + + it("does not scan another company's approved backlog after review writes", async () => { + const other = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company)); + const foreign = await request(gateway, db, other); + await db.update(toolActionRequests).set({ signedArguments: signToolArguments({ + invocationId: foreign.invocationId, toolName: "mcp-remote-fixture:update_note", + canonicalArguments: canonicalToolArguments({ noteId: "legacy", body: "inert" }), + signingSecret: testToolActionSigningSecret, + }) }).where(eq(toolActionRequests.id, foreign.id)); + await commitToolActionReview(db, { companyId: other.company.id, actionRequestId: foreign.id, + decision: "approved", actor: { userId: "board" } }); + await startRecovery(); await quiet(); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query, params) => { + if (/^select/i.test(query) && query.includes('"tool_action_requests"') && + query.includes('"tool_action_requests"."status" =') && params.includes("approved")) + reads.push({ query, params: [...params] }); + }; + try { + for (let index = 0; index < 3; index++) { + const pending = await request(); await settle(1500); + await approve(pending); await settle(1500); await executed(pending); + } + } finally { db.$client.options.debug = debug; } + expect(reads.length).toBeGreaterThan(0); + for (const read of reads) { + expect(read.query).toContain('"tool_action_requests"."company_id" ='); + expect(read.params).toContain(fixture.company.id); + expect(read.params).not.toContain(other.company.id); + } + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, foreign.id)))[0].status).toBe("approved"); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, foreign.invocationId)))[0].status).toBe("awaiting_approval"); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(3); + await quiet(); + }); + + it.each(["foreground", "recovery"] as const)("preserves the admitted rate slot for an agent action through %s approval", async mode => { + await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "One reviewed note", + policyType: "rate_limit", priority: 1, selectors: { toolName: "mcp-remote-fixture:update_note" }, + config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } }); + const pending = await request(); + const counters = await db.select().from(toolRateLimitCounters); + expect(counters).toHaveLength(1); expect(counters[0].remaining).toBe(0); + if (mode === "foreground") { + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + await startRecovery(); + } else { await approve(pending); await startRecovery(); } + await settle(); await executed(pending); await quiet(); + expect(await db.select().from(toolRateLimitCounters)).toEqual(counters); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1); + }); + + it.each(["new", "increased"] as const)("spends the newly available slot of a %s rate rule during approved recovery", async change => { + const rate = { companyId: fixture.company.id, name: "Changed reviewed-note limit", + policyType: "rate_limit" as const, priority: 1, selectors: { toolName: "mcp-remote-fixture:update_note" }, + config: { limit: 2, windowSeconds: 3600, keyBy: ["agent", "tool"] } }; + const original = change === "increased" ? (await db.insert(toolPolicies).values(rate).returning())[0] : null; + const first = await request(), second = await request(); + if (original) { + await db.update(toolPolicies).set({ config: { ...rate.config, limit: 3 }, updatedAt: new Date() }) + .where(eq(toolPolicies.id, original.id)); + } else { + await db.insert(toolPolicies).values({ ...rate, config: { ...rate.config, limit: 1 } }); + } + await approve(first); await approve(second); + await startRecovery(); await settle(); await quiet(); + const requests = await db.select().from(toolActionRequests); + expect(requests.filter(row => row.status === "executed")).toHaveLength(1); + expect(requests.filter(row => row.status === "failed")).toHaveLength(1); + const invocations = await db.select().from(toolInvocations); + expect(invocations.filter(row => row.errorCode === "rate_limited")).toHaveLength(1); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1); + const counters = await db.select().from(toolRateLimitCounters); + expect(counters).toHaveLength(1); + expect(counters[0]).toMatchObject({ limit: change === "increased" ? 3 : 1, remaining: 0 }); + }); + + it.each([ + ...["agent", "test-tab"].flatMap(origin => ["valid", "new-rate", "invalid-signature", "foreign-hash", "foreground-rate", "foreground-invalid-signature", "foreground-foreign-hash", "foreground-modern-review-hash"] + .map(state => ({ origin, state }))), + ])("recovers historical preview-hash approval ($origin, $state)", async ({ origin, state }) => { + const remoteCall = providerFixture(); remoteCall.release(); + const parameters = { value: "reviewed", apiKey: "synthetic-historical-approval-key" }; + const foreground = state.startsWith("foreground-"); + if (foreground) await db.insert(toolPolicies).values({ companyId: fixture.company.id, + name: "Already admitted historical action", policyType: "rate_limit", priority: 1, + selectors: {}, config: { limit: 1, windowSeconds: 3600, keyBy: ["company", "tool"] } }); + const pending = origin === "agent" ? await agentRequest(remoteCall.provider, parameters) + : (await testRequest(remoteCall.provider, false, parameters)).pending; + const [originalInvocation] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)); + const payload = readSignedToolArgumentsPayload({ signedArguments: pending.signedArguments, + invocationId: pending.invocationId, toolName: originalInvocation.toolName, signingSecret: testToolActionSigningSecret })!; + expect(payload).not.toBeNull(); + const legacyHash = summarizeToolValue(payload.arguments).sha256!; + const completeHash = toolRequestIdentityHash(payload.arguments, testToolActionSigningSecret); + expect(legacyHash).not.toBe(completeHash); + const storedHash = state.endsWith("foreign-hash") ? "e".repeat(64) : legacyHash; + await db.update(toolInvocations).set({ argumentsHash: storedHash }).where(eq(toolInvocations.id, pending.invocationId)); + await db.update(toolActionRequests).set({ canonicalArgumentsHash: state === "foreground-modern-review-hash" ? completeHash : legacyHash, + ...(["invalid-signature", "foreground-invalid-signature"].includes(state) ? { signedArguments: "unreadable-old-signature" } : {}), + }).where(eq(toolActionRequests.id, pending.id)); + if (state === "new-rate") await db.insert(toolPolicies).values({ companyId: fixture.company.id, + name: "New admission for historical approval", policyType: "rate_limit", priority: 1, + selectors: {}, config: { limit: 1, windowSeconds: 3600, keyBy: ["company", "tool"] } }); + const counters = foreground ? await db.select().from(toolRateLimitCounters) : []; + if (foreground) { + expect(counters).toEqual([expect.objectContaining({ remaining: 0 })]); + const approval = gateway.approveActionRequest({ companyId: fixture.company.id, + actionRequestId: pending.id, actor: { userId: "board" } }); + if (state === "foreground-invalid-signature") await expect(approval).rejects.toMatchObject({ reasonCode: "action_request_invalidated" }); + else if (state === "foreground-foreign-hash" && origin === "agent") await expect(approval).rejects.toMatchObject({ reasonCode: "rate_limited" }); + else await approval; + } else await approve(pending); + await startRecovery(); await settle(); + if (state === "valid" || state === "new-rate" || state === "foreground-rate" || state === "foreground-modern-review-hash") { + await executed(pending); + expect((await db.select().from(toolInvocations))[0].argumentsHash).toBe(completeHash); + expect(remoteCall.provider).toHaveBeenCalledOnce(); + expect(JSON.parse(String(remoteCall.provider.mock.calls[0][1].body)).params.arguments).toEqual(parameters); + if (state === "new-rate") expect(await db.select().from(toolRateLimitCounters)) + .toEqual([expect.objectContaining({ limit: 1, remaining: 0 })]); + } else { + expect((await db.select().from(toolActionRequests))[0].status) + .toBe(["invalid-signature", "foreground-invalid-signature"].includes(state) ? "cancelled" + : state === "foreground-foreign-hash" && origin === "agent" ? "pending" : "failed"); + expect((await db.select().from(toolInvocations))[0].argumentsHash).toBe(storedHash); + expect(remoteCall.provider).not.toHaveBeenCalled(); + } + if (foreground) expect(await db.select().from(toolRateLimitCounters)).toEqual(counters); + await quiet(); + }); + + it("does not rebind historical arguments after the execution claim is replaced", async () => { + const remoteCall = providerFixture(); remoteCall.release(); + const pending = await agentRequest(remoteCall.provider, { value: "reviewed", apiKey: "synthetic-stale-claim-key" }); + const [invocation] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)); + const payload = readSignedToolArgumentsPayload({ signedArguments: pending.signedArguments, + invocationId: invocation.id, toolName: invocation.toolName, signingSecret: testToolActionSigningSecret })!; + const preview = summarizeToolValue(payload.arguments).sha256; + const verified = toolRequestIdentityHash(payload.arguments, testToolActionSigningSecret); + await db.update(toolInvocations).set({ argumentsHash: preview }).where(eq(toolInvocations.id, invocation.id)); + await db.update(toolActionRequests).set({ canonicalArgumentsHash: preview }).where(eq(toolActionRequests.id, pending.id)); + await approve(pending); + const claim = (await claimToolTestApproval(db, pending.id, invocation.id))!; + expect(claim).not.toBeNull(); + await db.update(toolActionRequests).set({ updatedAt: new Date(claim.claimedAt.getTime() + 1) }) + .where(eq(toolActionRequests.id, pending.id)); + expect(await bindVerifiedToolApprovalArguments(db, claim, { preview, verified })).toBe(false); + expect((await db.select().from(toolInvocations))[0].argumentsHash).toBe(preview); + expect(remoteCall.provider).not.toHaveBeenCalled(); + }); + + it("reconstructs a saved approved action after restart", async () => { + const pending = await request(); await approve(pending); + await startRecovery(); await settle(); await executed(pending); await quiet(); + }); + + it("retries a failed approval dispatch only while the approved action remains unresolved", async () => { + const pending = await request(); await approve(pending); + const recover = vi.spyOn(gateway, "approveActionRequest").mockRejectedValue(new Error("Recovery dependency unavailable")); + await startRecovery(); + expect(recover).toHaveBeenCalledTimes(1); expect(coordinator!.nextWakeAt()).not.toBeNull(); + await settle(10_000); expect(recover).toHaveBeenCalledTimes(1); + recover.mockRestore(); await settle(); await executed(pending); await quiet(); + }); + + it("preserves a failed company's retry across writes while another company executes", async () => { + const pending = await request(); await approve(pending); + const original = gateway.approveActionRequest.bind(gateway); + const dispatch = vi.spyOn(gateway, "approveActionRequest").mockImplementation(async input => { + if (input.companyId === fixture.company.id) throw new Error("Company dependency unavailable"); + return original(input); + }); + await startRecovery(); + expect(dispatch).toHaveBeenCalledTimes(1); + const retryAt = coordinator!.nextWakeAt(); + expect(retryAt).not.toBeNull(); + const later = await request(); await approve(later); + const other = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company)); + const foreign = await request(gateway, db, other); + await commitToolActionReview(db, { companyId: other.company.id, actionRequestId: foreign.id, + decision: "approved", actor: { userId: "board" } }); + await settle(5000); await executed(foreign); + expect(dispatch.mock.calls.filter(([input]) => input.companyId === fixture.company.id)).toHaveLength(1); + expect(coordinator!.nextWakeAt()).toBe(retryAt); + dispatch.mockRestore(); + await settle(); await executed(pending); await executed(later); await quiet(); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(3); + }); + + it("disarms a failed company's recovery after company deletion commits", async () => { + const pending = await request(); await approve(pending); + const dispatch = vi.spyOn(gateway, "approveActionRequest").mockRejectedValue(new Error("Company dependency unavailable")); + await startRecovery(); expect(coordinator!.nextWakeAt()).not.toBeNull(); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }) + .where(eq(heartbeatRuns.id, fixture.run.id)); + await db.update(agents).set({ status: "terminated", lifecycleState: "terminated" }) + .where(eq(agents.id, fixture.agent.id)); + await deleteCompany(db, fixture.company.id); + await settle(1500); + expect(coordinator!.nextWakeAt()).toBeNull(); + expect(dispatch).toHaveBeenCalledTimes(1); await quiet(); + }); + + it.each([null, "unreadable-old-signature"])("settles an approved invalid signature (%s) without recurring recovery", async signedArguments => { + const pending = await request(); await approve(pending); + await db.update(toolActionRequests).set({ signedArguments }).where(eq(toolActionRequests.id, pending.id)); + const dispatch = vi.spyOn(gateway, "approveActionRequest"); + await startRecovery(); await settle(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled"); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ + status: "failed", errorCode: "action_request_invalidated", + }); + expect(dispatch).not.toHaveBeenCalled(); await quiet(); + }); + + it.each(["rejected", "cancelled"])("settles a %s formal approval without recurring recovery", async status => { + const pending = await request(); await approve(pending); + const [formal] = await db.insert(approvals).values({ companyId: fixture.company.id, + type: "request_board_approval", status, payload: {} }).returning(); + await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id)); + const dispatch = vi.spyOn(gateway, "approveActionRequest"); + await startRecovery(); await settle(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled"); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ + status: "denied", errorCode: "formal_approval_unavailable", + }); + expect(dispatch).not.toHaveBeenCalled(); await quiet(); + }); + + it.each([false, true])("parks a formal approval until its decision commits (rollback %s)", async rollback => { + const pending = await request(); await approve(pending); + const [formal] = await db.insert(approvals).values({ companyId: fixture.company.id, + type: "request_board_approval", status: "pending", payload: {} }).returning(); + await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id)); + await startRecovery(); await quiet(); + let release!: () => void, staged!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { staged = resolve; }); + const aborted = new Error("formal approval rollback fixture"); + const writing = db.transaction(async tx => { + await approvalService(tx as unknown as typeof db).approve(formal.id, "board"); + staged(); await gate; if (rollback) throw aborted; + }).catch(error => { if (error !== aborted) throw error; }); + await Promise.race([ready, writing]); + try { + await vi.advanceTimersByTimeAsync(60_000); + expect((await db.select().from(toolActionRequests))[0].status).toBe("approved"); + expect((await db.select().from(toolInvocations))[0].status).toBe("awaiting_approval"); + } finally { release(); } + await writing; await settle(); + if (rollback) { + expect((await db.select().from(toolActionRequests))[0].status).toBe("approved"); await quiet(); + await approvalService(db).approve(formal.id, "board"); await settle(); + } + await executed(pending); await quiet(); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1); + }); + + it.each(["reject", "cancel"] as const)("wakes a parked revision-requested formal approval on %s", async decision => { + const pending = await request(); await approve(pending); + const [formal] = await db.insert(approvals).values({ companyId: fixture.company.id, + type: "request_board_approval", status: "revision_requested", payload: {} }).returning(); + await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id)); + await startRecovery(); await quiet(); + const service = approvalService(db); + if (decision === "cancel") await service.cancel(formal.id, "Cancelled by user"); + else await service.reject(formal.id, "board"); + await settle(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled"); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "denied", errorCode: "formal_approval_unavailable" }); + await quiet(); + }); + + it("retains approved work when the signing key is temporarily unavailable", async () => { + const pending = await request(); await approve(pending); + vi.stubEnv("PAPERCLIP_TOOL_ACTION_SIGNING_SECRET", ""); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "invalid-master-key"); + gateway = createToolGatewayService(db); + await startRecovery(); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("approved"); + expect((await db.select().from(toolInvocations))[0].status).toBe("awaiting_approval"); + vi.stubEnv("PAPERCLIP_TOOL_ACTION_SIGNING_SECRET", testToolActionSigningSecret); + await settle(); await executed(pending); await quiet(); + }); + + it("does not accept another company's formal approval", async () => { + const pending = await request(); await approve(pending); + const other = await createRunFixture(db); + const [formal] = await db.insert(approvals).values({ companyId: other.company.id, + type: "request_board_approval", status: "approved", payload: {} }).returning(); + await db.update(toolActionRequests).set({ approvalId: formal.id }).where(eq(toolActionRequests.id, pending.id)); + await startRecovery(); await settle(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("cancelled"); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "denied", errorCode: "formal_approval_unavailable" }); + expect((await db.select().from(approvals))[0]).toEqual(formal); + await quiet(); + }); + + it("quarantines a mismatched request without modifying another company's invocation", async () => { + const pending = await request(); await approve(pending); + const [invocation] = await db.select().from(toolInvocations); + const other = await createRunFixture(db); + await db.update(toolActionRequests).set({ companyId: other.company.id }).where(eq(toolActionRequests.id, pending.id)); + await startRecovery(); await settle(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("failed"); + expect((await db.select().from(toolInvocations))[0]).toEqual(invocation); + await quiet(); + }); + + it("repairs completed agent history without executing the action again", async () => { + const pending = await request(); + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + await executed(pending); + const [invocation] = await db.select().from(toolInvocations); + await db.update(toolActionRequests).set({ status: "approved", signedArguments: "unreadable-old-signature" }) + .where(eq(toolActionRequests.id, pending.id)); + await startRecovery(); await settle(); await executed(pending); await quiet(); + expect((await db.select().from(toolInvocations))[0]).toEqual(invocation); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1); + }); + + it("keeps deliberately inert legacy approvals quiet", async () => { + const pending = await request(); + await db.update(toolActionRequests).set({ signedArguments: signToolArguments({ + invocationId: pending.invocationId, toolName: "mcp-remote-fixture:update_note", + canonicalArguments: canonicalToolArguments({ noteId: "legacy", body: "inert" }), + signingSecret: testToolActionSigningSecret, + }) }).where(eq(toolActionRequests.id, pending.id)); + await approve(pending); await startRecovery(); await quiet(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("approved"); + expect((await db.select().from(toolInvocations))[0].status).toBe("awaiting_approval"); + }); + + it.each([false, true])("waits for the outer approval transaction (rollback %s)", async rollback => { + await startRecovery(); const pending = await request(); + let release!: () => void, staged!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const ready = new Promise(resolve => { staged = resolve; }); + const aborted = new Error("approval rollback fixture"); + const writing = db.transaction(async tx => { + await approve(pending, tx as unknown as typeof db); staged(); await gate; + if (rollback) throw aborted; + }).catch(error => { if (error !== aborted) throw error; }); + await Promise.race([ready, writing]); + try { + await vi.advanceTimersByTimeAsync(60_000); + expect((await db.select().from(toolActionRequests))[0].status).toBe("pending"); + } finally { release(); } + await writing; await settle(); + if (rollback) expect((await db.select().from(toolActionRequests))[0].status).toBe("pending"); + else await executed(pending); + await quiet(); + }); + + it("retains the first approval during drain without passive queries", async () => { + await startRecovery(); await quiet(); draining = true; + const pending = await request(); await approve(pending); await settle(); + const reads = vi.spyOn(db, "select"); + await settle(60 * 60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; await settle(); await executed(pending); await quiet(); + }); + + async function testRequest(provider: NonNullable, rateLimited = false, parameters: Record = { value: "reviewed" }) { + await db.insert(companyMemberships).values({ companyId: fixture.company.id, principalType: "user", + principalId: "board", status: "active", membershipRole: "owner" }); + const remote = await createRemoteMcpToolFixture(db, fixture.company.id); + await db.update(toolCatalogEntries).set({ isReadOnly: false, riskLevel: "write" }) + .where(eq(toolCatalogEntries.id, remote.catalogEntry.id)); + await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "Review connected writes", + policyType: "require_approval", priority: 10, selectors: { connectionId: remote.connection.id } }); + if (rateLimited) await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "One reviewed test call", + policyType: "rate_limit", priority: 1, selectors: { connectionId: remote.connection.id }, + config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } }); + gateway = createTestToolGatewayService(db, { remoteHttpRequest: provider }); + const result = await gateway.executeTestCall({ companyId: fixture.company.id, connectionId: remote.connection.id, + agentId: fixture.agent.id, userId: "board", toolName: "needs_input", parameters }); + expect(result.decision).toBe("ask_first"); + const [pending] = await db.select().from(toolActionRequests); + return { pending, remote }; + } + function providerFixture() { + let release!: () => void, entered!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const started = new Promise(resolve => { entered = resolve; }); + const provider = vi.fn(async (_url, init) => { + const payload = JSON.parse(String(init.body)); + expect(payload.method).toBe("tools/call"); + entered(); await gate; + return Response.json({ jsonrpc: "2.0", id: payload.id, + result: { content: [{ type: "text", text: "Saved once" }] } }); + }); + return { provider, release, started }; + } + + it.each(["foreground", "recovery"] as const)("retries a Test-tab %s approval after a real pre-dispatch database timeout", async mode => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider); await approve(pending); + const preparationDb = createDb(externalDatabaseUrl ?? tempDb!.connectionString, { + maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0, + }); + const preparingGateway = createTestToolGatewayService(preparationDb, { remoteHttpRequest: remoteCall.provider }); + let unlock!: () => void, locked!: () => void; + const gate = new Promise(resolve => { unlock = resolve; }); + const acquired = new Promise(resolve => { locked = resolve; }); + const lock = db.transaction(async tx => { + await tx.execute(sql`lock table tool_policies in access exclusive mode`); + locked(); await gate; + }); + await acquired; + try { + await preparationDb.execute(sql`set lock_timeout = '2s'`); + const attempting = mode === "foreground" + ? preparingGateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }) + : preparingGateway.recoverApprovedActionReviews(fixture.company.id); + // Observe the real committed claim while its subsequent policy read + // is blocked. Provider execution has not started. + void attempting.catch(() => undefined); + await vi.waitFor(async () => { + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing"); + }); + if (mode === "foreground") await expect(attempting).rejects.toMatchObject({ cause: { code: "55P03" } }); + else expect(await attempting).toEqual({ scanned: 1, retry: true }); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0]) + .toMatchObject({ status: "approved", signedArguments: pending.signedArguments }); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0]) + .toMatchObject({ status: "awaiting_approval", startedAt: null }); + expect(remoteCall.provider).not.toHaveBeenCalled(); + } finally { unlock(); await lock; await preparationDb.$client.end({ timeout: 1 }); } + await startRecovery(); await settle(); await executed(pending); await quiet(); + expect(remoteCall.provider).toHaveBeenCalledOnce(); + }, 15_000); + + async function agentRequest(provider: NonNullable, parameters: Record = { value: "reviewed" }) { + const remote = await createRemoteMcpToolFixture(db, fixture.company.id); + await db.update(toolCatalogEntries).set({ isReadOnly: false, riskLevel: "write" }) + .where(eq(toolCatalogEntries.id, remote.catalogEntry.id)); + await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "Review agent writes", + policyType: "require_approval", selectors: { connectionId: remote.connection.id } }); + gateway = createTestToolGatewayService(db, { remoteHttpRequest: provider }); + const session = await gateway.createSession(runSessionInput(fixture.company, fixture.agent, fixture.run)); + const tool = (await gateway.listToolsForSession(session.token)).find(t => t.connectionId === remote.connection.id)!; + await expect(gateway.executeTool({ sessionToken: session.token, tool: tool.name, parameters })) + .rejects.toMatchObject({ reasonCode: "approval_required" }); + const [pending] = await db.select().from(toolActionRequests); + return pending; + } + + it.each(["foreground", "recovery"] as const)("retries an agent-origin %s approval after a real pre-dispatch database timeout", async mode => { + const remoteCall = providerFixture(); remoteCall.release(); + const pending = await agentRequest(remoteCall.provider); await approve(pending); + const preparationDb = createDb(externalDatabaseUrl ?? tempDb!.connectionString, { + maxConnections: 1, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0, + }); + const preparingGateway = createTestToolGatewayService(preparationDb, { remoteHttpRequest: remoteCall.provider }); + let unlock!: () => void, locked!: () => void; + const gate = new Promise(resolve => { unlock = resolve; }); + const acquired = new Promise(resolve => { locked = resolve; }); + const lock = db.transaction(async tx => { + await tx.execute(sql`lock table tool_policies in access exclusive mode`); + locked(); await gate; + }); + await acquired; + try { + await preparationDb.execute(sql`set lock_timeout = '2s'`); + const attempting = mode === "foreground" + ? preparingGateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }) + : preparingGateway.recoverApprovedActionReviews(fixture.company.id); + // Observe the real committed claim while its subsequent policy read + // is blocked. Provider execution has not started. + void attempting.catch(() => undefined); + await vi.waitFor(async () => { + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing"); + }); + if (mode === "foreground") expect(await attempting).toMatchObject({ status: "approved" }); + else expect(await attempting).toEqual({ scanned: 1, retry: true }); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0]) + .toMatchObject({ status: "approved", signedArguments: pending.signedArguments }); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0]) + .toMatchObject({ status: "awaiting_approval", startedAt: null }); + expect(remoteCall.provider).not.toHaveBeenCalled(); + } finally { unlock(); await lock; await preparationDb.$client.end({ timeout: 1 }); } + await startRecovery(); await settle(); await executed(pending); await quiet(); + expect(remoteCall.provider).toHaveBeenCalledOnce(); + }, 15_000); + + it("refuses agent dispatch after preparation loses its claim", async () => { + const remoteCall = providerFixture(); remoteCall.release(); + const pending = await agentRequest(remoteCall.provider); await approve(pending); + let unlock!: () => void, locked!: () => void; + const gate = new Promise(resolve => { unlock = resolve; }); + const acquired = new Promise(resolve => { locked = resolve; }); + const lock = db.transaction(async tx => { + // Pause tool discovery before admission locks the invocation. Revocation + // cannot legitimately take that lock from an in-flight admission. + await tx.execute(sql`lock table tool_catalog_entries in access exclusive mode`); + locked(); await gate; + }); + await acquired; + const attempting = gateway.approveActionRequest({ companyId: fixture.company.id, + actionRequestId: pending.id, actor: { userId: "board" } }); + void attempting.catch(() => undefined); + let successor!: NonNullable>>; + try { + await vi.waitFor(async () => { + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing"); + }); + const [claimed] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)); + expect(await releaseToolApproval(db, { actionRequestId: pending.id, + invocationId: pending.invocationId, claimedAt: claimed.updatedAt })).toBe(true); + await vi.advanceTimersByTimeAsync(10); + successor = (await claimToolTestApproval(db, pending.id, pending.invocationId))!; + expect(successor).not.toBeNull(); + expect(successor.claimedAt.getTime()).toBeGreaterThan(claimed.updatedAt.getTime()); + } finally { unlock(); await lock; } + expect(await attempting).toMatchObject({ status: "executing" }); + expect(remoteCall.provider).not.toHaveBeenCalled(); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0]) + .toMatchObject({ status: "awaiting_approval", startedAt: null }); + expect(await releaseToolApproval(db, successor)).toBe(true); + await startRecovery(); await settle(); await executed(pending); await quiet(); + expect(remoteCall.provider).toHaveBeenCalledOnce(); + }); + + it("does not retry an agent provider error as a preparation failure", async () => { + const provider = vi.fn(async () => { throw new Error("Provider outcome is unknown"); }); + const pending = await agentRequest(provider); await approve(pending); + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("failed"); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0].startedAt).not.toBeNull(); + await startRecovery(); await quiet(); + expect(provider).toHaveBeenCalledOnce(); + }); + + it.each(["started", "reclaimed", "mismatched", "prior_dispatch"] as const)("refuses to release a Test-tab preparation claim with %s ownership evidence", async state => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider); await approve(pending); + const claim = await claimToolTestApproval(db, pending.id, pending.invocationId); + expect(claim).not.toBeNull(); + if (state === "started") expect(await startToolInvocation(db, pending.invocationId, claim!)).toBe(true); + if (state === "reclaimed") await db.update(toolActionRequests) + .set({ updatedAt: new Date(claim!.claimedAt.getTime() + 1000) }).where(eq(toolActionRequests.id, pending.id)); + if (state === "prior_dispatch") await db.update(toolInvocations) + .set({ startedAt: new Date() }).where(eq(toolInvocations.id, pending.invocationId)); + const beforeRequest = (await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0]; + const beforeInvocation = (await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0]; + expect(await releaseToolApproval(db, state === "mismatched" ? { ...claim!, invocationId: randomUUID() } : claim!)).toBe(false); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0]).toEqual(beforeRequest); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0]).toEqual(beforeInvocation); + expect(remoteCall.provider).not.toHaveBeenCalled(); + }); + + it("rolls a Test-tab preparation release back with its transaction and invalidates the released claim", async () => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider); await approve(pending); + const claim = await claimToolTestApproval(db, pending.id, pending.invocationId); + expect(claim).not.toBeNull(); + await expect(db.transaction(async tx => { + expect(await releaseToolApproval(tx as unknown as typeof db, claim!)).toBe(true); + throw new Error("release rollback"); + })).rejects.toThrow("release rollback"); + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("executing"); + expect(await releaseToolApproval(db, claim!)).toBe(true); + expect(await startToolInvocation(db, pending.invocationId, claim!)).toBe(false); + await startRecovery(); await settle(); await executed(pending); await quiet(); + expect(remoteCall.provider).toHaveBeenCalledOnce(); + }); + + it.each(["foreground", "recovery"] as const)("claims a Test-tab action once when %s wins", async winner => { + const remoteCall = providerFixture(); + const { pending } = await testRequest(remoteCall.provider); + let running: Promise; + if (winner === "foreground") { + await startRecovery(); + running = gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + } else { + await approve(pending); + running = startRecovery(); + } + await Promise.race([remoteCall.started, running]); + try { + await vi.advanceTimersByTimeAsync(60_000); + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + expect(remoteCall.provider).toHaveBeenCalledTimes(1); + } finally { remoteCall.release(); } + await running; await settle(); await executed(pending); await quiet(); + expect(remoteCall.provider).toHaveBeenCalledTimes(1); + }); + + it.each(["foreground", "recovery"] as const)("preserves the admitted rate slot for a Test-tab action through %s approval", async mode => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider, true); + const counters = await db.select().from(toolRateLimitCounters); + expect(counters).toHaveLength(1); expect(counters[0].remaining).toBe(0); + if (mode === "foreground") { + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + await startRecovery(); + } else { await approve(pending); await startRecovery(); } + await settle(); await executed(pending); await quiet(); + expect(await db.select().from(toolRateLimitCounters)).toEqual(counters); + expect(remoteCall.provider).toHaveBeenCalledOnce(); + }); + + it("repairs completed Test-tab history without sending the request again", async () => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider); + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + await executed(pending); + await db.update(toolActionRequests).set({ status: "approved", signedArguments: "unreadable-old-signature" }) + .where(eq(toolActionRequests.id, pending.id)); + remoteCall.provider.mockClear(); + await startRecovery(); await settle(); await executed(pending); await quiet(); + expect(remoteCall.provider).not.toHaveBeenCalled(); + }); + + it.each(["pending", "authorized", "cancelled"])("settles inconsistent Test-tab %s history without replay", async status => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider); await approve(pending); + await db.update(toolInvocations).set({ status }).where(eq(toolInvocations.id, pending.invocationId)); + await startRecovery(); await settle(); + expect((await db.select().from(toolActionRequests))[0].status).toBe(status === "cancelled" ? "cancelled" : "failed"); + expect((await db.select().from(toolInvocations))[0]).toMatchObject(status === "cancelled" + ? { status: "cancelled" } + : { status: "failed", errorCode: "tool_recovery_state_invalid" }); + await quiet(); expect(remoteCall.provider).not.toHaveBeenCalled(); + }); + + it("does not replay an interrupted legacy Test-tab invocation", async () => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider); + await approve(pending); + await db.update(toolInvocations).set({ status: "executing", startedAt: new Date(Date.now() - 11 * 60_000) }) + .where(eq(toolInvocations.id, pending.invocationId)); + await startRecovery(); await settle(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("executing"); + await gateway.sweepActionReviews({ includeApproved: false, includeExpiry: false }); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "failed", errorCode: "tool_execution_outcome_unknown" }); + await quiet(); expect(remoteCall.provider).not.toHaveBeenCalled(); + }); + + it.each(["always", "once"])("does not replay a Test-tab call when persisting its external result fails %s", async mode => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending } = await testRequest(remoteCall.provider); + await approve(pending); + await db.execute(sql`create sequence sleep_review_failure_attempt`); + await db.execute(sql.raw(`create function sleep_review_fail() returns trigger language plpgsql as $$ + begin + if ${mode === "always" ? "true" : "nextval('sleep_review_failure_attempt') = 1"} then + raise exception 'approval result fixture failure'; + end if; + return new; + end $$`)); + await db.execute(sql`create trigger sleep_review_fail before update on tool_action_requests + for each row when (new.status in ('executed', 'failed')) execute function sleep_review_fail()`); + try { + await startRecovery(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("executing"); + expect((await db.select().from(toolInvocations))[0].status).toBe("executing"); + expect(remoteCall.provider).toHaveBeenCalledTimes(1); + } finally { + await db.execute(sql`drop trigger sleep_review_fail on tool_action_requests`); + } + await settle(); + await vi.advanceTimersByTimeAsync(11 * 60_000); + await gateway.sweepActionReviews({ includeApproved: false, includeExpiry: false }); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "failed", errorCode: "tool_execution_outcome_unknown" }); + await quiet(); expect(remoteCall.provider).toHaveBeenCalledTimes(1); + }); + + it("revalidates the reviewed Test-tab target before recovery dispatch", async () => { + const remoteCall = providerFixture(); remoteCall.release(); + const { pending, remote } = await testRequest(remoteCall.provider); + await approve(pending); + await db.update(toolCatalogEntries).set({ versionHash: "changed-after-review" }).where(eq(toolCatalogEntries.id, remote.catalogEntry.id)); + await startRecovery(); await settle(); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status: "failed", errorCode: "approved_tool_target_changed" }); + await quiet(); expect(remoteCall.provider).not.toHaveBeenCalled(); + }); + }); + async function requestWithExpiry(expiresAt: Date, context = fixture) { return db.transaction(async tx => { const source = tx as unknown as typeof db; @@ -876,13 +1585,21 @@ describeEmbeddedPostgres("tool gateway service", () => { await expect(gateway.executeTool({ sessionToken: session.token, tool: "mcp-remote-fixture:update_note", parameters: { noteId: "expire" } })).rejects.toMatchObject({ reasonCode: "approval_required" }); const [request] = await db.select().from(toolActionRequests); await db.update(toolActionRequests).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(toolActionRequests.id, request.id)); - await db.insert(toolActionRequests).values(Array.from({ length: 100 }, (_, i) => ({ + const [originalInvocation] = await db.select().from(toolInvocations).where(eq(toolInvocations.id, request.invocationId)); + const blockedInvocations = Array.from({ length: 101 }, (_, i) => ({ ...originalInvocation, + id: `10000000-0000-4000-8000-${String(i).padStart(12, "0")}`, approvalState: "approved", idempotencyKey: null, + })); + await db.insert(toolInvocations).values(blockedInvocations); + await db.insert(toolActionRequests).values(blockedInvocations.map((invocation, i) => ({ ...request, id: `00000000-0000-4000-8000-${String(i).padStart(12, "0")}`, - interactionId: null, status: "approved" as const, + invocationId: invocation.id, interactionId: null, status: "approved" as const, + signedArguments: signToolArguments({ invocationId: invocation.id, toolName: invocation.toolName, + canonicalArguments: canonicalToolArguments({ noteId: "blocked" }), executionOnApprove: true, + signingSecret: testToolActionSigningSecret }), }))); const recover = vi.spyOn(gateway, "approveActionRequest").mockRejectedValue(new Error("Unavailable approval dependency")); - expect(await gateway.sweepActionReviews()).toEqual({ scanned: 101 }); - expect(recover).toHaveBeenCalledTimes(100); + expect(await gateway.sweepActionReviews()).toEqual({ scanned: 102 }); + expect(recover).toHaveBeenCalledTimes(101); const [settled] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, request.interactionId!)); expect(settled.status).toBe("expired"); }); diff --git a/server/src/index.ts b/server/src/index.ts index ea24010565f..e9fc24fcf0b 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerToolReviewRecoveryWork } from "./services/tool-review-recovery-work.js"; import { registerToolReviewExpiryWork } from "./services/tool-review-expiry-work.js"; import { registerGatewayCredentialExpiryWork } from "./services/gateway-credential-expiry-work.js"; import { registerSecretProposalExpiryWork } from "./services/secret-proposal-expiry-work.js"; @@ -1472,7 +1473,13 @@ async function startServerWithDatabaseTeardown( } else { await app.locals.toolGateway.sweepExpiredActionReviews(); } - return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false }); + if (config.heartbeatSchedulerEnabled) { + await registerToolReviewRecoveryWork(deliveryWork, app.locals.toolGateway, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + } else { + await app.locals.toolGateway.recoverApprovedActionReviews(); + } + return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false }); }).catch((err: unknown) => logger.error({ err }, "startup tool review recovery failed")); await timeStartupRecovery("gateway_sessions", () => config.heartbeatSchedulerEnabled ? registerGatewayCredentialExpiryWork(deliveryWork, app.locals.toolGateway, @@ -1903,7 +1910,7 @@ async function startServerWithDatabaseTeardown( logger.error({ err }, "periodic tool connection health sweep failed"); })); - trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); + trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { diff --git a/server/src/services/approvals.ts b/server/src/services/approvals.ts index 290a59fdf4b..4f4149f773d 100644 --- a/server/src/services/approvals.ts +++ b/server/src/services/approvals.ts @@ -1,3 +1,4 @@ +import { writeToolReviewChange } from "./tool-review-work-notifications.js"; import { approvalResolutionRecords, resolvableApprovalStatuses as resolvableStatuses } from "../lib/approval-records.js"; import { notifyHireApproved } from "./hire-hook.js"; @@ -12,7 +13,15 @@ import { instanceSettingsService } from "./instance-settings.js"; export function approvalRecords(db: Db) { const instanceSettings = instanceSettingsService(db); - const { getExistingApproval, resolveApproval } = approvalResolutionRecords(db); + const { getExistingApproval, resolveApproval: resolveApprovalRecord } = approvalResolutionRecords(db); + + // Tool reviews waiting for a formal decision stay parked until this producer + // commits. The shared record owner retains approval validation and CAS rules. + async function resolveApproval(id: string, status: "approved" | "rejected", userId: string, + note: string | null | undefined, database: Db = db) { + const existing = await getExistingApproval(id, database); + return writeToolReviewChange(database, existing.companyId, tx => resolveApprovalRecord(id, status, userId, note, tx)); + } function redactApprovalComment(comment: T, censorUsernameInLogs: boolean): T { return { @@ -64,7 +73,9 @@ export function approvalRecords(db: Db) { // cleanup. Idempotent: a no-op on already-resolved approvals. cancel: async (id: string, reason?: string | null) => { const now = new Date(); - const updated = await db + const [existing] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, id)); + if (!existing) return null; + const updated = await writeToolReviewChange(db, existing.companyId, tx => tx .update(approvals) .set({ status: "cancelled", @@ -74,7 +85,7 @@ export function approvalRecords(db: Db) { }) .where(and(eq(approvals.id, id), inArray(approvals.status, resolvableStatuses))) .returning() - .then((rows) => rows[0] ?? null); + .then((rows) => rows[0] ?? null)); return updated; }, diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 210c830f91a..923ae4b7979 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + toolReviewRecovery: "tool-review-recovery", toolReviewExpiry: "tool-review-expiry", gatewayCredentialExpiry: "gateway-credential-expiry", secretProposalExpiry: "secret-proposal-expiry", diff --git a/server/src/services/tool-access-policy.ts b/server/src/services/tool-access-policy.ts index 16ac7f6fdce..cfd2f4478a6 100644 --- a/server/src/services/tool-access-policy.ts +++ b/server/src/services/tool-access-policy.ts @@ -1,7 +1,8 @@ import type { ConnectionManagementContext } from "./connection-management.js"; import { createHash } from "node:crypto"; import { boundRunIssueIdSql } from "./run-task-binding.js"; -import { and, asc, desc, eq, gt, inArray, isNull, ne, sql } from "drizzle-orm"; +import type { ToolApprovalClaim } from "./tool-approval-execution.js"; +import { and, asc, desc, eq, inArray, isNull, ne, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, @@ -53,6 +54,12 @@ import { import { recordToolRuntimeAuditWriteFailure } from "./tool-runtime-metrics.js"; import { canonicalToolArguments, readSignedToolArgumentsPayload, toolRequestIdentityHash } from "./tool-content-guards.js"; +class RejectedInvocationAdmission extends Error { + constructor(readonly decision: ToolAccessDecision) { + super(decision.explanation); + } +} + type ToolAccessContext = { companyId: string; actorType: "agent" | "user" | "system" | "plugin"; @@ -717,7 +724,7 @@ function assertSupportedGenericPolicyShape( } } -async function getGenericPolicyRow(db: Db, companyId: string, policyId: string) { +async function getGenericPolicyRow(db: Pick, companyId: string, policyId: string) { const [policy] = await db .select() .from(toolPolicies) @@ -785,7 +792,7 @@ function scopeAllowsTool(scope: Record | null, ctx: ToolAccessC return selectorMatches(selectors, ctx); } -export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecret?: string } = {}) { +export function toolAccessPolicyService(db: Pick, options: { toolActionSigningSecret?: string } = {}) { // Policy administration includes authorized self-targeted changes. It cannot // resolve governed action requests or modify principal permission grants. async function listPolicies(companyId: string) { @@ -1301,11 +1308,12 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr ], set: { limit: rule.limit, - remaining: sql`greatest(0, least(${toolRateLimitCounters.remaining}, ${rule.limit}) - 1)`, + // Preserve the number already spent when an operator changes the limit. + remaining: sql`greatest(0, ${rule.limit} - (${toolRateLimitCounters.limit} - ${toolRateLimitCounters.remaining}) - 1)`, resetAt, updatedAt: now, }, - setWhere: gt(toolRateLimitCounters.remaining, 0), + setWhere: sql`${toolRateLimitCounters.limit} - ${toolRateLimitCounters.remaining} < ${rule.limit}`, }) .returning({ remaining: toolRateLimitCounters.remaining }); if (!counter) { @@ -1376,7 +1384,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr }); } - async function admittedRateLimitAdmissions(ctx: ToolAccessContext, invocationId: string): Promise { + async function admittedRateLimitAdmissions(ctx: ToolAccessContext, invocationId: string, previewHash?: string): Promise { const [invocation] = await db.select({ actorType: toolInvocations.actorType, actorId: toolInvocations.actorId, agentId: toolInvocations.agentId, runId: toolInvocations.runId, issueId: toolInvocations.issueId, gatewayId: toolInvocations.gatewayId, @@ -1387,6 +1395,9 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr status: toolInvocations.status, policyDecision: toolInvocations.policyDecision, approvalState: toolInvocations.approvalState, policyExplanation: toolInvocations.policyExplanation, }).from(toolInvocations).where(and(eq(toolInvocations.companyId, ctx.companyId), eq(toolInvocations.id, invocationId))); + // A parked request already spent its slot before asking for review. Reusing + // that admission only skips the same rate rule; current policy and the + // gateway's approval/claim owners still decide whether it may execute. const delegated = invocation?.policyExplanation?.delegatedConnectionTest as { agentId?: string } | undefined; const gatewayIssuer = invocation?.gatewayId ? invocation.policyExplanation?.delegatedGatewayIssuer : undefined; const isDelegatedCaller = invocation?.actorType === "agent" && invocation.actorId === ctx.agentId @@ -1401,13 +1412,81 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr && invocation.catalogEntryId === ctx.catalogEntryId && invocation.toolName === ctx.toolName && invocation.providerType === ctx.providerType && invocation.applicationKey === ctx.applicationKey && invocation.upstreamToolName === ctx.upstreamToolName - && invocation.argumentsHash === ctx.argumentsHash; - const stored = authorized ? invocation.policyExplanation?.[isDelegatedCaller ? "delegatedCallerConsumedRateAdmissions" : "consumedRateAdmissions"] : null; + && (invocation.argumentsHash === ctx.argumentsHash || invocation.argumentsHash === previewHash); + if (!authorized) return null; + const stored = invocation.policyExplanation?.[isDelegatedCaller ? "delegatedCallerConsumedRateAdmissions" : "consumedRateAdmissions"]; return Array.isArray(stored) ? stored.filter((item): item is ConsumedRateAdmission => isRecord(item) && typeof item.policyId === "string" && CANONICAL_UUID_RE.test(item.policyId) && typeof item.versionHash === "string" && /^[a-f0-9]{64}$/.test(item.versionHash)) : []; } + /** Revalidate claimed work and persist new rate charges before provider dispatch. */ + async function admitInvocation( + input: ToolAccessDecisionInput, + invocationId: string, + claim?: ToolApprovalClaim, + evaluate?: ( + policy: Pick, "decide" | "writeAudit">, + input: ToolAccessDecisionInput, + transaction: Db, + ) => Promise, + ): Promise { + try { + return await db.transaction(async tx => { + const [invocation] = await tx.select().from(toolInvocations) + .where(and(eq(toolInvocations.companyId, input.companyId), eq(toolInvocations.id, invocationId))) + .for("update"); + // Historical reviewed requests can still have a pending invocation. + // Accept that state only with the exact executing claim owned by the + // gateway, locked in the same invocation-then-request order as dispatch. + let ownsClaim = false; + if (claim && claim.invocationId === invocationId) { + const [request] = await tx.select().from(toolActionRequests).where(and( + eq(toolActionRequests.id, claim.actionRequestId), + eq(toolActionRequests.invocationId, invocationId), + eq(toolActionRequests.companyId, input.companyId), + )).for("update"); + ownsClaim = request?.status === "executing" && request.updatedAt.getTime() === claim.claimedAt.getTime(); + } + if (!invocation || (claim && !ownsClaim) + || !(invocation.status === "executing" && invocation.policyDecision === "allow" + || ["awaiting_approval", "executing"].includes(invocation.status) + && invocation.policyDecision === "require_approval" && invocation.approvalState === "approved" + || invocation.status === "awaiting_approval" && invocation.policyDecision === "require_approval" + && invocation.approvalState === "pending" && ownsClaim)) { + return decision("deny", "deny_policy_block", "Tool invocation admission is unavailable.", [], []); + } + const admissionPolicy = toolAccessPolicyService(tx, options); + const admissionInput = { ...input, consumeRateLimit: true }; + // Composed caller/target policy must share this claim transaction so + // either all new charges and receipts commit or all roll back. + const result = evaluate + ? await evaluate(admissionPolicy, admissionInput, tx as unknown as Db) + : await admissionPolicy.decide(admissionInput, { admittedInvocationId: invocationId }); + if (!result.allowed && result.decision !== "require_approval") { + // A later policy can reject after an earlier rate rule spent a slot. + // Roll back both the new charges and their receipt on any rejection. + throw new RejectedInvocationAdmission(result); + } + const admissions = result.policyExplanation?.consumedRateAdmissions; + const delegatedAdmissions = result.policyExplanation?.delegatedCallerConsumedRateAdmissions; + if (Array.isArray(admissions) || Array.isArray(delegatedAdmissions)) { + await tx.update(toolInvocations).set({ + policyExplanation: { + ...invocation.policyExplanation, + ...(Array.isArray(admissions) ? { consumedRateAdmissions: admissions } : {}), + ...(Array.isArray(delegatedAdmissions) ? { delegatedCallerConsumedRateAdmissions: delegatedAdmissions } : {}), + }, + }).where(and(eq(toolInvocations.companyId, input.companyId), eq(toolInvocations.id, invocationId))); + } + return result; + }); + } catch (error) { + if (error instanceof RejectedInvocationAdmission) return error.decision; + throw error; + } + } + /** * Decides one tool access request. Pass the same `cache` to every call of a * batch that shares one actor and run context (for example, one tools/list @@ -1415,7 +1494,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr */ async function decide( input: ToolAccessDecisionInput, - options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; sharedRateDecision?: ToolAccessDecision } = {}, + options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; admittedPreviewHash?: string; sharedRateDecision?: ToolAccessDecision } = {}, ): Promise { const consumedRateAdmissions: ConsumedRateAdmission[] = []; const result = await decideInternal(input, options, consumedRateAdmissions); @@ -1426,7 +1505,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr async function decideInternal( input: ToolAccessDecisionInput, - options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; sharedRateDecision?: ToolAccessDecision }, + options: { cache?: ToolAccessDecisionCache; admittedInvocationId?: string; admittedPreviewHash?: string; sharedRateDecision?: ToolAccessDecision }, consumedRateAdmissions: ConsumedRateAdmission[], ): Promise { // A decision that consumes a rate limit can also record a trust-rule hit @@ -1438,9 +1517,14 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr // A final handoff rechecks current permissions for work already admitted by // the gateway. Its own last consumed slot cannot reject that same work. // Only durable consumption of an unchanged rule retains that slot; new or - // edited rules still apply. This option never approves or consumes a call. - const admittedRates = input.consumeRateLimit === false && options.admittedInvocationId - ? await admittedRateLimitAdmissions(ctx, options.admittedInvocationId) : []; + // edited rules still apply. admitInvocation owns atomic consuming rechecks; + // ordinary preflight and browser handoff callers remain read-only. + const admittedRates = options.admittedInvocationId + ? await admittedRateLimitAdmissions(ctx, options.admittedInvocationId, input.consumeRateLimit === true ? undefined : options.admittedPreviewHash) : []; + if (options.admittedInvocationId && input.consumeRateLimit === true && admittedRates === null) { + return decision("deny", "deny_policy_block", "Tool invocation no longer matches this admission.", [], []); + } + if (input.consumeRateLimit === true) consumedRateAdmissions.push(...(admittedRates ?? [])); const profileState = await effectiveProfiles(ctx, cache); const effectiveProfileIds = profileState.profiles.map((profile) => profile.id); const permittedByProfile = profileState.profiles.some((profile) => { @@ -1513,7 +1597,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr ); } const versionHash = ratePolicyVersionHash(policy); - if (admittedRates.some(admission => admission.policyId === policy.id && admission.versionHash === versionHash)) continue; + if (admittedRates?.some(admission => admission.policyId === policy.id && admission.versionHash === versionHash)) continue; // Paired caller/target checks for one delegated test share a slot only // when the exact policy version and counter bucket are identical. const shared = options.sharedRateDecision?.policyExplanation?.consumedRateAdmissions; @@ -1548,7 +1632,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr { redactionPlan: redaction.redactionPlan, policyExplanation }, ); } - if (input.consumeRateLimit === true) { + if (input.consumeRateLimit === true && !options.admittedInvocationId) { await recordTrustRuleHit(policy, ctx, redaction); } return decision("allow", "allow_trust_rule", policy.description ?? "Tool access allowed by trust rule.", effectiveProfileIds, [policy.id], { redactionPlan: redaction.redactionPlan, policyExplanation }); @@ -2178,6 +2262,7 @@ export function toolAccessPolicyService(db: Db, options: { toolActionSigningSecr return { decide, + admitInvocation, writeAudit, recordInvocation, summarizeAndRedact, diff --git a/server/src/services/tool-approval-execution.ts b/server/src/services/tool-approval-execution.ts new file mode 100644 index 00000000000..f5f4fb78abf --- /dev/null +++ b/server/src/services/tool-approval-execution.ts @@ -0,0 +1,119 @@ +import { and, eq } from "drizzle-orm"; +import { toolActionRequests, toolInvocations, type Db } from "@paperclipai/db"; +import type { ToolRedactedValueSummary } from "@paperclipai/shared"; +import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; + +export type ToolApprovalClaim = { actionRequestId: string; invocationId: string; claimedAt: Date }; +export type ToolTestOutcome = { + status: "succeeded" | "failed" | "timed_out" | "rate_limited"; + result?: ToolRedactedValueSummary; + errorCode?: string; + errorMessage?: string; +}; + +export async function claimToolTestApproval(db: Db, actionRequestId: string, invocationId: string): Promise { + return db.transaction(async tx => { + const [invocation] = await tx.select().from(toolInvocations) + .where(eq(toolInvocations.id, invocationId)).for("update"); + const [request] = await tx.select().from(toolActionRequests) + .where(and(eq(toolActionRequests.id, actionRequestId), eq(toolActionRequests.invocationId, invocationId))).for("update"); + if (!invocation || request?.status !== "approved") return null; + await notifyToolReviewChange(tx, request.companyId); + const now = new Date(); + if (invocation.status !== "awaiting_approval") { + if (["succeeded", "failed", "denied", "timed_out", "rate_limited"].includes(invocation.status)) { + await tx.update(toolActionRequests).set({ status: invocation.status === "succeeded" ? "executed" : "failed", + resolvedAt: invocation.completedAt ?? now, updatedAt: now }).where(eq(toolActionRequests.id, actionRequestId)); + } else if (invocation.status === "executing") { + await tx.update(toolActionRequests).set({ status: "executing", updatedAt: invocation.startedAt ?? invocation.updatedAt }) + .where(eq(toolActionRequests.id, actionRequestId)); + } + return null; + } + await tx.update(toolActionRequests).set({ status: "executing", updatedAt: now }) + .where(eq(toolActionRequests.id, actionRequestId)); + return { actionRequestId, invocationId, claimedAt: now }; + }); +} + +/** Return only this still-undispatched claim to its existing approval queue. */ +export async function releaseToolApproval(db: Db, claim: ToolApprovalClaim) { + return db.transaction(async tx => { + const [invocation] = await tx.select().from(toolInvocations) + .where(eq(toolInvocations.id, claim.invocationId)).for("update"); + const [request] = await tx.select().from(toolActionRequests) + .where(and(eq(toolActionRequests.id, claim.actionRequestId), + eq(toolActionRequests.invocationId, claim.invocationId))).for("update"); + if (invocation?.status !== "awaiting_approval" || invocation.startedAt !== null || + request?.status !== "executing" || request.companyId !== invocation.companyId || + request.updatedAt.getTime() !== claim.claimedAt.getTime()) return false; + await notifyToolReviewChange(tx, request.companyId); + await tx.update(toolActionRequests).set({ status: "approved", updatedAt: new Date() }) + .where(eq(toolActionRequests.id, request.id)); + return true; + }); +} + +export async function bindVerifiedToolApprovalArguments(db: Db, claim: ToolApprovalClaim, hashes: { preview: string; verified: string }) { + return db.transaction(async tx => { + const [invocation] = await tx.select().from(toolInvocations) + .where(eq(toolInvocations.id, claim.invocationId)).for("update"); + const [request] = await tx.select().from(toolActionRequests) + .where(and(eq(toolActionRequests.id, claim.actionRequestId), + eq(toolActionRequests.invocationId, claim.invocationId))).for("update"); + if (invocation?.status !== "awaiting_approval" || invocation.startedAt !== null || + request?.status !== "executing" || request.companyId !== invocation.companyId || + request.updatedAt.getTime() !== claim.claimedAt.getTime() || + ![hashes.preview, hashes.verified].includes(request.canonicalArgumentsHash) || + ![hashes.preview, hashes.verified].includes(invocation.argumentsHash ?? "")) return false; + if (invocation.argumentsHash !== hashes.verified) await tx.update(toolInvocations) + .set({ argumentsHash: hashes.verified, updatedAt: new Date() }).where(eq(toolInvocations.id, claim.invocationId)); + return true; + }); +} + +export async function startToolInvocation(db: Db, invocationId: string, claim?: ToolApprovalClaim) { + const now = new Date(); + if (!claim) { + await db.update(toolInvocations).set({ status: "executing", startedAt: now, updatedAt: now }) + .where(eq(toolInvocations.id, invocationId)); + return true; + } + return db.transaction(async tx => { + const [invocation] = await tx.select().from(toolInvocations) + .where(eq(toolInvocations.id, invocationId)).for("update"); + const [request] = await tx.select().from(toolActionRequests) + .where(eq(toolActionRequests.id, claim.actionRequestId)).for("update"); + if (invocationId !== claim.invocationId || invocation?.status !== "awaiting_approval" || invocation.startedAt !== null || + request?.status !== "executing" || request.invocationId !== invocationId || request.companyId !== invocation.companyId || + request.updatedAt.getTime() !== claim.claimedAt.getTime()) return false; + await tx.update(toolInvocations).set({ status: "executing", approvalState: "approved", startedAt: now, updatedAt: now }) + .where(eq(toolInvocations.id, invocationId)); + return true; + }); +} + +export async function persistToolTestOutcome(db: Db, invocationId: string, outcome: ToolTestOutcome, claim?: ToolApprovalClaim) { + const now = new Date(); + const values = { status: outcome.status, ...(outcome.result ? { + resultHash: outcome.result.sha256 ?? null, resultSummary: outcome.result, + resultSizeBytes: outcome.result.sizeBytes ?? null, + } : {}), errorCode: outcome.errorCode, errorMessage: outcome.errorMessage, completedAt: now, updatedAt: now }; + if (!claim) { + await db.update(toolInvocations).set(values).where(eq(toolInvocations.id, invocationId)); + return true; + } + return db.transaction(async tx => { + const [invocation] = await tx.select({ status: toolInvocations.status }).from(toolInvocations) + .where(eq(toolInvocations.id, invocationId)).for("update"); + const [request] = await tx.select().from(toolActionRequests) + .where(eq(toolActionRequests.id, claim.actionRequestId)).for("update"); + if (!invocation || !["awaiting_approval", "executing"].includes(invocation.status) || + request?.status !== "executing" || request.updatedAt.getTime() !== claim.claimedAt.getTime()) return false; + await notifyToolReviewChange(tx, request.companyId); + await tx.update(toolInvocations).set(values).where(eq(toolInvocations.id, invocationId)); + await tx.update(toolActionRequests).set({ status: outcome.status === "succeeded" ? "executed" : "failed", + resolvedAt: now, updatedAt: now }).where(eq(toolActionRequests.id, claim.actionRequestId)); + return true; + }); +} diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 2ff322ad308..ff5e1028ffc 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1,3 +1,5 @@ +import { bindVerifiedToolApprovalArguments, claimToolTestApproval, releaseToolApproval, persistToolTestOutcome, startToolInvocation, type ToolApprovalClaim } from "./tool-approval-execution.js"; +import { approvedToolReviewCompanies, prepareApprovedToolReview, recoverApprovedToolReviews } from "./tool-review-recovery.js"; import { toolReviewExpiryService } from "./tool-review-expiry.js"; import { writeToolReviewChange } from "./tool-review-work-notifications.js"; import { restConnectorFor } from "./connectors/rest-registry.js"; @@ -1245,46 +1247,56 @@ export function createToolGatewayService( const interactions = issueThreadInteractionService(db); const basePolicyService = toolAccessPolicyService(db, { toolActionSigningSecret: options.toolActionSigningSecret }); const issuerDecisionCaches = new WeakMap(); + async function decideGatewayPolicy( + input: ToolAccessDecisionInput, + decisionOptions: NonNullable[1]> & { requiredGatewayAction?: ToolMcpGatewayTokenAction } = {}, + policy: Pick = basePolicyService, + database: Db = db, + ): Promise { + const issuerId = input.runContext?.originatingAgentId; + if (!issuerId || !input.runContext?.gatewayId || input.actor.actorType !== "agent") return policy.decide(input, decisionOptions); + if (!input.runContext.gatewayTokenId) throw new ToolGatewayHttpError(403, "Delegated gateway token is no longer available", "gateway_token_run_context_invalid"); + let issuerCache: ToolAccessDecisionCache | undefined; + if (decisionOptions.cache) { + issuerCache = issuerDecisionCaches.get(decisionOptions.cache); + if (!issuerCache) { issuerCache = createToolAccessDecisionCache(); issuerDecisionCaches.set(decisionOptions.cache, issuerCache); } + } + const requiredAction = decisionOptions.requiredGatewayAction ?? (input.consumeRateLimit ? "tools/call" : undefined); + const authorityKey = `gateway-issuer:${input.companyId}:${input.runContext.gatewayTokenId}:${issuerId}:${requiredAction ?? "inspect"}`; + let authority = issuerCache?.get(authorityKey); + if (!authority) { + authority = assertGatewayIssuerAuthority(input, requiredAction, database); + issuerCache?.set(authorityKey, authority); + } + await authority; + if (issuerId === input.actor.agentId) { + const decision = await policy.decide(input, decisionOptions); + return { ...decision, policyExplanation: { ...decision.policyExplanation, delegatedGatewayIssuer: issuerId } }; + } + const issuerInput = { ...input, actor: { ...input.actor, agentId: issuerId } }; + const issuer = await policy.decide(issuerInput, { ...decisionOptions, cache: issuerCache }); + const target = await policy.decide(input, { ...decisionOptions, sharedRateDecision: issuer }); + const chosen = !issuer.allowed && issuer.decision !== "require_approval" ? issuer + : !target.allowed ? target : issuer.decision === "require_approval" ? issuer : target; + if (input.consumeRateLimit) await policy.writeAudit(issuerInput, issuer); + return { ...chosen, matchedPolicyIds: [...new Set([...issuer.matchedPolicyIds, ...target.matchedPolicyIds])], policyExplanation: { + ...chosen.policyExplanation, + consumedRateAdmissions: target.policyExplanation?.consumedRateAdmissions, + delegatedCallerConsumedRateAdmissions: issuer.policyExplanation?.consumedRateAdmissions, + delegatedGatewayIssuer: issuerId, + } }; + } const policyService = { ...basePolicyService, - async decide(input: ToolAccessDecisionInput, decisionOptions: NonNullable[1]> & { requiredGatewayAction?: ToolMcpGatewayTokenAction } = {}): Promise { - const issuerId = input.runContext?.originatingAgentId; - if (!issuerId || !input.runContext?.gatewayId || input.actor.actorType !== "agent") return basePolicyService.decide(input, decisionOptions); - if (!input.runContext.gatewayTokenId) throw new ToolGatewayHttpError(403, "Delegated gateway token is no longer available", "gateway_token_run_context_invalid"); - let issuerCache: ToolAccessDecisionCache | undefined; - if (decisionOptions.cache) { - issuerCache = issuerDecisionCaches.get(decisionOptions.cache); - if (!issuerCache) { issuerCache = createToolAccessDecisionCache(); issuerDecisionCaches.set(decisionOptions.cache, issuerCache); } - } - const requiredAction = decisionOptions.requiredGatewayAction ?? (input.consumeRateLimit ? "tools/call" : undefined); - const authorityKey = `gateway-issuer:${input.companyId}:${input.runContext.gatewayTokenId}:${issuerId}:${requiredAction ?? "inspect"}`; - let authority = issuerCache?.get(authorityKey); - if (!authority) { - authority = assertGatewayIssuerAuthority(input, requiredAction); - issuerCache?.set(authorityKey, authority); - } - await authority; - if (issuerId === input.actor.agentId) { - const decision = await basePolicyService.decide(input, decisionOptions); - return { ...decision, policyExplanation: { ...decision.policyExplanation, delegatedGatewayIssuer: issuerId } }; - } - const issuerInput = { ...input, actor: { ...input.actor, agentId: issuerId } }; - const issuer = await basePolicyService.decide(issuerInput, { ...decisionOptions, cache: issuerCache }); - const target = await basePolicyService.decide(input, { ...decisionOptions, sharedRateDecision: issuer }); - const chosen = !issuer.allowed && issuer.decision !== "require_approval" ? issuer - : !target.allowed ? target : issuer.decision === "require_approval" ? issuer : target; - if (input.consumeRateLimit) await basePolicyService.writeAudit(issuerInput, issuer); - return { ...chosen, matchedPolicyIds: [...new Set([...issuer.matchedPolicyIds, ...target.matchedPolicyIds])], policyExplanation: { - ...chosen.policyExplanation, - consumedRateAdmissions: target.policyExplanation?.consumedRateAdmissions, - delegatedCallerConsumedRateAdmissions: issuer.policyExplanation?.consumedRateAdmissions, - delegatedGatewayIssuer: issuerId, - } }; + decide: decideGatewayPolicy, + admitInvocation(input: ToolAccessDecisionInput, invocationId: string, claim?: ToolApprovalClaim) { + return basePolicyService.admitInvocation(input, invocationId, claim, (policy, admissionInput, transaction) => + decideGatewayPolicy(admissionInput, { admittedInvocationId: invocationId, requiredGatewayAction: "tools/call" }, policy, transaction)); }, }; - async function assertGatewayIssuerAuthority(input: ToolAccessDecisionInput, requiredAction?: ToolMcpGatewayTokenAction) { - const [row] = await db.select({ token: toolMcpGatewayTokens, gateway: toolMcpGateways }).from(toolMcpGatewayTokens) + async function assertGatewayIssuerAuthority(input: ToolAccessDecisionInput, requiredAction?: ToolMcpGatewayTokenAction, database: Db = db) { + const [row] = await database.select({ token: toolMcpGatewayTokens, gateway: toolMcpGateways }).from(toolMcpGatewayTokens) .innerJoin(toolMcpGateways, eq(toolMcpGateways.id, toolMcpGatewayTokens.gatewayId)) .where(and(eq(toolMcpGatewayTokens.id, input.runContext!.gatewayTokenId!), eq(toolMcpGatewayTokens.companyId, input.companyId))); const issuerId = input.runContext!.originatingAgentId!; @@ -1299,8 +1311,8 @@ export function createToolGatewayService( const resource = { type: "connection_management" as const, companyId: input.companyId, gatewayId: row.gateway.id, profileId: row.gateway.profileId, purpose: "tool" as const, operation: "gateway_token.use", targetAgentId: row.gateway.agentId, projectId: row.gateway.projectId, issueId: row.gateway.issueId }; - await connectionManagementService(db).authorize(actor, "connections:gateway", resource, { mutation: false }); - await connectionManagementService(db).authorize(actor, "connections:test", resource, { mutation: false }); + await connectionManagementService(database).authorize(actor, "connections:gateway", resource, { mutation: false }); + await connectionManagementService(database).authorize(actor, "connections:test", resource, { mutation: false }); } async function recordToolInvocation(input: ToolAccessDecisionInput, decision: ToolAccessDecision) { try { @@ -7548,6 +7560,7 @@ export function createToolGatewayService( matchedPolicyIds: string[]; timeoutMs?: number; signal?: AbortSignal; + approvedClaim?: ToolApprovalClaim; managementContext?: ConnectionManagementContext; }): Promise< | { decision: "allowed"; invocationId: string; result: unknown } @@ -7557,14 +7570,9 @@ export function createToolGatewayService( error: { message: string; reasonCode: string }; } > { - await db - .update(toolInvocations) - .set({ - status: "executing", - startedAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(toolInvocations.id, args.invocationId)); + if (!(await startToolInvocation(db, args.invocationId, args.approvedClaim))) { + throw new ToolGatewayHttpError(409, "Tool action execution claim is no longer current", "action_already_consumed"); + } await writeAudit({ session: args.session, companyId: args.companyId, @@ -7587,6 +7595,8 @@ export function createToolGatewayService( }); const startedAt = Date.now(); + let connectedMcpExecution: Awaited> | Awaited> | null; + let resultValidation: Awaited>; try { await args.managementContext?.reauthorize(); if (args.managementContext?.actor.type === "agent") { @@ -7598,7 +7608,7 @@ export function createToolGatewayService( } } const executionTimeoutMs = timeoutMs(args.timeoutMs, args.tool); - const connectedMcpExecution = + connectedMcpExecution = (args.tool.providerType === "mcp_remote_http" || args.tool.providerType === "provider_rest") ? await executeRemoteHttpTool( args.session, @@ -7629,7 +7639,7 @@ export function createToolGatewayService( ); } const result = connectedMcpExecution.result; - const resultValidation = await validateToolContent({ + resultValidation = await validateToolContent({ value: result, direction: "result", sensitiveMode: "redact", @@ -7640,69 +7650,6 @@ export function createToolGatewayService( throw new ToolGatewayHttpError(502, String(providerResult.content || providerResult.error), "tool_error", { execution: connectedMcpExecution.execution }); } - await db - .update(toolInvocations) - .set({ - status: "succeeded", - resultHash: resultValidation.summary.sha256 ?? null, - resultSummary: resultValidation.summary, - resultSizeBytes: resultValidation.summary.sizeBytes ?? null, - completedAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(toolInvocations.id, args.invocationId)); - await writeToolCallEvent({ - invocationId: args.invocationId, - session: args.session, - eventType: "call_completed", - outcome: "success", - toolName: args.tool.name, - policyDecision: "allow", - reasonCode: "tool_completed", - argumentsSummary: args.argumentsSummary, - resultSummary: resultValidation.summary, - detectorMetadata: resultValidation.detectorMetadata, - metadata: { - source: "test", - headerSummary: connectedMcpExecution.headerSummary ?? undefined, - execution: connectedMcpExecution.execution, - }, - tool: args.tool, - }); - await writeAudit({ - session: args.session, - companyId: args.companyId, - agentId: args.agentId, - runId: args.session.runId, - issueId: args.session.issueId, - actorType: args.session.actorType, - actorId: args.session.actorId ?? args.userId, - action: "tool_gateway.call_completed", - details: { - source: "test", - invocationId: args.invocationId, - decision: "allow", - reasonCode: "tool_completed", - tool: args.tool.name, - ...toolAuditMetadata(args.tool), - durationMs: Date.now() - startedAt, - argumentsSummary: args.argumentsSummary, - result: summarizeResult(resultValidation.value), - resultSummary: resultValidation.summary, - promptInjection: resultValidation.detectorMetadata, - headerSummary: connectedMcpExecution.headerSummary ?? undefined, - execution: connectedMcpExecution.execution, - }, - }); - // After the bookkeeping above: a throw there is caught below, overwrites - // the row to failed, and emits — an earlier success emit would make one - // execution report both outcomes. - void emitConnectionInvoked(db, args.invocationId); - return { - decision: "allowed" as const, - invocationId: args.invocationId, - result: resultValidation.value, - }; } catch (err) { const status = err instanceof ToolGatewayHttpError ? err.status : 502; const reasonCode = @@ -7712,21 +7659,10 @@ export function createToolGatewayService( ? err.reasonCode : "tool_execution_failed"; const message = err instanceof Error ? err.message : String(err); - await db - .update(toolInvocations) - .set({ - status: - status === 504 - ? "timed_out" - : status === 429 - ? "rate_limited" - : "failed", - errorCode: reasonCode, - errorMessage: message, - completedAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(toolInvocations.id, args.invocationId)); + await persistToolTestOutcome(db, args.invocationId, { + status: status === 504 ? "timed_out" : status === 429 ? "rate_limited" : "failed", + errorCode: reasonCode, errorMessage: message, + }, args.approvedClaim); void emitConnectionInvoked(db, args.invocationId); await writeToolCallEvent({ invocationId: args.invocationId, @@ -7785,132 +7721,161 @@ export function createToolGatewayService( : {}), }; } + await persistToolTestOutcome(db, args.invocationId, { + status: "succeeded", result: resultValidation.summary, + }, args.approvedClaim); + await writeToolCallEvent({ + invocationId: args.invocationId, + session: args.session, + eventType: "call_completed", + outcome: "success", + toolName: args.tool.name, + policyDecision: "allow", + reasonCode: "tool_completed", + argumentsSummary: args.argumentsSummary, + resultSummary: resultValidation.summary, + detectorMetadata: resultValidation.detectorMetadata, + metadata: { + source: "test", + headerSummary: connectedMcpExecution.headerSummary ?? undefined, + execution: connectedMcpExecution.execution, + }, + tool: args.tool, + }); + await writeAudit({ + session: args.session, + companyId: args.companyId, + agentId: args.agentId, + runId: args.session.runId, + issueId: args.session.issueId, + actorType: args.session.actorType, + actorId: args.session.actorId ?? args.userId, + action: "tool_gateway.call_completed", + details: { + source: "test", + invocationId: args.invocationId, + decision: "allow", + reasonCode: "tool_completed", + tool: args.tool.name, + ...toolAuditMetadata(args.tool), + durationMs: Date.now() - startedAt, + argumentsSummary: args.argumentsSummary, + result: summarizeResult(resultValidation.value), + resultSummary: resultValidation.summary, + promptInjection: resultValidation.detectorMetadata, + headerSummary: connectedMcpExecution.headerSummary ?? undefined, + execution: connectedMcpExecution.execution, + }, + }); + void emitConnectionInvoked(db, args.invocationId); + return { + decision: "allowed" as const, + invocationId: args.invocationId, + result: resultValidation.value, + }; + + } + + function isApprovalPreparationRejection(error: unknown) { + return error instanceof ToolContentValidationError || + (error instanceof ToolGatewayHttpError || error instanceof HttpError) && error.status >= 400 && error.status < 500 && + ![408, 425, 429].includes(error.status); + } + + async function bindVerifiedApprovalArguments(invocation: typeof toolInvocations.$inferSelect, claim: ToolApprovalClaim, parameters: unknown) { + const preview = summarizeToolValue(parameters).sha256; + const verified = toolRequestIdentityHash(parameters, options.toolActionSigningSecret); + if (invocation.argumentsHash === preview && invocation.argumentsHash !== verified && + !await bindVerifiedToolApprovalArguments(db, claim, { preview, verified })) { + throw new ToolGatewayHttpError(409, "Tool action execution claim is no longer current", "action_already_consumed"); + } } - /** - * Drive a freshly-approved test-origin ask-first request to completion. The - * Test tab has no agent run to re-invoke the parked call, so approving it in - * the Review tab is what executes it — the live status panel then surfaces the - * real result. Reconstructs the impersonated test session and the signed - * arguments, and never throws: any failure is recorded on the invocation so it - * shows up as an error in the panel rather than rolling back the approval. - */ async function runApprovedTestInvocation( invocation: typeof toolInvocations.$inferSelect, - parameters: unknown, + signedPayload: NonNullable>, actionRequestId: string, ): Promise { - const agentId = invocation.agentId; - if (!invocation.connectionId || !agentId) return; - const delegated = invocation.policyExplanation?.delegatedConnectionTest as { - agentId: string; responsibleUserId: string; runId: string | null; keyId: string | null; - resource: ConnectionManagementContext["resource"]; - } | undefined; - const userId = delegated?.responsibleUserId ?? invocation.actorId ?? "board"; - const session: ToolGatewaySession = { - id: "test-call", - token: "test-call", - companyId: invocation.companyId, - agentId, - runId: null, - issueId: null, - projectId: null, - actorType: delegated ? "agent" : "user", - actorId: delegated?.agentId ?? userId, - responsibleUserId: userId, - createdAt: new Date(), - expiresAt: new Date(Date.now() + DEFAULT_SESSION_TTL_MS), - }; - let managementContext: ConnectionManagementContext | undefined; - let tool: ToolGatewayDescriptor | undefined; - try { - tool = ( - await connectedMcpToolsForConnection( - invocation.companyId, - invocation.connectionId, - ) - ).find( - (candidate) => - candidate.name === invocation.toolName || - candidate.upstreamToolName === invocation.toolName, - ); - } catch { - tool = undefined; - } - if (!tool) { - await db - .update(toolInvocations) - .set({ - status: "failed", - errorCode: "tool_not_found", - errorMessage: `Tool "${invocation.toolName}" is no longer connected`, - completedAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(toolInvocations.id, invocation.id)); - void emitConnectionInvoked(db, invocation.id); - await reflectToolActionInteractionLifecycle({ - actionRequestId, - status: "failed", - errorCode: "tool_not_found", - errorMessage: `Tool "${invocation.toolName}" is no longer connected`, - }); - return; - } - const argumentsSummary = (await validateToolContent({ - value: parameters, - direction: "arguments", - sensitiveMode: "redact", - promptInjectionMode: "ignore", - })).summary; + const claim = await claimToolTestApproval(db, actionRequestId, invocation.id); + if (!claim) return; + let execution: Parameters[0]; try { + const agentId = invocation.agentId; + if (!invocation.connectionId || !agentId) { + throw new ToolGatewayHttpError(409, "Test action context is no longer available", "action_origin_invalid"); + } + const delegated = invocation.policyExplanation?.delegatedConnectionTest as { + agentId: string; responsibleUserId: string; runId: string | null; keyId: string | null; + resource: ConnectionManagementContext["resource"]; + } | undefined; + const userId = delegated?.responsibleUserId ?? invocation.actorId ?? "board"; + const session: ToolGatewaySession = { + id: "test-call", token: "test-call", companyId: invocation.companyId, agentId, + runId: null, issueId: null, projectId: null, actorType: delegated ? "agent" : "user", actorId: delegated?.agentId ?? userId, + responsibleUserId: userId, createdAt: new Date(), expiresAt: new Date(Date.now() + DEFAULT_SESSION_TTL_MS), + }; + let managementContext: ConnectionManagementContext | undefined; if (delegated) { managementContext = await connectionManagementService(db).authorize({ type: "agent", companyId: invocation.companyId, agentId: delegated.agentId, onBehalfOfUserId: delegated.responsibleUserId, runId: delegated.runId, keyId: delegated.keyId, }, "connections:test", delegated.resource, { targetAgentIds: [agentId], afterHumanAction: true }); Object.assign(session, managementContext.workContext, { originatingAgentId: delegated.agentId, approvedInvocationId: invocation.id }); - const current = await policyService.decide(policyInputForTool({ session, tool, parameters }), { admittedInvocationId: invocation.id }); - if (!current.allowed && current.decision !== "require_approval") throw new ToolGatewayHttpError(403, current.explanation, current.reasonCode); } - const outcome = await runTestToolInvocation({ - session, - managementContext, - tool, - parameters, - invocationId: invocation.id, - companyId: invocation.companyId, - agentId, - userId, - argumentsSummary, - reasonCode: "approval_granted", - matchedPolicyIds: invocation.matchedPolicyIds ?? [], - }); - await reflectToolActionInteractionLifecycle({ - actionRequestId, - status: "error" in outcome ? "failed" : "executed", - ...("error" in outcome ? { errorCode: outcome.error.reasonCode, errorMessage: outcome.error.message } : {}), - }); + const tool = (await connectedMcpToolsForConnection(invocation.companyId, invocation.connectionId)) + .find(candidate => candidate.name === invocation.toolName || candidate.upstreamToolName === invocation.toolName); + if (!tool) throw new ToolGatewayHttpError(404, `Tool "${invocation.toolName}" is no longer connected`, "tool_not_found"); + if (!approvalSnapshotsMatch(signedPayload.approvalSnapshot, await connectedRemoteApprovalSnapshot(session, tool))) { + throw new ToolGatewayHttpError(409, "Tool definition or connection changed; request a new review", "approved_tool_target_changed"); + } + if (!(await approvedManagedArgumentsRemainCurrent(session, tool, signedPayload.arguments))) { + throw new ToolGatewayHttpError(409, "Approved tool action managed arguments changed after review", "approved_tool_managed_arguments_changed"); + } + await bindVerifiedApprovalArguments(invocation, claim, signedPayload.arguments); + const access = await basePolicyService.admitInvocation( + policyInputForTool({ session, tool, parameters: signedPayload.arguments, consumeRateLimit: false }), + invocation.id, + claim, + async (policy, admissionInput) => { + const callerSession = delegatedTestCallerSession(session); + const callerInput = callerSession ? policyInputForTool({ session: callerSession, tool, + parameters: signedPayload.arguments, consumeRateLimit: true }) : null; + const options = { admittedInvocationId: invocation.id }; + const caller = callerInput ? await policy.decide(callerInput, options) : null; + const target = await policy.decide(admissionInput, { ...options, sharedRateDecision: caller ?? undefined }); + const chosen = caller && !caller.allowed && caller.decision !== "require_approval" ? caller + : !target.allowed ? target : caller?.decision === "require_approval" ? caller : target; + if (callerInput && caller) await policy.writeAudit(callerInput, caller); + return { ...chosen, policyExplanation: { + ...chosen.policyExplanation, + consumedRateAdmissions: target.policyExplanation?.consumedRateAdmissions, + delegatedCallerConsumedRateAdmissions: caller?.policyExplanation?.consumedRateAdmissions, + } }; + }, + ); + if (!access.allowed && access.decision !== "require_approval") { + throw new ToolGatewayHttpError(403, access.explanation, access.reasonCode); + } + const argumentsSummary = (await validateToolContent({ value: signedPayload.arguments, + direction: "arguments", sensitiveMode: "redact", promptInjectionMode: "ignore" })).summary; + execution = { session, managementContext, tool, parameters: signedPayload.arguments, invocationId: invocation.id, + companyId: invocation.companyId, agentId, userId, argumentsSummary, reasonCode: "approval_granted", + approvedClaim: claim, matchedPolicyIds: invocation.matchedPolicyIds ?? [] }; } catch (err) { + if (!isApprovalPreparationRejection(err)) { + // Preparation has not entered runTestToolInvocation. Preserve approval + // through dependency failures; the claim owner still verifies no dispatch. + await releaseToolApproval(db, claim); + throw err; + } const message = err instanceof Error ? err.message : String(err); - await db - .update(toolInvocations) - .set({ - status: "failed", - errorCode: "tool_execution_failed", - errorMessage: message, - completedAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(toolInvocations.id, invocation.id)); - void emitConnectionInvoked(db, invocation.id); - await reflectToolActionInteractionLifecycle({ - actionRequestId, - status: "failed", - errorCode: "tool_execution_failed", - errorMessage: message, - }); + const errorCode = err instanceof ToolGatewayHttpError || err instanceof ToolContentValidationError + ? err.reasonCode : "tool_execution_failed"; + const settled = await persistToolTestOutcome(db, invocation.id, { status: "failed", errorCode, errorMessage: message }, claim); + if (settled) void emitConnectionInvoked(db, invocation.id); + return; } + await runTestToolInvocation(execution); } async function waitForActionRequestExecution(actionRequestId: string) { @@ -8080,6 +8045,7 @@ export function createToolGatewayService( } async function expireApprovedActionForManagedArgumentDrift(input: { + companyId: string; actionRequestId: string; invocationId: string; toolName: string; @@ -8097,7 +8063,7 @@ export function createToolGatewayService( ); const now = new Date(); await options.beforeManagedArgumentDriftExpiry?.(); - const [expired] = await db + const [expired] = await writeToolReviewChange(db, input.companyId, tx => tx .update(toolActionRequests) .set({ status: "expired", resolvedAt: now, updatedAt: now }) .where( @@ -8108,7 +8074,7 @@ export function createToolGatewayService( : eq(toolActionRequests.status, "approved"), ), ) - .returning({ id: toolActionRequests.id }); + .returning({ id: toolActionRequests.id })); if (!expired) return error; await db .update(toolInvocations) @@ -8227,7 +8193,7 @@ export function createToolGatewayService( ); } - const [claimed] = await db + const [claimed] = await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "executing", updatedAt: new Date() }) .where( @@ -8236,7 +8202,7 @@ export function createToolGatewayService( eq(toolActionRequests.status, "approved"), ), ) - .returning(); + .returning()); if (!claimed) { const settled = await waitForActionRequestExecution(actionRequest.id); const [settledInvocation] = await db @@ -8262,92 +8228,141 @@ export function createToolGatewayService( ); } - // Terminal-issue expiry revokes pending/approved requests, but a claim that - // committed just before the issue closed slips past that revocation. Recheck - // the issue after winning the claim so a governed action never runs external - // side effects for an issue that is already done or cancelled. - const issue = await assertIssueOpenForApprovedAction({ - claimed, - invocation, - }); - - const signedPayload = readSignedToolArgumentsPayload({ - signedArguments: claimed.signedArguments, - invocationId: invocation.id, - toolName: invocation.toolName, - signingSecret: options.toolActionSigningSecret, - }); - if (!signedPayload) { - const error = new ToolGatewayHttpError( - 409, - "Approved tool action arguments signature is invalid", - "signed_arguments_invalid", - ); - await markApprovedActionFailed({ - actionRequestId: claimed.id, - invocationId: invocation.id, - claimUpdatedAt: claimed.updatedAt, - expectedInvocationStatus: "awaiting_approval", - error, - }); - throw error; - } - if (signedPayload.executionOnApprove !== true) { - const error = new ToolGatewayHttpError( - 409, - "This approval predates execute-on-approve and must remain inert", - "legacy_approved_action_inert", - ); - await markApprovedActionFailed({ - actionRequestId: claimed.id, - invocationId: invocation.id, - claimUpdatedAt: claimed.updatedAt, - expectedInvocationStatus: "awaiting_approval", - error, - }); - throw error; - } - - const session: ToolGatewaySession = { - id: `approved-action:${claimed.id}`, - token: "", - companyId: invocation.companyId, - agentId: invocation.agentId, - runId: invocation.runId, - issueId: invocation.issueId, - projectId: issue?.projectId ?? null, - gatewayId: invocation.gatewayId, - gatewayPublicId: invocation.gatewayPublicId, - gatewayTokenId: invocation.gatewayTokenId, - actorType: "agent", - actorId: invocation.actorType === "agent" ? invocation.actorId ?? invocation.agentId : invocation.agentId, - originatingAgentId: typeof invocation.policyExplanation?.delegatedGatewayIssuer === "string" ? invocation.policyExplanation.delegatedGatewayIssuer : invocation.actorType === "agent" && invocation.gatewayId && !invocation.runId && !invocation.gatewayTokenId ? invocation.actorId : null, - createdAt: new Date(), - expiresAt: new Date(Date.now() + DEFAULT_SESSION_TTL_MS), + const claim: ToolApprovalClaim = { + actionRequestId: claimed.id, invocationId: invocation.id, claimedAt: claimed.updatedAt, }; - let tool: ToolGatewayDescriptor; - let liveApprovalSnapshot: Awaited< - ReturnType - >; - try { + const agentId = invocation.agentId; + async function prepareExecution() { + // Terminal-issue expiry revokes pending/approved requests, but a claim that + // committed just before the issue closed slips past that revocation. Recheck + // the issue after winning the claim so a governed action never runs external + // side effects for an issue that is already done or cancelled. + const issue = await assertIssueOpenForApprovedAction({ + claimed, + invocation, + }); + + const signedPayload = readSignedToolArgumentsPayload({ + signedArguments: claimed.signedArguments, + invocationId: invocation.id, + toolName: invocation.toolName, + signingSecret: options.toolActionSigningSecret, + }); + if (!signedPayload) { + const error = new ToolGatewayHttpError( + 409, + "Approved tool action arguments signature is invalid", + "signed_arguments_invalid", + ); + throw error; + } + if (signedPayload.executionOnApprove !== true) { + const error = new ToolGatewayHttpError( + 409, + "This approval predates execute-on-approve and must remain inert", + "legacy_approved_action_inert", + ); + throw error; + } + + const session: ToolGatewaySession = { + id: `approved-action:${claimed.id}`, + token: "", + companyId: invocation.companyId, + agentId, + runId: invocation.runId, + issueId: invocation.issueId, + projectId: issue?.projectId ?? null, + gatewayId: invocation.gatewayId, + gatewayPublicId: invocation.gatewayPublicId, + gatewayTokenId: invocation.gatewayTokenId, + actorType: invocation.actorType as ToolGatewaySession["actorType"], + actorId: invocation.actorId, + originatingAgentId: typeof invocation.policyExplanation?.delegatedGatewayIssuer === "string" ? invocation.policyExplanation.delegatedGatewayIssuer : invocation.actorType === "agent" && invocation.gatewayId && !invocation.runId && !invocation.gatewayTokenId ? invocation.actorId : null, + createdAt: new Date(), + expiresAt: new Date(Date.now() + DEFAULT_SESSION_TTL_MS), + }; await restoreApprovedActionIdentity( session, signedPayload.identityContextId, ); session.approvedInvocationId = invocation.id; session.approvedSlackInvocationId = invocation.id; - tool = await findToolForSession(session, invocation.toolName); - liveApprovalSnapshot = await connectedRemoteApprovalSnapshot( + const tool = await findToolForSession(session, invocation.toolName); + const liveApprovalSnapshot = await connectedRemoteApprovalSnapshot( session, tool, ); - const currentAccess = await policyService.decide( + if ( + !approvalSnapshotsMatch( + signedPayload.approvalSnapshot, + liveApprovalSnapshot, + ) + ) { + const error = new ToolGatewayHttpError( + 409, + "Approved tool action target changed after review", + "approved_tool_target_changed", + ); + throw error; + } + const parameters = signedPayload.arguments; + const canonicalArguments = canonicalToolArguments(parameters); + if ( + ![ + toolRequestIdentityHash(parameters, options.toolActionSigningSecret), + // Historical approvals used the redacted preview hash. Their signature + // still binds the complete canonical arguments checked below. + summarizeToolValue(parameters).sha256, + ].includes(claimed.canonicalArgumentsHash) || + !verifyToolArgumentsSignature({ + signedArguments: claimed.signedArguments, + invocationId: invocation.id, + toolName: invocation.toolName, + canonicalArguments, + approvalSnapshot: signedPayload.approvalSnapshot, + executionOnApprove: true, + identityContextId: signedPayload.identityContextId, + signingSecret: options.toolActionSigningSecret, + }) + ) { + const error = new ToolGatewayHttpError( + 409, + "Approved tool action arguments do not match reviewed hash", + "signed_arguments_mismatch", + ); + throw error; + } + const managedArgumentsRemainCurrent = + await approvedManagedArgumentsRemainCurrent(session, tool, parameters); + if (!managedArgumentsRemainCurrent) { + throw await expireApprovedActionForManagedArgumentDrift({ + companyId: claimed.companyId, + actionRequestId: claimed.id, + invocationId: invocation.id, + toolName: invocation.toolName, + ownsExecutingClaim: true, + }); + } + + const argumentsSummary = (await validateToolContent({ + value: parameters, + direction: "arguments", + sensitiveMode: "redact", + promptInjectionMode: "ignore", + })).summary; + // Validate the signed and managed payload before consuming admission. + // Historical payload drift belongs to the expiry owner above. + await bindVerifiedApprovalArguments(invocation, claim, signedPayload.arguments); + const currentAccess = await policyService.admitInvocation( policyInputForTool({ session, tool, parameters: signedPayload.arguments, + consumeRateLimit: false, }), - { admittedInvocationId: invocation.id, requiredGatewayAction: "tools/call" }, + invocation.id, + claim, ); if ( !currentAccess.allowed && @@ -8358,120 +8373,43 @@ export function createToolGatewayService( currentAccess.explanation, currentAccess.reasonCode, ); - } catch (error) { - await markApprovedActionFailed({ - actionRequestId: claimed.id, - invocationId: invocation.id, - claimUpdatedAt: claimed.updatedAt, - expectedInvocationStatus: "awaiting_approval", - error, - }); - throw error; + // Final recheck at the last DB write before dispatch: tool and snapshot + // resolution above involve network calls, so re-verify the issue is still + // open now that only the provider call remains. A close that commits after + // this read has raced an execution that was approved, claimed, and verified + // while the issue was open; that instant is irreducible for an external + // side effect gated by DB state (holding a DB lock across a remote provider + // call is not an option), and the accepted linearization is that the + // execution wins — its result still lands on the expired card via + // reflectToolActionInteractionLifecycle. + await assertIssueOpenForApprovedAction({ claimed, invocation }); + return { session, tool, parameters, argumentsSummary }; } - if ( - !approvalSnapshotsMatch( - signedPayload.approvalSnapshot, - liveApprovalSnapshot, - ) - ) { - const error = new ToolGatewayHttpError( - 409, - "Approved tool action target changed after review", - "approved_tool_target_changed", - ); - await markApprovedActionFailed({ - actionRequestId: claimed.id, - invocationId: invocation.id, - claimUpdatedAt: claimed.updatedAt, - expectedInvocationStatus: "awaiting_approval", - error, - }); - throw error; - } - const parameters = signedPayload.arguments; - const canonicalArguments = canonicalToolArguments(parameters); - if ( - ![ - toolRequestIdentityHash(parameters, options.toolActionSigningSecret), - // Historical approvals used the redacted preview hash. Their signature - // still binds the complete canonical arguments checked below. - summarizeToolValue(parameters).sha256, - ].includes(claimed.canonicalArgumentsHash) || - !verifyToolArgumentsSignature({ - signedArguments: claimed.signedArguments, - invocationId: invocation.id, - toolName: invocation.toolName, - canonicalArguments, - approvalSnapshot: signedPayload.approvalSnapshot, - executionOnApprove: true, - identityContextId: signedPayload.identityContextId, - signingSecret: options.toolActionSigningSecret, - }) - ) { - const error = new ToolGatewayHttpError( - 409, - "Approved tool action arguments do not match reviewed hash", - "signed_arguments_mismatch", - ); - await markApprovedActionFailed({ - actionRequestId: claimed.id, - invocationId: invocation.id, - claimUpdatedAt: claimed.updatedAt, - expectedInvocationStatus: "awaiting_approval", - error, - }); - throw error; - } - let managedArgumentsRemainCurrent: boolean; + let prepared: Awaited>; try { - managedArgumentsRemainCurrent = - await approvedManagedArgumentsRemainCurrent(session, tool, parameters); + prepared = await prepareExecution(); } catch (error) { - await markApprovedActionFailed({ - actionRequestId: claimed.id, - invocationId: invocation.id, - claimUpdatedAt: claimed.updatedAt, - expectedInvocationStatus: "awaiting_approval", - error, - }); + if (isApprovalPreparationRejection(error)) { + await markApprovedActionFailed({ + actionRequestId: claimed.id, + invocationId: invocation.id, + claimUpdatedAt: claimed.updatedAt, + expectedInvocationStatus: "awaiting_approval", + error, + }); + } else { + await releaseToolApproval(db, claim); + } throw error; } - if (!managedArgumentsRemainCurrent) { - throw await expireApprovedActionForManagedArgumentDrift({ - actionRequestId: claimed.id, - invocationId: invocation.id, - toolName: invocation.toolName, - ownsExecutingClaim: true, - }); - } - - const argumentsSummary = (await validateToolContent({ - value: parameters, - direction: "arguments", - sensitiveMode: "redact", - promptInjectionMode: "ignore", - })).summary; - // Final recheck at the last DB write before dispatch: tool and snapshot - // resolution above involve network calls, so re-verify the issue is still - // open now that only the provider call remains. A close that commits after - // this read has raced an execution that was approved, claimed, and verified - // while the issue was open; that instant is irreducible for an external - // side effect gated by DB state (holding a DB lock across a remote provider - // call is not an option), and the accepted linearization is that the - // execution wins — its result still lands on the expired card via - // reflectToolActionInteractionLifecycle. - await assertIssueOpenForApprovedAction({ claimed, invocation }); + const { session, tool, parameters, argumentsSummary } = prepared; const startedAt = Date.now(); - await db - .update(toolInvocations) - .set({ - status: "executing", - approvalState: "approved", - startedAt: new Date(), - updatedAt: new Date(), - }) - .where(eq(toolInvocations.id, invocation.id)); + // Keep dispatch and outcome persistence outside the preparation retry + // boundary. Only the still-current claim may record the provider start. + if (!(await startToolInvocation(db, invocation.id, claim))) { + throw new ToolGatewayHttpError(409, "Tool action execution claim is no longer current", "action_already_consumed"); + } await reflectToolActionInteractionLifecycle({ actionRequestId: claimed.id, status: "executing", @@ -9977,7 +9915,33 @@ export function createToolGatewayService( actionReviewExpiryDeadlines: reviewExpiry.deadlines, sweepExpiredActionReviewsForCompany: reviewExpiry.sweepForCompany, - async sweepActionReviews(input: { includeExpiry?: boolean } = {}) { + approvedActionReviewCompanies: (companyId?: string) => approvedToolReviewCompanies(db, companyId), + + async recoverApprovedActionReviews(companyId?: string) { + return recoverApprovedToolReviews(db, async row => { + try { + const prepared = await prepareApprovedToolReview(db, { actionRequestId: row.request.id, + invocationId: row.invocation.id, signingSecret: options.toolActionSigningSecret }); + if (prepared.kind === "skip") return false; + if (prepared.kind === "settled") { + await reflectToolActionInteractionLifecycle(prepared.projection); + return false; + } + const { request, invocation, payload } = prepared; + if (!isTestOriginInvocation(invocation) && payload.executionOnApprove !== true) return false; + await this.approveActionRequest({ companyId: request.companyId, actionRequestId: request.id, + actor: { userId: request.decidedByUserId ?? request.resolvedByUserId } }); + const [remaining] = await db.select({ status: toolActionRequests.status }).from(toolActionRequests) + .where(eq(toolActionRequests.id, request.id)); + return remaining?.status === "approved"; + } catch (err) { + logger.warn({ err, actionRequestId: row.request.id }, "Could not recover approved tool action"); + return true; + } + }, companyId); + }, + + async sweepActionReviews(input: { includeExpiry?: boolean; includeApproved?: boolean } = {}) { // A process may stop between persisting a provider outcome and updating the // feed projection. Reconcile from authoritative rows before delivering it. const unreflected = await db @@ -10016,38 +9980,21 @@ export function createToolGatewayService( const staleAt = new Date(now.getTime() - 10 * 60_000); let cursor: string | undefined; let scanned = input.includeExpiry === false ? 0 : (await reviewExpiry.sweep(now)).scanned; + if (input.includeApproved !== false) scanned += (await this.recoverApprovedActionReviews()).scanned; for (;;) { const rows = await db .select() .from(toolActionRequests) .where( and( - or( - eq(toolActionRequests.status, "approved"), - and( - eq(toolActionRequests.status, "executing"), - lte(toolActionRequests.updatedAt, staleAt), - ), - ), + eq(toolActionRequests.status, "executing"), + lte(toolActionRequests.updatedAt, staleAt), cursor ? gt(toolActionRequests.id, cursor) : undefined, ), ) .orderBy(asc(toolActionRequests.id)) .limit(100); for (const row of rows) { - if (row.status === "approved") { - await this.approveActionRequest({ - companyId: row.companyId, - actionRequestId: row.id, - actor: { userId: row.decidedByUserId ?? row.resolvedByUserId }, - }).catch((error) => - logger.warn( - { err: error, actionRequestId: row.id }, - "Could not recover approved tool action", - ), - ); - continue; - } const status = "failed"; const errorCode = "tool_execution_outcome_unknown"; const errorMessage = "Execution was interrupted; the external outcome is unknown. Inspect the provider before retrying."; @@ -10239,10 +10186,12 @@ export function createToolGatewayService( actionRequestId: actionRequest.id, status: "approved", }); - if ( - !isTestOriginInvocation(invocation) && - signedPayload.executionOnApprove === true - ) { + if (isTestOriginInvocation(invocation)) { + await runApprovedTestInvocation(invocation, signedPayload, actionRequest.id); + const [settled] = await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, actionRequest.id)); + return actionRequestResolution(settled ?? actionRequest); + } + if (signedPayload.executionOnApprove === true) { try { await executeApprovedAgentInvocation({ actionRequest, invocation }); } catch { @@ -10318,13 +10267,18 @@ export function createToolGatewayService( "Tool definition or connection changed; request a new review", "approved_tool_target_changed", ); + const previewHash = summarizeToolValue(signedPayload.arguments).sha256; const access = await policyService.decide( policyInputForTool({ session, tool, parameters: signedPayload.arguments, + consumeRateLimit: false, }), - { admittedInvocationId: invocation.id, requiredGatewayAction: "tools/call" }, + { admittedInvocationId: invocation.id, requiredGatewayAction: "tools/call", + admittedPreviewHash: [previewHash, toolRequestIdentityHash(signedPayload.arguments, options.toolActionSigningSecret)] + .includes(actionRequest.canonicalArgumentsHash) ? previewHash : undefined, + }, ); if (!access.allowed && access.decision !== "require_approval") throw new ToolGatewayHttpError( @@ -10347,7 +10301,7 @@ export function createToolGatewayService( if (isTestOriginInvocation(invocation)) { await runApprovedTestInvocation( { ...invocation, approvalState: "approved" }, - signedPayload.arguments, + signedPayload, updated.id, ); } else if (signedPayload.executionOnApprove === true) { @@ -10911,7 +10865,7 @@ export function createToolGatewayService( ); await options.beforeLegacyApprovedActionClaim?.(); const claimedAt = new Date(); - const [claimed] = await db + const [claimed] = await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "executing", @@ -10924,7 +10878,7 @@ export function createToolGatewayService( eq(toolActionRequests.status, "approved"), ), ) - .returning(); + .returning()); if (!claimed) { throw new ToolGatewayHttpError( 409, @@ -11004,13 +10958,14 @@ export function createToolGatewayService( )) ) { throw await expireApprovedActionForManagedArgumentDrift({ + companyId: actionRequest.companyId, actionRequestId: actionRequest.id, invocationId: storedInvocation.id, toolName: storedInvocation.toolName, }); } const claimedAt = new Date(); - const [claimed] = await db + const [claimed] = await writeToolReviewChange(db, actionRequest.companyId, tx => tx .update(toolActionRequests) .set({ status: "executing", @@ -11023,7 +10978,7 @@ export function createToolGatewayService( eq(toolActionRequests.status, "approved"), ), ) - .returning(); + .returning()); if (!claimed) { throw new ToolGatewayHttpError( 409, diff --git a/server/src/services/tool-review-recovery-work.ts b/server/src/services/tool-review-recovery-work.ts new file mode 100644 index 00000000000..5611f5cf155 --- /dev/null +++ b/server/src/services/tool-review-recovery-work.ts @@ -0,0 +1,33 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerScopedDeadlineWork } from "./scoped-deadline-work.js"; +import type { createToolGatewayService } from "./tool-gateway.js"; + +export function registerToolReviewRecoveryWork( + coordinator: ReturnType, + gateway: Pick, "recoverApprovedActionReviews" | "approvedActionReviewCompanies">, + canRun: () => boolean, +) { + // Dispatch failures are reported by the domain owner, not thrown. Preserve + // their retry deadline when another write reloads the same company's work. + const retries = new Map(); + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.toolReviewRecovery, + batchSize: 1, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: (entry, companyId?: string) => companyId === undefined || entry.companyId === companyId, + load: async (companyId?: string) => { + const rows = await gateway.approvedActionReviewCompanies(companyId); + const retained = new Set(rows.map(row => row.companyId)); + for (const id of retries.keys()) if ((companyId === undefined || id === companyId) && !retained.has(id)) retries.delete(id); + return rows.map(row => ({ ...row, id: row.companyId, dueAt: retries.get(row.companyId) ?? Date.now() })); + }, + check: async entry => { + const result = await gateway.recoverApprovedActionReviews(entry.companyId); + if (!result.retry) { retries.delete(entry.companyId); return null; } + const dueAt = Date.now() + 30_000; + retries.set(entry.companyId, dueAt); + return { ...entry, dueAt }; + }, + }, canRun); +} diff --git a/server/src/services/tool-review-recovery.ts b/server/src/services/tool-review-recovery.ts new file mode 100644 index 00000000000..69aa98c76f0 --- /dev/null +++ b/server/src/services/tool-review-recovery.ts @@ -0,0 +1,119 @@ +import { and, asc, eq, gt } from "drizzle-orm"; +import { approvals, toolActionRequests, toolInvocations, type Db } from "@paperclipai/db"; + +import { readSignedToolArgumentsPayload } from "./tool-content-guards.js"; +import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; + +type ReviewProjection = { + actionRequestId: string; + status: "executed" | "executing" | "failed" | "cancelled"; + errorCode?: string | null; + errorMessage?: string | null; + resultSummary?: string | null; +}; + +/** Reconcile saved history before dispatch; never infer a safe provider replay. */ +export async function prepareApprovedToolReview(db: Db, input: { + actionRequestId: string; + invocationId: string; + signingSecret?: string; +}) { + return db.transaction(async tx => { + // Match Test-tab claims/outcome persistence. Recheck after the locks because + // a foreground approval may have claimed the row since the recovery scan. + const [invocation] = await tx.select().from(toolInvocations) + .where(eq(toolInvocations.id, input.invocationId)).for("update"); + const [request] = await tx.select().from(toolActionRequests) + .where(and(eq(toolActionRequests.id, input.actionRequestId), + eq(toolActionRequests.invocationId, input.invocationId))).for("update"); + if (!invocation || request?.status !== "approved") return { kind: "skip" } as const; + const now = new Date(); + + async function settle(projection: ReviewProjection, failure?: { + status: "failed" | "denied"; + errorCode: string; + errorMessage: string; + }) { + await notifyToolReviewChange(tx, request.companyId); + if (failure) await tx.update(toolInvocations).set({ ...failure, completedAt: now, updatedAt: now }) + .where(eq(toolInvocations.id, invocation.id)); + await tx.update(toolActionRequests).set({ status: projection.status, + ...(projection.status === "executing" + ? { updatedAt: invocation.startedAt ?? invocation.updatedAt } + : { resolvedAt: invocation.completedAt ?? now, updatedAt: now }), + }).where(eq(toolActionRequests.id, request.id)); + return { kind: "settled", projection } as const; + } + function fail(status: "failed" | "cancelled", invocationStatus: "failed" | "denied", errorCode: string, errorMessage: string) { + return settle({ actionRequestId: request.id, status, errorCode, errorMessage }, + { status: invocationStatus, errorCode, errorMessage }); + } + if (invocation.companyId !== request.companyId) { + // Quarantine the request without modifying another company's invocation. + return settle({ actionRequestId: request.id, status: "failed", + errorCode: "action_context_mismatch", errorMessage: "Tool action and invocation companies do not match." }); + } + if (["succeeded", "failed", "denied", "cancelled", "timed_out", "rate_limited", "executing"].includes(invocation.status)) { + return settle({ actionRequestId: request.id, + status: invocation.status === "succeeded" ? "executed" + : invocation.status === "executing" ? "executing" + : invocation.status === "cancelled" ? "cancelled" : "failed", + errorCode: invocation.errorCode, errorMessage: invocation.errorMessage, + resultSummary: invocation.resultSummary?.summary }); + } + if (invocation.status !== "awaiting_approval") { + return fail("failed", "failed", "tool_recovery_state_invalid", + "Approved action has inconsistent invocation history. Inspect the provider before retrying; recovery did not replay it."); + } + + // Missing-key/configuration errors deliberately propagate as retryable; + // only a decoded invalid signature is a permanent request failure. + const payload = readSignedToolArgumentsPayload({ signedArguments: request.signedArguments, + invocationId: invocation.id, toolName: invocation.toolName, signingSecret: input.signingSecret }); + if (!payload) return fail("cancelled", "failed", "action_request_invalidated", + "Stored tool arguments are no longer verifiable; submit a new action for review."); + if (request.approvalId) { + const [formal] = await tx.select({ status: approvals.status }).from(approvals) + .where(and(eq(approvals.id, request.approvalId), eq(approvals.companyId, request.companyId))); + if (!formal || formal.status === "rejected" || formal.status === "cancelled") { + return fail("cancelled", "denied", "formal_approval_unavailable", + "Required formal board approval is unavailable or was declined; submit a new action for review."); + } + // Human input has no deadline. The approval owner publishes a committed + // resolution/cancellation to this queue; startup also rediscovers the row. + if (formal.status === "pending" || formal.status === "revision_requested") { + return { kind: "skip" } as const; + } + } + return { kind: "ready", request, invocation, payload } as const; + }); +} + +export function approvedToolReviewCompanies(db: Db, companyId?: string) { + return db.select({ companyId: toolActionRequests.companyId }).from(toolActionRequests) + .innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and(eq(toolActionRequests.status, "approved"), + companyId ? eq(toolActionRequests.companyId, companyId) : undefined)) + .groupBy(toolActionRequests.companyId); +} + +export async function recoverApprovedToolReviews(db: Db, recover: (input: { + request: typeof toolActionRequests.$inferSelect; + invocation: typeof toolInvocations.$inferSelect; +}) => Promise, companyId?: string) { + let cursor: string | undefined; + let scanned = 0, retry = false; + for (;;) { + const rows = await db.select({ request: toolActionRequests, invocation: toolInvocations }) + .from(toolActionRequests).innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and(eq(toolActionRequests.status, "approved"), + companyId ? eq(toolActionRequests.companyId, companyId) : undefined, + cursor ? gt(toolActionRequests.id, cursor) : undefined)) + .orderBy(asc(toolActionRequests.id)).limit(100); + for (const row of rows) retry = await recover(row) || retry; + scanned += rows.length; + if (rows.length < 100) break; + cursor = rows[rows.length - 1].request.id; + } + return { scanned, retry }; +} diff --git a/server/src/services/tool-review-work-notifications.ts b/server/src/services/tool-review-work-notifications.ts index 19c8e433d75..1fc52d02b56 100644 --- a/server/src/services/tool-review-work-notifications.ts +++ b/server/src/services/tool-review-work-notifications.ts @@ -1,8 +1,9 @@ import type { Db } from "@paperclipai/db"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; -export function notifyToolReviewChange(transaction: object, companyId: string) { - return notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewExpiry, companyId); +export async function notifyToolReviewChange(transaction: object, companyId: string) { + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewExpiry, companyId); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewRecovery, companyId); } export function writeToolReviewChange(db: Db, companyId: string, write: (transaction: Db) => Promise): Promise { diff --git a/server/src/services/tool-runtime-metrics.ts b/server/src/services/tool-runtime-metrics.ts index 3d877f0dee9..c984ba24915 100644 --- a/server/src/services/tool-runtime-metrics.ts +++ b/server/src/services/tool-runtime-metrics.ts @@ -11,7 +11,7 @@ function minuteBucket(date: Date): Date { } export async function incrementToolRuntimeMetricCounter( - db: Db, + db: Pick, input: { companyId: string; metric: string; @@ -42,7 +42,7 @@ export async function incrementToolRuntimeMetricCounter( }); } -export async function recordToolRuntimeAuditWriteFailure(db: Db, companyId: string) { +export async function recordToolRuntimeAuditWriteFailure(db: Pick, companyId: string) { try { await incrementToolRuntimeMetricCounter(db, { companyId, From 56bc470ab25efa265ed7248bb9a9bc73f7a031c4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:30:37 -0500 Subject: [PATCH 10/41] perf(server): project continuation history result fields (#17930) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A continuation carries completed tool receipts so the agent can avoid repeat actions. > - It also checks saved conversation and workspace recovery fields before choosing a prior terminal run. > - The history query transfers every result field for every matching run. > - Native output and unused receipt request fields can make this read large. > - This pull request selects the fields those consumers use and keeps each completed receipt result in full. > - The benefit is less database transfer with the same history, recovery decisions, and completed-action evidence. ## Linked Issues or Issue Description **What existing behavior does this improve?** The history read in `buildExecutionContinuation`. **Current behavior** The query returns complete `result_json` values for all matching runs. Consumers read completed receipt state, operation ID and result, plus conversation and workspace recovery controls. Large native output and receipt request fields cross the connection without being used. **Proposed behavior** Select only those result fields in PostgreSQL. Keep typed policy and recovery values. Keep all eligible completed receipts and their complete result values. Retain missing versus null receipt results, run and receipt order, company/task/agent scope, and the source-run exception. Keep the full retained history without a new limit. **Reason and benefit** The synthetic wide-output fixture removes 99.879% of this query's DataRow bytes. A required 2 MiB receipt-result control saves only 0.0074%, because that complete result is needed. Actual savings depend on result contents; the wide-output result is not a forecast for all deployments. **Breaking changes** None intended. No schema, API, migration, workflow, history limit, or receipt truncation changes. **Additional context** Related: #17481 projects a separate recovery read, and #17896 narrows explicit continuation admission. This PR covers the continuation history query and has no dependency on the other byte reduction PRs. ## What Changed - Project completed tool receipts to their state, operation ID and complete result. Omit fields that the continuation does not read. - Project typed conversation/workspace controls and the recovery schema. Keep the shared policy helper unchanged. - Add real PostgreSQL comparisons for complete continuation envelopes, ordering, scope, legacy JSON, recovery flags, and receipt result preservation. - Add external isolated database support to the existing continuation tests and document a reproducible query benchmark. ## Verification - Pinned baseline: `3cd3af5d75f200185ed4d9703a3145594a489b13`. - PostgreSQL 17.11 with synthetic local data. The harness captures SQL from the real baseline and treatment services. Imported helpers remain unchanged. - Command: `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-history.ts --baseline-ref=3cd3af5d75f200185ed4d9703a3145594a489b13`. Set `PAPERCLIP_CONTINUATION_HISTORY_BENCHMARK_DATABASE_URL` to a disposable database ending in `_test` to use an external server. The default uses disposable embedded PostgreSQL. - The baseline was measured before the production edit with `--baseline-only`. The treatment compares complete envelopes, every retained scalar column, completed receipts, and shared policy decisions. The wide unused-output fixture must remove more than 90% of DataRow bytes or the benchmark fails. - Wide unused-output fixture: 25 runs with independent incompressible 192 KiB native output and 32 KiB unused receipt request fields. DataRow bytes: 5,744,950 to 6,925 (99.879% less). - Required-result control: one complete 2 MiB receipt result. DataRow bytes: 2,097,556 to 2,097,401 (0.0074% less). The whole result is checked for equality. - Exact byte boundary: PostgreSQL text-format DataRow frames, including field length prefixes. Excludes RowDescription, command metadata, TLS, TCP, and other service queries. - Seven measured query round trips per variant follow two warmups and alternate order. Final-head wide-output median: 32.86 to 4.41 ms (ranges 26.24–35.54 and 3.62–5.21 ms). Required-result median: 17.01 to 20.60 ms (ranges 16.30–32.89 and 18.85–22.48 ms). The required-result control shows projection overhead and overlapping timing ranges; it has no latency improvement claim. These are local measurements, not deployed latency results. - `PAPERCLIP_EXECUTION_CONTINUATION_TEST_DATABASE_URL= pnpm exec vitest run server/src/services/execution-continuation.test.ts server/src/services/execution-continuation-history-read.test.ts --maxWorkers=1`: 55 tests pass in PostgreSQL 17. - Cases cover numeric receipt key order, empty operation IDs, malformed receipt entries, missing versus null results, complete large results, malformed root JSON, all closed workspace restore failure codes, invalid recovery schemas, cancelled admission rows, and company/task/agent/source-run boundaries. - `pnpm -r typecheck` and `pnpm build` pass on final head `d665d9f5c47b237c81b9cc80fdec1d39ad90754f`. The final-head focused PostgreSQL run passes all 55 tests. The final benchmark records that SHA, an empty source diff, and passing equivalence/byte assertions. The first typecheck found optional array accesses in new test assertions; explicit required-array assertions fixed them. Local full `pnpm test:run` is not claimed because the host embedded PostgreSQL runtime is unavailable; the full test matrix runs in CI. Exact-head Greptile review completed success at 5/5 with no actionable findings or unresolved threads. The full final-head CI matrix passes. All 58 current-head checks are terminal: 54 success, two neutral, and two optional Storybook skips. One original server shard ended with a canceled test step and no assertion failure; its single targeted retry passed. No product change was needed for that interruption. ## Risks - Future continuation consumers must add any newly required result field to this projection. Existing shared policy semantics stay in the helper. - Completed receipt results can still be large and can dominate transfer. This PR preserves them without a cap. The receipt-heavy control adds JSON projection work for only 155 bytes of savings, so it can be slower despite equal output. - JSON malformed-value and property-presence rules affect action evidence and recovery. Real PostgreSQL comparisons check those decisions and complete output. - Production transfer and latency improvements need a fresh deployed capture. Nothing was merged or deployed by this work. ## Model Used OpenAI GPT-6 through Codex. The exact runtime model ID and context window are not exposed by this session. The work used repository tools, code execution, and real PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/DEVELOPING.md | 11 ++ scripts/benchmark-continuation-history.ts | 120 ++++++++++++++++ ...xecution-continuation-history-read.test.ts | 128 ++++++++++++++++++ .../services/execution-continuation.test.ts | 12 +- server/src/services/execution-continuation.ts | 24 +++- 5 files changed, 291 insertions(+), 4 deletions(-) create mode 100644 scripts/benchmark-continuation-history.ts create mode 100644 server/src/services/execution-continuation-history-read.test.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 4853284df55..6f7b2b181cc 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1888,6 +1888,17 @@ TLS and protocol metadata. To use an isolated external database, set `PAPERCLIP_CONTINUATION_BENCHMARK_DATABASE_URL` to a disposable database ending in `_test`. The harness applies migrations there and adds synthetic fixtures. +Continuation history reads select completed tool receipts and the typed +conversation/workspace recovery fields used to choose the prior terminal run. +Every completed receipt keeps its full result. Receipt order, company/agent/task +scope, and the complete retained history stay unchanged. Saved results remain +available to detail, recovery, and execution paths. +Use `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-continuation-history.ts --baseline-ref=` +to compare generated SQL, complete continuation envelopes, and DataRow bytes. +The benchmark includes a large required-receipt control. Its external database +option is `PAPERCLIP_CONTINUATION_HISTORY_BENCHMARK_DATABASE_URL`, with the same +disposable `_test` database restriction. + Retry scheduling is in `server/src/services/heartbeat/retries.ts`. It owns bounded retry schedules, connection and workspace contention deferrals, shared-workspace holder checks, due retry promotion, and retry-now requests. `createHeartbeatRetries` diff --git a/scripts/benchmark-continuation-history.ts b/scripts/benchmark-continuation-history.ts new file mode 100644 index 00000000000..c26a118c589 --- /dev/null +++ b/scripts/benchmark-continuation-history.ts @@ -0,0 +1,120 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { randomBytes, randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { agents, applyPendingMigrations, companies, createDb, heartbeatRuns, issues, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js"; +import { buildExecutionContinuation } from "../server/src/services/execution-continuation.js"; +import { hasConversationContinuationPolicy } from "../server/src/services/conversation-continuation.js"; +import { resolveExternalTestDatabaseUrl } from "../server/src/__tests__/helpers/external-test-database.js"; + +const repoRoot = fileURLToPath(new URL("../", import.meta.url)); +const git = (...args: string[]) => execFileSync("git", args, { cwd: repoRoot, encoding: "utf8" }); +const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15); +assert.ok(baselineRef, "Pass a pinned pre-fix --baseline-ref"); +const baselineRevision = git("rev-parse", "--verify", `${baselineRef}^{commit}`).trim(); +const baselineOnly = process.argv.includes("--baseline-only"); +const keepAlive = setInterval(() => {}, 1000); +let database: Awaited> | undefined; +let db: Db | undefined; +let baselineDir: string | undefined; +try { + const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_CONTINUATION_HISTORY_BENCHMARK_DATABASE_URL"); + if (externalUrl) { + await applyPendingMigrations(externalUrl); + database = { connectionString: externalUrl, cleanup: async () => {} }; + } else database = await startEmbeddedPostgresTestDatabase("continuation-history-benchmark-"); + db = createDb(database.connectionString); + const original = new URL("../server/src/services/execution-continuation.ts", import.meta.url); + baselineDir = await fs.mkdtemp(path.join(repoRoot, "server/scripts/.history-baseline-")); + const source = git("show", `${baselineRevision}:server/src/services/execution-continuation.ts`); + const baselinePath = path.join(baselineDir, "continuation.ts"); + await fs.writeFile(baselinePath, source.replace(/(from |import\()"(\.{1,2}\/[^\"]+)"/g, + (_match, prefix: string, specifier: string) => `${prefix}${JSON.stringify(fileURLToPath(new URL(specifier, original)))}`)); + const baseline = await import(pathToFileURL(baselinePath).href) as { buildExecutionContinuation: typeof buildExecutionContinuation }; + const object = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + const receipts = (value: unknown) => Object.entries(object(object(value).apiToolReceipts)).flatMap(([id, entry]) => { + const receipt = object(entry); + return receipt.state === "completed" && typeof receipt.operationId === "string" ? [{ id, operationId: receipt.operationId, result: receipt.result }] : []; + }); + const results = []; + for (const receiptHeavy of [false, true]) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Synthetic history", issuePrefix: `B${companyId.replaceAll("-", "")}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Fixture", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Continue", status: "in_progress", assigneeAgentId: agentId }); + const runCount = receiptHeavy ? 1 : 25; + const requiredResult = receiptHeavy ? randomBytes(1536 * 1024).toString("base64") : { receipt: "required output" }; + for (let index = 0; index < runCount; index++) await db.insert(heartbeatRuns).values({ companyId, agentId, + contextSnapshot: { issueId }, status: "failed", errorCode: "adapter_failed", runtimeMode: "legacy", + createdAt: new Date(Date.UTC(2020, 0, 1, 0, 0, index)), + resultJson: { nativeResult: { output: receiptHeavy ? "small" : randomBytes(144 * 1024).toString("base64") }, + conversationContinuation: "continue_conversation_v1", + apiToolReceipts: { saved: { state: "completed", operationId: "save_document", result: requiredResult, + request: receiptHeavy ? "small" : randomBytes(24 * 1024).toString("base64") }, pending: { state: "started", operationId: "unfinished", result: "not a completed action" } } }, + }); + await db.$client.unsafe("ANALYZE"); + async function capture(invoke: typeof buildExecutionContinuation) { + let query: { sql: string; params: unknown[] } | undefined; + let history: Array> | undefined; + const recorded = new Proxy(db!, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = from(...args), execute = selection.execute.bind(selection); + if (args[0] === heartbeatRuns) selection.execute = async (...executeArgs: Parameters) => { + const statement = selection.toSQL(), rows = await execute(...executeArgs); + if (statement.sql.includes('order by "heartbeat_runs"."created_at" asc')) { query = statement; history = rows; } + return rows; + }; + return selection; + }) as typeof builder.from; + return builder; + }; + }}); + const envelope = await invoke({ db: recorded, companyId, issueId, agentId, context: {}, summary: null, exposeLowTrustRaw: false }); + assert.ok(query); assert.ok(history); + return { query, history, envelope }; + } + const before = await capture(baseline.buildExecutionContinuation); + const after = await capture(baselineOnly ? baseline.buildExecutionContinuation : buildExecutionContinuation); + assert.deepEqual(after.envelope, before.envelope); + assert.equal(after.history.length, before.history.length); + for (let index = 0; index < before.history.length; index++) { + const oldRow = before.history[index], newRow = after.history[index]; + for (const [key, value] of Object.entries(oldRow)) if (key !== "result") assert.deepEqual(newRow[key], value); + assert.deepEqual(receipts(newRow.result), receipts(oldRow.result)); + assert.equal(hasConversationContinuationPolicy(newRow.result as Record), hasConversationContinuationPolicy(oldRow.result as Record)); + } + const queries = [before.query, after.query], bytes: number[] = [], samples: number[][] = [[], []]; + for (let iteration = 0; iteration < 9; iteration++) for (const variant of iteration % 2 ? [1, 0] : [0, 1]) { + const query = queries[variant], start = performance.now(); + const rows = await db.$client.unsafe(query.sql, query.params as never[]).raw(); + if (iteration >= 2) samples[variant].push(performance.now() - start); + bytes[variant] = rows.reduce((sum, row) => sum + 7 + row.reduce((n: number, value: Buffer | null) => n + 4 + (value?.length ?? 0), 0), 0); + } + if (!baselineOnly && !receiptHeavy) assert.ok(bytes[1] < bytes[0] / 10, "Wide unused-result fixture must remove more than 90% of DataRow bytes"); + if (receiptHeavy) { + assert.ok(after.envelope.completedActions); + assert.deepEqual(after.envelope.completedActions[0].result, requiredResult); + } + const plans = []; + for (const query of queries) { + const [plan] = await db.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]); + plans.push(plan["QUERY PLAN"]); + } + const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]; + results.push({ fixture: receiptHeavy ? "required-receipt-result" : "wide-unused-output", runCount, rows: after.history.length, + equalEnvelope: true, equalRetainedColumnsAndReceipts: true, baselineDataRowBytes: bytes[0], treatmentDataRowBytes: bytes[1], + reduction: 1 - bytes[1] / bytes[0], baselineMedianMs: median(samples[0]), treatmentMedianMs: median(samples[1]), samples, queries, plans }); + } + console.log(JSON.stringify({ baselineRevision, treatmentRevision: git("rev-parse", "HEAD").trim(), + treatmentDiff: git("diff", "HEAD", "--", "server/src/services/execution-continuation.ts"), baselineOnly, + byteBoundary: "Exact text-format PostgreSQL DataRow frames, excluding metadata, TLS and TCP", postgres: await db.$client.unsafe("select version()"), results }, null, 2)); +} finally { + try { await db?.$client.end({ timeout: 0 }); await database?.cleanup(); + if (baselineDir) await fs.rm(baselineDir, { recursive: true, force: true }); + } finally { clearInterval(keepAlive); } +} diff --git a/server/src/services/execution-continuation-history-read.test.ts b/server/src/services/execution-continuation-history-read.test.ts new file mode 100644 index 00000000000..78b288849dc --- /dev/null +++ b/server/src/services/execution-continuation-history-read.test.ts @@ -0,0 +1,128 @@ +import assert from "node:assert/strict"; +import { randomBytes, randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { agents, applyPendingMigrations, companies, createDb, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; +import { buildExecutionContinuation } from "./execution-continuation.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EXECUTION_CONTINUATION_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("continuation history PostgreSQL projection", () => { + let database: Awaited>; + let db: Db; + beforeAll(async () => { + if (externalUrl) { + await applyPendingMigrations(externalUrl); + database = { connectionString: externalUrl, cleanup: async () => {} }; + } else database = await startEmbeddedPostgresTestDatabase("continuation-history-read-"); + db = createDb(database.connectionString); + }, 60_000); + afterAll(async () => { await db?.$client.end(); await database?.cleanup(); }); + + async function fixture(result: unknown) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "History", issuePrefix: `H${companyId.replaceAll("-", "")}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Executor", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Continue", status: "in_progress", assigneeAgentId: agentId }); + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", errorCode: "adapter_failed", + contextSnapshot: { issueId }, createdAt: new Date("2020-01-01T00:00:00Z"), resultJson: result as Record | null }).returning(); + return { companyId, agentId, issueId, run }; + } + + async function invoke(f: Awaited>, fullResult: boolean, context: Record = {}) { + const reads: Array>> = []; + const recorded = new Proxy(db, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const historyRead = Object.hasOwn(fields[0] ?? {}, "retryOfRunId"); + const builder = historyRead && fullResult + ? target.select({ ...fields[0], result: heartbeatRuns.resultJson }) : target.select(...fields); + const from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const query = from(...args), execute = query.execute.bind(query); + if (historyRead) query.execute = async (...executeArgs: Parameters) => { + const rows = await execute(...executeArgs); reads.push(rows); return rows; + }; + return query; + }) as typeof builder.from; + return builder; + }; + }}); + const result = await buildExecutionContinuation({ db: recorded, companyId: f.companyId, agentId: f.agentId, + issueId: f.issueId, context, summary: null, exposeLowTrustRaw: false }); + return { result, reads }; + } + + async function compare(f: Awaited>, context: Record = {}) { + const before = await invoke(f, true, context), after = await invoke(f, false, context); + expect(after.result).toEqual(before.result); + expect(after.reads).toHaveLength(1); + expect(after.reads[0].map(({ result: _result, ...fields }) => fields)).toEqual(before.reads[0].map(({ result: _result, ...fields }) => fields)); + return { before, after }; + } + + it("keeps every completed receipt, its full result, missing values and receipt order", async () => { + const payload = randomBytes(128 * 1024).toString("base64"); + const result = { nativeResult: { output: payload }, conversationContinuation: "continue_conversation_v1", apiToolReceipts: { + "10": { state: "completed", operationId: "ten", result: null, unusedRequest: payload }, + "2": { state: "completed", operationId: "two", result: { nested: [payload, false, null] } }, + missing: { state: "completed", operationId: "" }, + pending: { state: "started", operationId: "pending", result: payload }, + invalid: { state: "completed", operationId: 42, result: payload }, + array: ["completed", "operation"], + scalar: "completed", number: 42, null: null, + } }; + const f = await fixture(result), { before, after } = await compare(f); + assert.ok(after.result.completedActions); + expect(after.result.completedActions.map(receipt => receipt.receiptId)).toEqual(["2", "10", "missing"]); + expect(after.result.completedActions[0].result).toEqual(result.apiToolReceipts["2"].result); + expect(after.result.completedActions[1].result).toBeNull(); + expect(after.result.completedActions[2].result).toBeUndefined(); + expect(Buffer.byteLength(JSON.stringify(after.reads))).toBeLessThan(Buffer.byteLength(JSON.stringify(before.reads)) / 2); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.run.id)))[0].resultJson).toEqual(result); + }); + + const recovery = { schema: "paperclip.workspace-restore-recovery.v1" }; + it.each([ + { value: null, interrupted: false }, { value: [], interrupted: false }, { value: "scalar", interrupted: false }, { value: 42, interrupted: false }, + { value: {}, interrupted: false }, { value: { conversationContinuation: "continue_conversation_v1", apiToolReceipts: [1, 2] }, interrupted: true }, + { value: { conversationContinuation: 42, apiToolReceipts: "malformed" }, interrupted: false }, + ...["restore_permission_denied", "restore_lock_timeout", "restore_unsafe_archive", "restore_failed"].map(workspaceRestoreFailure => ({ + value: { conversationContinuation: "continue_conversation_v1", workspaceRestoreFailure, workspaceRestoreRecovery: recovery }, interrupted: false, + })), + { value: { conversationContinuation: "continue_conversation_v1", workspaceRestoreFailure: "restore_unsafe_archive" }, interrupted: false }, + { value: { conversationContinuation: "continue_conversation_v1", workspaceRestoreFailure: "custom", workspaceRestoreRecovery: recovery }, interrupted: true }, + ...[null, [], "schema", { schema: 42 }, { schema: "other" }].map(workspaceRestoreRecovery => ({ + value: { conversationContinuation: "continue_conversation_v1", workspaceRestoreFailure: "restore_failed", workspaceRestoreRecovery }, interrupted: true, + })), + ])("keeps continuation/recovery decisions for typed legacy JSON $value", async ({ value, interrupted }) => { + const f = await fixture(value), { after } = await compare(f); + expect(after.result.interruptedRunId).toBe(interrupted ? f.run.id : undefined); + expect(after.result.completedActions).toEqual([]); + }); + + it("keeps full ordered history and source-agent scope without cross-company or cross-task receipts", async () => { + const completed = (operationId: string) => ({ apiToolReceipts: { saved: { state: "completed", operationId, result: { operationId } } } }); + const f = await fixture({ ...completed("first"), conversationContinuation: "continue_conversation_v1" }); + const otherAgentId = randomUUID(); + await db.insert(agents).values({ id: otherAgentId, companyId: f.companyId, name: "Previous owner", adapterType: "paperclip_runner" }); + const [source] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: otherAgentId, status: "succeeded", + contextSnapshot: { issueId: f.issueId }, resultJson: completed("other-agent"), createdAt: new Date("2020-01-01T00:01:00Z") }).returning(); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "cancelled", errorCode: "execution_reconciliation_required", + contextSnapshot: { issueId: f.issueId }, resultJson: completed("cancelled"), createdAt: new Date("2020-01-01T00:02:00Z") }); + await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId, status: "failed", + contextSnapshot: { issueId: randomUUID() }, resultJson: completed("foreign-task") }); + const foreign = await fixture(completed("foreign-company")); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId } }).where(eq(heartbeatRuns.id, foreign.run.id)); + const ordinary = await compare(f); + assert.ok(ordinary.after.result.completedActions); + expect(ordinary.after.result.completedActions.map(receipt => receipt.operationId)).toEqual(["first", "cancelled"]); + expect(ordinary.after.result.interruptedRunId).toBe(f.run.id); + const resumed = await compare(f, { previousRunId: source.id }); + assert.ok(resumed.after.result.completedActions); + expect(resumed.after.result.completedActions.map(receipt => receipt.operationId)).toEqual(["first", "other-agent", "cancelled"]); + expect(resumed.after.result.interruptedRunId).toBeUndefined(); + }); +}); diff --git a/server/src/services/execution-continuation.test.ts b/server/src/services/execution-continuation.test.ts index 834bf515ee7..fc9e88795ad 100644 --- a/server/src/services/execution-continuation.test.ts +++ b/server/src/services/execution-continuation.test.ts @@ -3,6 +3,7 @@ import { eq } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { agents, + applyPendingMigrations, companies, createDb, heartbeatRuns, @@ -15,6 +16,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase, } from "../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; import { StaleExecutionContinuationError, buildExecutionContinuation, currentContinuationOrigins, projectHumanInteractionResponse } from "./execution-continuation.js"; const expectStaleContinuation = async ( @@ -29,7 +31,8 @@ const expectMissingContinuationContext = async (run: () => Promise) => await expect(result).rejects.toThrow("continuation_source_context_missing"); await expect(result).rejects.not.toBeInstanceOf(StaleExecutionContinuationError); }; -const support = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EXECUTION_CONTINUATION_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); (support.supported ? describe : describe.skip)( "authorized continuation context", () => { @@ -44,13 +47,16 @@ const support = await getEmbeddedPostgresTestSupport(); laterId = randomUUID(), interactionId = randomUUID(); beforeAll(async () => { - database = await startEmbeddedPostgresTestDatabase( + if (externalUrl) { + await applyPendingMigrations(externalUrl); + database = { connectionString: externalUrl, cleanup: async () => {} }; + } else database = await startEmbeddedPostgresTestDatabase( "paperclip-continuation-context-", ); db = createDb(database.connectionString); await db .insert(companies) - .values({ id: companyId, name: "Continuation", issuePrefix: "CTX" }); + .values({ id: companyId, name: "Continuation", issuePrefix: `CTX${companyId.replaceAll("-", "")}` }); await db .insert(agents) .values({ diff --git a/server/src/services/execution-continuation.ts b/server/src/services/execution-continuation.ts index c44ce6ee325..378e7f1382c 100644 --- a/server/src/services/execution-continuation.ts +++ b/server/src/services/execution-continuation.ts @@ -31,6 +31,28 @@ const object = (v: unknown): Record => : {}; const string = (v: unknown) => typeof v === "string" && v.length > 0 ? v : null; +const historyResult = sql | null>`case + when jsonb_typeof(${heartbeatRuns.resultJson}) = 'object' then ( + select coalesce(jsonb_object_agg(field.key, case + when field.key = 'apiToolReceipts' and jsonb_typeof(field.value) = 'object' then ( + select coalesce(jsonb_object_agg(receipt.key, ( + select jsonb_object_agg(detail.key, detail.value) + from jsonb_each(receipt.value) as detail(key, value) + where detail.key in ('state', 'operationId', 'result') + )), '{}'::jsonb) + from jsonb_each(field.value) as receipt(key, value) + where receipt.value->'state' = '"completed"'::jsonb + and jsonb_typeof(receipt.value->'operationId') = 'string' + ) + when field.key = 'workspaceRestoreRecovery' and jsonb_typeof(field.value) = 'object' then ( + select coalesce(jsonb_object_agg(detail.key, detail.value), '{}'::jsonb) + from jsonb_each(field.value) as detail(key, value) + where detail.key = 'schema' + ) + else field.value end), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.resultJson}) as field(key, value) + where field.key in ('apiToolReceipts', 'conversationContinuation', 'workspaceRestoreFailure', 'workspaceRestoreRecovery') + ) else ${heartbeatRuns.resultJson} end`; export function continuationOriginCommentIds(context: unknown): string[] { const c = object(context); const prior = object(c.executionContinuation); @@ -335,7 +357,7 @@ export async function buildExecutionContinuation(input: { ? { kind: "description" as const, id: issue.id, revision: hashObjectiveSource(issue.description) } : { kind: "title" as const, id: issue.id, revision: hashObjectiveSource(issue.title) }; const priorRuns = await db - .select({ id: heartbeatRuns.id, result: heartbeatRuns.resultJson, status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode, runtimeMode: heartbeatRuns.runtimeMode, retryOfRunId: heartbeatRuns.retryOfRunId }) + .select({ id: heartbeatRuns.id, result: historyResult, status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode, runtimeMode: heartbeatRuns.runtimeMode, retryOfRunId: heartbeatRuns.retryOfRunId }) .from(heartbeatRuns) .where( and( From c3a9a8e92af0ad4008481f8eed57df434acccdaa Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:31:21 -0500 Subject: [PATCH 11/41] perf(server): scope run identity reads to operation needs (#17931) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool and connection operations capture the current accepted requester. > - Capture locks the task and run, then reconciles pending steering. > - Most callers use identity or task fields from the returned run. > - The default query also transfers saved prompts, runner configuration, results, and logs. > - This pull request gives each production caller an explicit read scope. > - Narrow scopes keep their authority checks and return only selected columns. > - The Muse project path still reads its complete saved execution input. ## Linked Issues or Issue Description **What existing behavior does this improve?** Operation-time run identity capture for native tools, gateway sessions, project and workspace access, Slack, voice, connection management, and connection token admission. **Current behavior** These callers use the default full run read. Large saved execution documents cross the database connection even when the caller only needs the current accepted requester or task binding. **Proposed behavior** Use typed identity, task, broker, and project-execution scopes. Keep task-before-run locks, company and agent predicates, active status validation, and pending-steering reconciliation. Return the complete saved native input only where project admission uses it. **Reason and benefit** A synthetic PostgreSQL 17.11 fixture matches a wide run with a 142 KB context and a 77 KB runner profile. Its locked run DataRow falls from 225,101 bytes to 190–605 bytes for authority reads. The project scope returns 66,004 bytes because it must retain a complete 64 KB execution input. These are local DataRow measurements, not a forecast of production billing or latency. **Breaking changes** None to public APIs, saved data, or authority policy. Internal narrow return types omit columns that were not selected. The default full compatibility read and the existing GitHub-credential and authentication scopes keep their current contracts. **Additional context** The public issue and PR duplicate searches found no matching open change. This is one independent query optimization. It does not require other performance PRs. ## What Changed - Add typed scopes for accepted identity metadata, task authorization, token attribution, and saved project execution input. - Assign explicit scopes at all production `captureRunIdentity` call sites. - Keep the full native execution input for Muse project admission while omitting unrelated profile, context, result, and log data. - Narrow the broker's initial read and the existing Slack and connection-management historical reads to the same required fields. - Extend steering and acknowledgement regressions to all new scopes. Add real PostgreSQL projection, malformed JSON, tenant and agent, session-generation, and saved-input coverage. - Add a service-generated SQL benchmark with exact raw DataRow accounting, retained-field equality, plans, and alternating transaction timings. - Document the scope contracts and reproducible benchmark. ## Verification - Baseline: `623f1dc289a14a699ebbe3f09b94179d38fc2ae2`. - `vitest run server/src/__tests__/run-identity-projections.test.ts server/src/__tests__/run-identity.test.ts`: 54 tests passed on an owned disposable PostgreSQL 17.11 database. - The tests retain task-before-run and audit foreign-key lock behavior. Every scope rejects uncertain steering, accepts its durable acknowledgement, and preserves requester replay ordering. - A full-row mutation makes the narrow identity regression fail. The restored treatment passes. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-run-identity-projections.ts --baseline-ref=623f1dc289a14a699ebbe3f09b94179d38fc2ae2` captures the real locked SQL for both versions. - Every retained column and selected JSON field matches the baseline. Identity contexts match. The complete stored run remains unchanged. Capture keeps the same five queries. | Scope | Before DataRow bytes | After DataRow bytes | Reduction | | --- | ---: | ---: | ---: | | Identity | 225,101 | 190 | 99.92% | | Task authorization | 225,101 | 410 | 99.82% | | Token broker | 225,101 | 605 | 99.73% | | Project execution input | 225,101 | 66,004 | 70.68% | - DataRow bytes include row framing and raw text-format column values. They exclude response metadata, TLS, and TCP. - Two warmups and seven alternating samples measure the complete capture transaction. Several timing ranges overlap. The primary claim is byte reduction; production latency is not verified. - Initial fixture runs exposed an empty external schema and an incomplete synthetic conversation binding. The test setup now applies migrations and supplies the required user/state fields. The final 54 tests passed. - `pnpm -r typecheck` and `pnpm build` passed locally. - Four existing skill-setup tests passed. Four assembled connection-management/native-tool consumer cases passed with real JWT/API-key authentication. These include per-request identity freshness and task-scoped activity authority. - A CI-only stale mock assertion expected two capture arguments; it now verifies the explicit identity scope. A repository-wide capture mock audit found no other stale argument assertions. - Review follow-up: the projection suite uses the existing embedded-support guard while always enabling an explicitly supplied external database. The audit-lock regression merges URL query parameters and retains existing PostgreSQL options. - `vitest run server/src/__tests__/run-identity-projections.test.ts server/src/__tests__/run-identity.test.ts server/src/__tests__/company-skills-routes.test.ts server/src/__tests__/project-skill-setup-identity.test.ts`: 123 tests passed. The external PostgreSQL URL included both `application_name` and `options`; lock tests verified `lock_timeout=1s` and retained `statement_timeout=5s`. - Full local tests are not run because the host's embedded PostgreSQL setup is unavailable. Greptile reviewed `06e1dd53a3fb24df74d5fbfb2ad397f291f1e30f` at 5/5 with zero unresolved threads. The full final-head CI matrix passes. All 58 checks are terminal: 55 success, one neutral, and two optional Storybook skips. - No migration, lockfile, workflow, new polling, or cache change. ## Risks A narrow read must retain each caller's actual fields. The caller audit includes Slack's wake-request fallback, conversation generation, legacy broker attribution, and the Muse parser's complete native input. JSON selection retains value types, including malformed legacy values. The saved-input scope preserves missing-key behavior. The default full read still returns the complete stored row for compatibility. Locks and live authority checks are unchanged. ## Model Used OpenAI GPT-6, accessed through Codex. The exact runtime model ID and context-window size were not exposed in this session. Used repository inspection, code execution, and PostgreSQL validation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/run-identity-reads.md | 35 +++++ scripts/benchmark-run-identity-projections.ts | 120 ++++++++++++++++++ .../__tests__/company-skills-routes.test.ts | 2 +- .../run-identity-projection-fixture.ts | 41 ++++++ .../project-skill-setup-identity.test.ts | 2 +- .../run-identity-projections.test.ts | 116 +++++++++++++++++ server/src/__tests__/run-identity.test.ts | 37 ++++-- server/src/routes/projects.ts | 2 +- server/src/services/connection-intents.ts | 2 +- server/src/services/connection-management.ts | 8 +- .../services/connectors/slack-authority.ts | 6 +- .../execution-workspace-repositories.ts | 2 +- .../paperclip-runner-tool-authority.ts | 6 +- .../services/project-skill-source-bindings.ts | 2 +- server/src/services/project-tool-context.ts | 11 +- server/src/services/run-identity.ts | 65 +++++++++- .../tool-access/token-broker/admission.ts | 6 +- server/src/services/tool-gateway.ts | 2 +- .../src/services/voice/speko-agent-tools.ts | 2 +- 19 files changed, 427 insertions(+), 40 deletions(-) create mode 100644 doc/run-identity-reads.md create mode 100644 scripts/benchmark-run-identity-projections.ts create mode 100644 server/src/__tests__/helpers/run-identity-projection-fixture.ts create mode 100644 server/src/__tests__/run-identity-projections.test.ts diff --git a/doc/run-identity-reads.md b/doc/run-identity-reads.md new file mode 100644 index 00000000000..a41b252b411 --- /dev/null +++ b/doc/run-identity-reads.md @@ -0,0 +1,35 @@ +# Run identity reads + +`captureRunIdentity` locks the task before the run. It validates company, agent, +and active status. Pending steering still requires its durable acknowledgement. +The accepted requester is read for each operation. No scope adds a cache or skips +these checks. + +Callers choose a typed read scope: + +| Scope | Retained run data | Consumers | +| --- | --- | --- | +| `identity` | Run and accepted requester identifiers, status | Gateway, native connection tools, connection intents, project skill setup | +| `task_authorization` | Identity, task bindings, wake request, conversation generation, skill-test marker | Projects, workspace repositories, Slack, voice, connection management | +| `token_broker` | Identity, invocation source, task/project/routine and requester attribution | Connection token broker | +| `project_execution` | Task authorization plus runtime mode and the complete saved native execution input | Muse project admission | + +Narrow return types omit unselected columns. JSON projections preserve the types +of retained values. The default full read remains available for compatibility and +explicit inspection. Existing GitHub-credential and request-authentication scopes +keep their current contracts. Execution preparation and recovery retain their +separate complete-input reads. + +The synthetic real-PostgreSQL benchmark captures the actual locked query and +compares every retained column, context and identity result. It keeps the same +five-query reconciliation transaction, checks the unchanged stored run, records +plans and alternating timings, and measures raw PostgreSQL DataRow bytes: + +```sh +pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-run-identity-projections.ts --baseline-ref=623f1dc289a14a699ebbe3f09b94179d38fc2ae2 +``` + +The default uses disposable embedded PostgreSQL. `PAPERCLIP_TEST_DATABASE_URL` +can select a disposable local database whose name ends in `_test`. DataRow bytes +exclude response metadata, TLS and TCP. Byte reduction does not establish a +production latency or billing change. diff --git a/scripts/benchmark-run-identity-projections.ts b/scripts/benchmark-run-identity-projections.ts new file mode 100644 index 00000000000..4ddc589ef3b --- /dev/null +++ b/scripts/benchmark-run-identity-projections.ts @@ -0,0 +1,120 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { applyPendingMigrations, createDb, startEmbeddedPostgresTestDatabase } from "../packages/db/src/index.js"; +import { captureRunIdentity } from "../server/src/services/run-identity.js"; +import { captureIdentityQueries, seedIdentityProjection, type IdentityQuery } from "../server/src/__tests__/helpers/run-identity-projection-fixture.js"; + +const root = fileURLToPath(new URL("../", import.meta.url)); +const git = (args: string[]) => execFileSync("git", args, { cwd: root, encoding: "utf8" }); +const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15); +assert.ok(baselineRef, "Pass an explicit pre-fix --baseline-ref="); +const baselineRevision = git(["rev-parse", "--verify", `${baselineRef}^{commit}`]).trim(); +const modulePath = "server/src/services/run-identity.ts"; +const unchangedHelpers = ["run-task-binding.ts", "task-retry.ts", "issue-queued-comment-queue.ts"]; +for (const name of unchangedHelpers) { + const helper = `server/src/services/${name}`; + assert.equal(git(["show", `${baselineRevision}:${helper}`]), await fs.readFile(path.join(root, helper), "utf8"), + `Baseline helper ${name} changed; isolate it before benchmarking`); +} +const keepAlive = setInterval(() => {}, 1_000); +let temporary: Awaited> | undefined; +let db: ReturnType | undefined; +let baselineDir: string | undefined; +try { + const oldSource = git(["show", `${baselineRevision}:${modulePath}`]); + const moduleUrl = new URL(`../${modulePath}`, import.meta.url); + baselineDir = await fs.mkdtemp(path.join(root, "server/scripts/.identity-projection-baseline-")); + const baselineFile = path.join(baselineDir, "run-identity.ts"); + await fs.writeFile(baselineFile, oldSource.replace(/from "(\.{1,2}\/[^\"]+)"/g, + (_match, specifier: string) => `from ${JSON.stringify(fileURLToPath(new URL(specifier, moduleUrl)))}`)); + const baseline = await import(pathToFileURL(baselineFile).href) as typeof import("../server/src/services/run-identity.js"); + const externalUrl = process.env.PAPERCLIP_TEST_DATABASE_URL; + if (externalUrl) { + const parsed = new URL(externalUrl); + assert.ok(["127.0.0.1", "localhost"].includes(parsed.hostname) && /_test$/.test(parsed.pathname), + "External benchmark database must be a disposable local *_test database"); + await applyPendingMigrations(externalUrl); + db = createDb(externalUrl); + } else { + temporary = await startEmbeddedPostgresTestDatabase("identity-projection-benchmark-"); + db = createDb(temporary.connectionString); + } + const f = await seedIdentityProjection(db); + await db.$client.unsafe("ANALYZE"); + const beforeQueries: IdentityQuery[] = []; + const before = await baseline.captureRunIdentity(captureIdentityQueries(db, beforeQueries), f); + const findRunQuery = (queries: IdentityQuery[]) => { + const query = queries.find(query => query.sql.includes('from "heartbeat_runs"') && query.sql.includes("for no key update")); + assert.ok(query, "Capture the actual locked identity query"); + return query; + }; + const beforeQuery = findRunQuery(beforeQueries); + const dataRowBytes = async (query: IdentityQuery) => { + const rows = await db!.$client.unsafe(query.sql, query.params as never[]).raw(); + return rows.reduce((total, row) => total + 7 + 4 * row.length + row.reduce((bytes, value) => bytes + (value?.byteLength ?? 0), 0), 0); + }; + const baselineBytes = await dataRowBytes(beforeQuery); + async function plan(query: IdentityQuery) { + const [result] = await db!.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]); + return result!["QUERY PLAN"]; + } + const results = []; + for (const readScope of ["identity", "task_authorization", "token_broker", "project_execution"] as const) { + const afterQueries: IdentityQuery[] = []; + const after = await captureRunIdentity(captureIdentityQueries(db, afterQueries), f, { readScope }); + assert.deepEqual(after.context, before.context); + for (const [key, value] of Object.entries(after.run)) { + if (key === "runnerProfileJson") assert.deepEqual(value, { nativeExecutionInput: before.run.runnerProfileJson?.nativeExecutionInput }); + else if (key !== "contextSnapshot") assert.deepEqual(value, before.run[key as keyof typeof before.run], `Retained ${key}`); + else { + const snapshot = value as Record; + for (const [name, selected] of Object.entries(snapshot)) { + if (name !== "paperclipIssue") assert.deepEqual(selected, before.run.contextSnapshot?.[name] ?? null, `Retained context ${name}`); + else for (const [nested, item] of Object.entries(selected as Record)) + assert.deepEqual(item, (before.run.contextSnapshot?.paperclipIssue as Record)?.[nested] ?? null, `Retained nested ${nested}`); + } + } + } + const afterQuery = findRunQuery(afterQueries), afterBytes = await dataRowBytes(afterQuery); + assert.ok(afterBytes < baselineBytes * (readScope === "project_execution" ? 0.5 : 0.05), + "Fixture acceptance: remove at least 95% of authority bytes or 50% while preserving complete execution input"); + assert.equal(afterQueries.length, beforeQueries.length, "No queries added or removed from identity reconciliation"); + const samples = { baseline: [] as number[], treatment: [] as number[] }; + for (let n = 0; n < 9; n++) { + for (const version of n % 2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const) { + const start = performance.now(); + if (version === "baseline") await baseline.captureRunIdentity(db, f); + else await captureRunIdentity(db, f, { readScope }); + if (n >= 2) samples[version].push(performance.now() - start); + } + } + const stats = (values: number[]) => { + const sorted = [...values].sort((a, b) => a - b); + return { samplesMs: values, medianMs: sorted[Math.floor(sorted.length / 2)], minMs: sorted[0], maxMs: sorted.at(-1) }; + }; + results.push({ readScope, baselineBytes, treatmentBytes: afterBytes, reductionPercent: 100 * (1 - afterBytes / baselineBytes), + retainedColumnsAndContextEqual: true, identityContextEqual: true, queryCount: afterQueries.length, + generatedSql: { baseline: beforeQuery, treatment: afterQuery }, + timings: { scope: "complete captureRunIdentity service transaction", warmups: 2, samples: 7, order: "alternating", + baseline: stats(samples.baseline), treatment: stats(samples.treatment) }, + plans: { baseline: await plan(beforeQuery), treatment: await plan(afterQuery) } }); + } + assert.deepEqual((await captureRunIdentity(db, f)).run, before.run, "Complete stored run unchanged"); + console.log(JSON.stringify({ baselineRevision, treatmentRevision: git(["rev-parse", "HEAD"]).trim(), + treatmentDiff: git(["diff", "HEAD", "--", "server/src"]), unchangedHelpers, + postgresVersion: (await db.$client.unsafe("SELECT version()"))[0]!.version, + settings: await db.$client.unsafe("SELECT name, setting FROM pg_settings WHERE name IN ('block_size','shared_buffers','work_mem','jit') ORDER BY name"), + fixture: { runs: 1, synthetic: true, contextBytes: Buffer.byteLength(JSON.stringify(f.payloads.contextSnapshot)), + profileBytes: Buffer.byteLength(JSON.stringify(f.payloads.runnerProfileJson)) }, + measurement: "PostgreSQL DataRow messages from raw text-format buffers; excludes metadata, TLS and TCP", results, + }, null, 2)); +} finally { + try { + await db?.$client.end({ timeout: 0 }); + await temporary?.cleanup(); + if (baselineDir) await fs.rm(baselineDir, { recursive: true, force: true }); + } finally { clearInterval(keepAlive); } +} diff --git a/server/src/__tests__/company-skills-routes.test.ts b/server/src/__tests__/company-skills-routes.test.ts index f40bfa0f8ce..11fe51213a7 100644 --- a/server/src/__tests__/company-skills-routes.test.ts +++ b/server/src/__tests__/company-skills-routes.test.ts @@ -1264,7 +1264,7 @@ describe("company skill mutation permissions", () => { const response = await request(createApp(agent)).post("/api/companies/company-1/skills/import") .send({ source: "https://github.com/acme/public-skills" }); expect(response.status, JSON.stringify(response.body)).toBe(201); - expect(mockCaptureRunIdentity).toHaveBeenCalledWith(expect.anything(), { companyId: "company-1", agentId: agent.agentId, runId: "run-1" }); + expect(mockCaptureRunIdentity).toHaveBeenCalledWith(expect.anything(), { companyId: "company-1", agentId: agent.agentId, runId: "run-1" }, { readScope: "identity" }); expect(mockSkillSourceService.importFromUrl).toHaveBeenCalledWith("company-1", expect.anything(), expect.objectContaining({ setup: { userId: "validated-owner", agentId: agent.agentId, localTrusted: false } })); }); diff --git a/server/src/__tests__/helpers/run-identity-projection-fixture.ts b/server/src/__tests__/helpers/run-identity-projection-fixture.ts new file mode 100644 index 00000000000..5072e37060f --- /dev/null +++ b/server/src/__tests__/helpers/run-identity-projection-fixture.ts @@ -0,0 +1,41 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { agents, companies, heartbeatRuns, issueComments, issues, type Db } from "@paperclipai/db"; +import { initializeRunIdentity } from "../../services/run-identity.js"; + +export async function seedIdentityProjection(db: Db) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), messageId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Synthetic identity", issuePrefix: companyId }); + await db.insert(agents).values({ id: agentId, companyId, name: "Identity reader", adapterType: "process" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Identity task", assigneeAgentId: agentId, + conversationAgentId: agentId, conversationUserId: "operator", conversationState: "active", conversationSessionGeneration: 3, workMode: "standard" }); + const payloads = { contextSnapshot: { issueId, conversationSessionGeneration: 3, paperclipSkillTest: false, + projectId: randomUUID(), routineId: randomUUID(), paperclipIssue: { id: issueId, projectId: randomUUID(), + responsibleUserId: "nested-user", document: randomBytes(8_192).toString("base64") }, + savedInput: randomBytes(96 * 1_024).toString("base64") }, + runnerProfileJson: { nativeExecutionInput: { savedInput: randomBytes(48 * 1_024).toString("base64") }, + unrelatedProfileDocument: randomBytes(8_192).toString("base64") }, + resultJson: { retained: randomBytes(4_096).toString("base64") }, stdoutExcerpt: "retained stdout", stderrExcerpt: "retained stderr" }; + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", nativeIssueId: issueId, + runtimeMode: "native", invocationSource: "automation", ...payloads }); + await db.insert(issueComments).values({ id: messageId, companyId, issueId, authorUserId: "next-user", body: "New request" }); + const context = await initializeRunIdentity(db, { companyId, runId, issueId, responsibleUserId: "operator", cause: "instruction" }); + return { companyId, agentId, issueId, runId, messageId, context, payloads }; +} + +export type IdentityQuery = { sql: string; params: unknown[] }; +export function captureIdentityQueries(db: Db, queries: IdentityQuery[]): Db { + return new Proxy(db, { get(target, key) { + if (key === "transaction") return (action: (tx: Db) => Promise) => + target.transaction(tx => action(captureIdentityQueries(tx as unknown as Db, queries))); + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = from(...args), execute = selection.execute.bind(selection); + selection.execute = async () => { queries.push(selection.toSQL()); return execute(); }; + return selection; + }) as typeof builder.from; + return builder; + }; + }}); +} diff --git a/server/src/__tests__/project-skill-setup-identity.test.ts b/server/src/__tests__/project-skill-setup-identity.test.ts index 844c4f7df71..d08cd9ec867 100644 --- a/server/src/__tests__/project-skill-setup-identity.test.ts +++ b/server/src/__tests__/project-skill-setup-identity.test.ts @@ -13,7 +13,7 @@ it('keeps responsibility without searching personal GitHub grants when the run h it('pins only the connection and grant already authorized for the active run', async () => { capture.mockResolvedValue({ run: { responsibleUserId: 'person' }, context: { github: { status: 'available', connectionId: 'selected', grantId: 'grant' } } }); expect(await projectSkillSetupIdentity(db, 'company', actor)).toMatchObject({ refreshUserId: 'person', connectionId: 'selected', refreshGrantId: 'grant' }); - expect(capture).toHaveBeenCalledWith(db, { companyId: 'company', agentId: 'agent', runId: 'run' }); + expect(capture).toHaveBeenCalledWith(db, { companyId: 'company', agentId: 'agent', runId: 'run' }, { readScope: 'identity' }); }); it('does not reuse an unavailable run credential', async () => { capture.mockResolvedValue({ run: { responsibleUserId: 'person' }, context: { github: { status: 'unavailable', connectionId: 'old', grantId: 'old' } } }); diff --git a/server/src/__tests__/run-identity-projections.test.ts b/server/src/__tests__/run-identity-projections.test.ts new file mode 100644 index 00000000000..1e7132029a2 --- /dev/null +++ b/server/src/__tests__/run-identity-projections.test.ts @@ -0,0 +1,116 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { applyPendingMigrations, createDb, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { captureIdentityQueries, seedIdentityProjection, type IdentityQuery } from "./helpers/run-identity-projection-fixture.js"; +import { captureRunIdentity } from "../services/run-identity.js"; +import { projectToolContext } from "../services/project-tool-context.js"; +import { boundRunIssueId } from "../services/run-task-binding.js"; + +const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL; +const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const scopes = ["identity", "task_authorization", "token_broker"] as const; +(support.supported ? describe : describe.skip)("operation identity read projections", () => { + let temporary: Awaited> | undefined; + let db: Db; + beforeAll(async () => { + if (externalDatabaseUrl) { + await applyPendingMigrations(externalDatabaseUrl); + db = createDb(externalDatabaseUrl); + } else { + temporary = await startEmbeddedPostgresTestDatabase("identity-projection-"); + db = createDb(temporary.connectionString); + } + }, 90_000); + afterAll(async () => { await db?.$client.end({ timeout: 0 }); await temporary?.cleanup(); }); + + it.each(scopes)("bounds %s SQL results and preserves the stored execution", async readScope => { + const f = await seedIdentityProjection(db), queries: IdentityQuery[] = []; + const full = await captureRunIdentity(db, f); + const narrow = await captureRunIdentity(captureIdentityQueries(db, queries), f, { readScope }); + expect(narrow.context).toEqual(full.context); + expect(narrow.run).toMatchObject({ id: f.runId, companyId: f.companyId, agentId: f.agentId, + status: "running", responsibleUserId: "operator", activeIdentityContextId: full.run.activeIdentityContextId }); + expect(narrow.run).not.toHaveProperty("runnerProfileJson"); + expect(narrow.run).not.toHaveProperty("resultJson"); + const query = queries.find(query => query.sql.includes('from "heartbeat_runs"') && query.sql.includes("for no key update"))!; + const rows = await db.$client.unsafe(query.sql, query.params as never[]).raw(); + expect(rows).toHaveLength(1); + expect(rows[0]!.reduce((bytes, value) => bytes + (value?.byteLength ?? 0), 0)).toBeLessThan(2_048); + expect((await captureRunIdentity(db, f)).run).toEqual(full.run); + }); + + it("preserves task binding types and legacy non-object contexts", async () => { + const f = await seedIdentityProjection(db); + for (const snapshot of [null, false, 42, "text", [], ["value"], {}, ...[null, false, 42, " 3 ", [], {}, { nested: true }] + .map(value => ({ issueId: f.issueId, conversationSessionGeneration: value, paperclipSkillTest: value }))]) { + await db.$client.unsafe("UPDATE heartbeat_runs SET context_snapshot = $1::jsonb WHERE id = $2", [JSON.stringify(snapshot), f.runId]); + const full = await captureRunIdentity(db, f), narrow = await captureRunIdentity(db, f, { readScope: "task_authorization" }); + expect(boundRunIssueId(narrow.run)).toBe(boundRunIssueId(full.run)); + expect(Number(narrow.run.contextSnapshot?.conversationSessionGeneration ?? 0)).toBe(Number(full.run.contextSnapshot?.conversationSessionGeneration ?? 0)); + expect(Boolean(narrow.run.contextSnapshot?.paperclipSkillTest)).toBe(Boolean(full.run.contextSnapshot?.paperclipSkillTest)); + } + for (const issueId of [true, 123, [], {}, "malformed"]) { + await db.$client.unsafe("UPDATE heartbeat_runs SET context_snapshot = $1::jsonb WHERE id = $2", [JSON.stringify({ issueId }), f.runId]); + await expect(captureRunIdentity(db, f)).rejects.toThrow("task identity is invalid"); + await expect(captureRunIdentity(db, f, { readScope: "task_authorization" })).rejects.toThrow("task identity is invalid"); + } + }); + + it("retains broker fields and nested attribution without document payloads", async () => { + const f = await seedIdentityProjection(db); + for (const value of [null, true, 42, " padded ", [], {}, { retained: true }]) { + const snapshot = { issueId: f.issueId, projectId: value, routineId: value, responsibleUserId: value, + responsible_user_id: value, paperclipIssue: { id: value, projectId: value, responsibleUserId: value, + responsible_user_id: value, savedDocument: f.payloads.contextSnapshot.savedInput } }; + await db.update(heartbeatRuns).set({ contextSnapshot: snapshot }).where(eq(heartbeatRuns.id, f.runId)); + const narrow = await captureRunIdentity(db, f, { readScope: "token_broker" }); + expect(narrow.run.invocationSource).toBe("automation"); + expect(narrow.run.contextSnapshot).toEqual({ ...snapshot, paperclipIssue: { id: value, + projectId: value, responsibleUserId: value, responsible_user_id: value } }); + } + for (const paperclipIssue of [null, true, 42, "legacy", []]) { + await db.update(heartbeatRuns).set({ contextSnapshot: { paperclipIssue } }).where(eq(heartbeatRuns.id, f.runId)); + expect((await captureRunIdentity(db, f, { readScope: "token_broker" })).run.contextSnapshot?.paperclipIssue).toBeNull(); + } + }); + + it.each(scopes.flatMap(readScope => (["company", "agent", "run", "finished"] as const).map(fault => ({ readScope, fault }))))( + "denies $fault mismatches for $readScope", async ({ readScope, fault }) => { + const f = await seedIdentityProjection(db); + if (fault === "finished") await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, f.runId)); + await expect(captureRunIdentity(db, { ...f, ...(fault === "company" ? { companyId: randomUUID() } : + fault === "agent" ? { agentId: randomUUID() } : fault === "run" ? { runId: randomUUID() } : {}) }, { readScope })).rejects.toThrow("active run"); + }, + ); + + it("preserves every saved native input shape without unrelated profile fields", async () => { + const f = await seedIdentityProjection(db); + for (const nativeExecutionInput of [null, true, 42, "legacy", [], {}, { opaque: { retained: true } }]) { + await db.update(heartbeatRuns).set({ runnerProfileJson: { nativeExecutionInput, unrelated: "drop" } }).where(eq(heartbeatRuns.id, f.runId)); + expect((await captureRunIdentity(db, f, { readScope: "project_execution" })).run.runnerProfileJson).toEqual({ nativeExecutionInput }); + } + await db.update(heartbeatRuns).set({ runnerProfileJson: { unrelated: "drop" } }).where(eq(heartbeatRuns.id, f.runId)); + expect((await captureRunIdentity(db, f, { readScope: "project_execution" })).run.runnerProfileJson).toEqual({}); + for (const profile of [null, true, 42, "legacy", []]) { + await db.$client.unsafe("UPDATE heartbeat_runs SET runner_profile_json = $1::jsonb WHERE id = $2", [JSON.stringify(profile), f.runId]); + expect((await captureRunIdentity(db, f, { readScope: "project_execution" })).run.runnerProfileJson).toBeNull(); + } + }); + + it("keeps the complete execution input for the project admission consumer", async () => { + const f = await seedIdentityProjection(db); + const actor = { type: "agent" as const, source: "agent_jwt" as const, companyId: f.companyId, agentId: f.agentId, runId: f.runId }; + const narrow = await projectToolContext(db, actor); + expect(narrow.run).not.toHaveProperty("runnerProfileJson"); + const full = await projectToolContext(db, actor, true, "Project", { includeExecutionInput: true }); + expect(full.run.runnerProfileJson).toEqual({ nativeExecutionInput: f.payloads.runnerProfileJson.nativeExecutionInput }); + expect(full.run.contextSnapshot).toEqual(narrow.run.contextSnapshot); + expect(full.run).not.toHaveProperty("resultJson"); + expect(full.run.contextSnapshot).not.toHaveProperty("savedInput"); + await db.update(issues).set({ conversationSessionGeneration: 4 }).where(eq(issues.id, f.issueId)); + await expect(projectToolContext(db, actor)).rejects.toThrow("session has changed"); + await expect(projectToolContext(db, actor, true, "Project", { includeExecutionInput: true })).rejects.toThrow("session has changed"); + }); +}); diff --git a/server/src/__tests__/run-identity.test.ts b/server/src/__tests__/run-identity.test.ts index 069139a8123..1cc8ed20f7a 100644 --- a/server/src/__tests__/run-identity.test.ts +++ b/server/src/__tests__/run-identity.test.ts @@ -1,19 +1,23 @@ import { randomUUID } from "node:crypto"; import { eq, sql } from "drizzle-orm"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; -import { agentWakeupRequests, agents, companies, createDb, heartbeatRuns, heartbeatRunEvents, issueComments, issueThreadInteractions, issues, secretAccessEvents } from "@paperclipai/db"; +import { agentWakeupRequests, agents, companies, applyPendingMigrations, createDb, closeRegisteredClients, heartbeatRuns, heartbeatRunEvents, issueComments, issueThreadInteractions, issues, secretAccessEvents } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { acceptSteeredIdentity, captureRunIdentity, initializeRunIdentity, listRunIdentityContexts, rejectSteeredIdentity, reserveSteeredIdentity } from "../services/run-identity.js"; -const support = await getEmbeddedPostgresTestSupport(); +const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL; +const support = externalDatabaseUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); (support.supported ? describe : describe.skip)("durable execution identity", () => { let database: Awaited>; let db: ReturnType; beforeAll(async () => { - database = await startEmbeddedPostgresTestDatabase("paperclip-run-identity-"); - db = createDb(database.connectionString); + if (externalDatabaseUrl) { await applyPendingMigrations(externalDatabaseUrl); db = createDb(externalDatabaseUrl); } + else { + database = await startEmbeddedPostgresTestDatabase("paperclip-run-identity-"); + db = createDb(database.connectionString); + } }, 30_000); - afterAll(async () => { await database?.cleanup(); }, 60_000); + afterAll(async () => { if (externalDatabaseUrl) await closeRegisteredClients(externalDatabaseUrl); await database?.cleanup(); }, 60_000); async function seed() { const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(); @@ -123,7 +127,7 @@ const support = await getEmbeddedPostgresTestSupport(); }, ); - it.each([undefined, "github_credentials", "request_authentication"] as const)("holds acquisition during uncertain steering and never rewinds on replay with %s", async (readScope) => { + it.each([undefined, "github_credentials", "request_authentication", "identity", "task_authorization", "token_broker", "project_execution"] as const)("holds acquisition during uncertain steering and never rewinds on replay with %s", async (readScope) => { const options = readScope ? { readScope } : undefined; const input = await seed(); await initializeRunIdentity(db, { ...input, messageIds: [input.messageIds[0]], responsibleUserId: "A", cause: "instruction" }); @@ -176,7 +180,7 @@ const support = await getEmbeddedPostgresTestSupport(); expect(captured.run.responsibleUserId).toBeNull(); expect(captured.context?.parentContextId).toBe(origin.id); }); - it.each([undefined, "github_credentials", "request_authentication"] as const)("recovers an uncertain native acknowledgement with %s", async (readScope) => { + it.each([undefined, "github_credentials", "request_authentication", "identity", "task_authorization", "token_broker", "project_execution"] as const)("recovers an uncertain native acknowledgement with %s", async (readScope) => { const options = readScope ? { readScope } : undefined; const input = await seed(); await initializeRunIdentity(db, { ...input, messageIds: [], responsibleUserId: "A", cause: "instruction" }); @@ -260,9 +264,9 @@ const support = await getEmbeddedPostgresTestSupport(); await expect(initialization).resolves.toMatchObject({ responsibleUserId: "A" }); }); - it.each(["initialize", "capture"] as const)( - "can %s identity while an audit append holds foreign-key locks", - async (operation) => { + it.each((["initialize", "capture"] as const).flatMap(operation => [false, true].map(withQuery => ({ operation, withQuery }))))( + "can $operation identity while an audit append holds foreign-key locks (query: $withQuery)", + async ({ operation, withQuery }) => { const input = await seed(); if (operation === "capture") { await initializeRunIdentity(db, { ...input, messageIds: [], responsibleUserId: "A", cause: "instruction" }); @@ -270,11 +274,22 @@ const support = await getEmbeddedPostgresTestSupport(); // A real append holds KEY SHARE on both parent rows until it commits. // Bound the other connection's wait so a conflicting lock fails this // regression instead of leaving both transactions waiting for each other. - const identityDb = createDb(`${database.connectionString}?options=-c%20lock_timeout%3D1000`, { + const identityUrl = new URL(externalDatabaseUrl ?? database.connectionString); + if (withQuery) { + identityUrl.searchParams.set("application_name", "identity-query-test"); + identityUrl.searchParams.set("options", [identityUrl.searchParams.get("options"), "-c statement_timeout=5000"].filter(Boolean).join(" ")); + } + identityUrl.searchParams.set("options", [identityUrl.searchParams.get("options"), "-c lock_timeout=1000"].filter(Boolean).join(" ")); + const identityDb = createDb(identityUrl.toString(), { maxConnections: 1, }); const [settings] = await identityDb.execute(sql`show lock_timeout`); expect(settings?.lock_timeout).toBe("1s"); + if (withQuery) { + expect(identityUrl.searchParams.get("application_name")).toBe("identity-query-test"); + const [timeouts] = await identityDb.execute(sql`show statement_timeout`); + expect(timeouts?.statement_timeout).toBe("5s"); + } await db.transaction(async (audit) => { await audit.insert(secretAccessEvents).values({ companyId: input.companyId, diff --git a/server/src/routes/projects.ts b/server/src/routes/projects.ts index c6479982335..272051bf521 100644 --- a/server/src/routes/projects.ts +++ b/server/src/routes/projects.ts @@ -423,7 +423,7 @@ export function projectRoutes(db: Db) { // The receipt lock can wait too. Keep current authority behind every // admission wait and ahead of both receipt replay and project creation. await withExternalExecutionAdmission(tx, actorCompanyId, agentId, runId, async () => { - const current = await projectToolContext(tx as unknown as Db, museActor, true); + const current = await projectToolContext(tx as unknown as Db, museActor, true, "Project", { includeExecutionInput: true }); try { const execution = parseNativeExecutionInput(current.run.runnerProfileJson?.nativeExecutionInput); const broker = museExternalAgents(db); diff --git a/server/src/services/connection-intents.ts b/server/src/services/connection-intents.ts index 1788c0ebd60..f0d143861e5 100644 --- a/server/src/services/connection-intents.ts +++ b/server/src/services/connection-intents.ts @@ -214,7 +214,7 @@ export function connectionIntentService(db: Db) { throw forbidden("Runtime tool token is no longer active"); } if (run.activeIdentityContextId && !failedAuthRun) { - const current = await captureRunIdentity(db, { companyId: run.companyId, agentId: run.agentId, runId: run.id }); + const current = await captureRunIdentity(db, { companyId: run.companyId, agentId: run.agentId, runId: run.id }, { readScope: "identity" }); run = { ...run, responsibleUserId: current.run.responsibleUserId }; } if (!run.responsibleUserId) throw forbidden("This task needs a responsible user to connect a service"); diff --git a/server/src/services/connection-management.ts b/server/src/services/connection-management.ts index fcc532476be..0c1740f04ba 100644 --- a/server/src/services/connection-management.ts +++ b/server/src/services/connection-management.ts @@ -3,7 +3,7 @@ import { isUuidLike } from "@paperclipai/shared"; import { agents, agentApiKeys, heartbeatRuns, runIdentityContexts, issues, projects, toolApplications, toolCatalogEntries, toolConnections, toolConnectionInstalls, toolProfiles, toolProfileEntries, toolProfileBindings, toolMcpGateways, aiConnectionPools, connectionGrants, routines, type Db } from "@paperclipai/db"; import { forbidden } from "../errors.js"; import { authorizationService, canActorReadIssuePrivacy, canActorReadProjectPrivacy, type AuthorizationActor, type AuthorizationDecision } from "./authorization.js"; -import { captureRunIdentity } from "./run-identity.js"; +import { captureRunIdentity, runIdentityTaskFields, type RunIdentityCapture } from "./run-identity.js"; import type { LogActivityInput } from "./activity-log.js"; import type { ConnectionManagementAction, ConnectionManagementResource } from "./connection-management-policy.js"; @@ -17,7 +17,7 @@ export type ConnectionManagementContext = { reauthorize: (continuation?: { afterHumanAction: true }) => Promise; }; -type CapturedConnectionRun = Omit>, "context"> & { context: Awaited>["context"] | null }; +type CapturedConnectionRun = RunIdentityCapture<"task_authorization">; const connectionReadRuns = new WeakMap(); export function responsibleConnectionUser(actor: AuthorizationActor): string | null { @@ -51,12 +51,12 @@ export function connectionManagementService(db: Db) { const reusable = options.capabilityProbe && !options.afterHumanAction && options.readContext?.actor === actor && options.readContext.resource.companyId === resource.companyId ? connectionReadRuns.get(options.readContext) : undefined; const captured = reusable ?? (options.afterHumanAction ? await (async () => { - const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, actor.runId!), eq(heartbeatRuns.agentId, actor.agentId!), eq(heartbeatRuns.companyId, resource.companyId))); + const [run] = await db.select(runIdentityTaskFields).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, actor.runId!), eq(heartbeatRuns.agentId, actor.agentId!), eq(heartbeatRuns.companyId, resource.companyId))); if (!run || !["running", "succeeded"].includes(run.status)) throw forbidden("The originating run cannot continue this authentication"); const [pending] = await db.select().from(runIdentityContexts).where(and(eq(runIdentityContexts.runId, run.id), eq(runIdentityContexts.status, "pending"))); if (pending) throw forbidden("Run identity is being reconciled"); return { run, context: null }; - })() : await captureRunIdentity(db, { companyId: resource.companyId, agentId: actor.agentId, runId: actor.runId })); + })() : await captureRunIdentity(db, { companyId: resource.companyId, agentId: actor.agentId, runId: actor.runId }, { readScope: "task_authorization" })); capturedRun = captured; const { run, context } = captured; if ((context?.responsibleUserId ?? run.responsibleUserId) !== responsibleUserId) throw forbidden("Run responsible-user authority has changed"); diff --git a/server/src/services/connectors/slack-authority.ts b/server/src/services/connectors/slack-authority.ts index 06da6455d35..ee19440f45e 100644 --- a/server/src/services/connectors/slack-authority.ts +++ b/server/src/services/connectors/slack-authority.ts @@ -8,7 +8,7 @@ import { type chatConversations, } from "@paperclipai/db"; import { and, eq, inArray } from "drizzle-orm"; -import { captureRunIdentity } from "../run-identity.js"; +import { captureRunIdentity, runIdentityTaskFields } from "../run-identity.js"; import { forbidden } from "../../errors.js"; import { boundRunIssueId } from "../run-task-binding.js"; import { chatRunSourceMessageId } from "../chat-channels/run-origin.js"; @@ -110,7 +110,7 @@ export async function slackRunOrigin(db: Db, binding: SlackTaskBinding) { if (!approved || !binding.identityContextId) throw forbidden("Slack approval is not executing for this task"); const [run] = await db - .select() + .select(runIdentityTaskFields) .from(heartbeatRuns) .where( and( @@ -141,7 +141,7 @@ export async function slackRunOrigin(db: Db, binding: SlackTaskBinding) { ); captured = { run, context }; } else { - captured = await captureRunIdentity(db, binding); + captured = await captureRunIdentity(db, binding, { readScope: "task_authorization" }); if ( binding.identityContextId && binding.identityContextId !== captured.context?.id diff --git a/server/src/services/execution-workspace-repositories.ts b/server/src/services/execution-workspace-repositories.ts index fae172296ff..4d4cfe16b86 100644 --- a/server/src/services/execution-workspace-repositories.ts +++ b/server/src/services/execution-workspace-repositories.ts @@ -134,7 +134,7 @@ export function executionWorkspaceRepositoryService(db: Db) { let localTrusted = input.actor.source === "local_implicit"; if (input.actor.type === "agent") { if (input.actor.source !== "agent_jwt" || !input.actor.agentId || !input.actor.runId || input.actor.companyId !== input.companyId) throw forbidden("Repository preparation requires an authenticated task run"); - const identity = await captureRunIdentity(db, { companyId: input.companyId, agentId: input.actor.agentId, runId: input.actor.runId }); + const identity = await captureRunIdentity(db, { companyId: input.companyId, agentId: input.actor.agentId, runId: input.actor.runId }, { readScope: "task_authorization" }); const runIssueId = identity.run.nativeIssueId ?? identity.run.contextSnapshot?.issueId; if (runIssueId !== input.issueId) throw forbidden("Repository preparation is scoped to the current task"); userId = identity.run.responsibleUserId; diff --git a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts index aa6863205fb..71f9e3ea02b 100644 --- a/server/src/services/native-runtime/paperclip-runner-tool-authority.ts +++ b/server/src/services/native-runtime/paperclip-runner-tool-authority.ts @@ -458,7 +458,7 @@ export class PaperclipRunnerToolAuthority { } if (RUNTIME_CONNECTION_TOOL_DEFINITIONS.some((tool) => tool.name === call.tool)) { await this.#boundContext(); - const { run } = await captureRunIdentity(this.db, this.binding); + const { run } = await captureRunIdentity(this.db, this.binding, { readScope: "identity" }); if (!run.responsibleUserId) throw forbidden("This task needs a responsible user before requesting a connection"); const claims = { sub: this.binding.agentId, company_id: this.binding.companyId, @@ -794,9 +794,9 @@ export class PaperclipRunnerToolAuthority { } case "report_progress": return this.#reportProgress(input); case "request_human_input": return this.#requestHumanInput(input, - (await captureRunIdentity(this.db, this.binding)).context?.id ?? null); + (await captureRunIdentity(this.db, this.binding, { readScope: "identity" })).context?.id ?? null); case "create_task": return this.#createTask(input, - (await captureRunIdentity(this.db, this.binding)).context?.id ?? null); + (await captureRunIdentity(this.db, this.binding, { readScope: "identity" })).context?.id ?? null); case "reassign_task": return this.#reassignTask(input); case "set_task_title": return this.#setTaskTitle(input); case "set_task_monitor": return this.#setTaskMonitor(input); diff --git a/server/src/services/project-skill-source-bindings.ts b/server/src/services/project-skill-source-bindings.ts index 29ff8bb1875..259e0dda705 100644 --- a/server/src/services/project-skill-source-bindings.ts +++ b/server/src/services/project-skill-source-bindings.ts @@ -12,7 +12,7 @@ export async function projectSkillSetupIdentity(db: Db, companyId: string, actor let connectionId: string | null = null, refreshGrantId: string | null = null; if (actor?.type === 'agent' && actor.source === 'agent_jwt') { if (!actor.runId || !actor.agentId || actor.companyId !== companyId) throw forbidden('Project skill setup requires an authenticated agent run.'); - const identity = await captureRunIdentity(db, { companyId, agentId: actor.agentId, runId: actor.runId }); + const identity = await captureRunIdentity(db, { companyId, agentId: actor.agentId, runId: actor.runId }, { readScope: 'identity' }); userId = identity.run.responsibleUserId; // Only reuse the connection already authorized by managed GitHub delivery // for this active run. Responsibility alone cannot discover another grant. diff --git a/server/src/services/project-tool-context.ts b/server/src/services/project-tool-context.ts index a0cc9468351..89be328b052 100644 --- a/server/src/services/project-tool-context.ts +++ b/server/src/services/project-tool-context.ts @@ -2,16 +2,21 @@ import type { Request } from "express"; import { and, eq } from "drizzle-orm"; import { issues, type Db } from "@paperclipai/db"; import { forbidden } from "../errors.js"; -import { captureRunIdentity } from "./run-identity.js"; +import { captureRunIdentity, type RunIdentityCapture } from "./run-identity.js"; +type ProjectToolResult = { run: Run; issue: typeof issues.$inferSelect; userId: string | null; localTrusted: boolean }; /** Resolve authority from the authenticated run, never caller-supplied user/task IDs. */ -export async function projectToolContext(db: Db, actor: Request["actor"], write = false, resource = "Project") { +export function projectToolContext(db: Db, actor: Request["actor"], write?: boolean, resource?: string): Promise["run"]>>; +export function projectToolContext(db: Db, actor: Request["actor"], write: boolean, resource: string, options: { includeExecutionInput: true }): Promise["run"]>>; +export async function projectToolContext(db: Db, actor: Request["actor"], write = false, resource = "Project", options?: { includeExecutionInput: true }) { if (actor.type !== "agent" || actor.source !== "agent_jwt" || !actor.runId || !actor.agentId || !actor.companyId) { throw forbidden(`${resource} tools require an authenticated agent run`); } // Acquire task/run locks before checking mode and session generation. A reset, // cancellation, or steering update cannot race a committing project mutation. - const identity = await captureRunIdentity(db, { companyId: actor.companyId, agentId: actor.agentId, runId: actor.runId }); + const binding = { companyId: actor.companyId, agentId: actor.agentId, runId: actor.runId }; + const identity = options?.includeExecutionInput ? await captureRunIdentity(db, binding, { readScope: "project_execution" }) : + await captureRunIdentity(db, binding, { readScope: "task_authorization" }); const run = identity.run; const snapshot = run.contextSnapshot ?? {}; const issueId = run.nativeIssueId ?? (typeof snapshot.issueId === "string" ? snapshot.issueId : null); diff --git a/server/src/services/run-identity.ts b/server/src/services/run-identity.ts index 166b1ae8cac..f72cc974ca7 100644 --- a/server/src/services/run-identity.ts +++ b/server/src/services/run-identity.ts @@ -465,16 +465,71 @@ export async function acceptSteeredIdentity( if (accepted) await activate(executor, accepted); } -export async function captureRunIdentity( +const runIdentityFields = { + id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, agentId: heartbeatRuns.agentId, + status: heartbeatRuns.status, responsibleUserId: heartbeatRuns.responsibleUserId, + activeIdentityContextId: heartbeatRuns.activeIdentityContextId, +}; + +export const runIdentityTaskFields = { + ...runIdentityFields, nativeIssueId: heartbeatRuns.nativeIssueId, issueId: heartbeatRuns.issueId, + wakeupRequestId: heartbeatRuns.wakeupRequestId, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'taskId', ${heartbeatRuns.contextSnapshot}->'taskId', + 'conversationSessionGeneration', ${heartbeatRuns.contextSnapshot}->'conversationSessionGeneration', + 'paperclipSkillTest', ${heartbeatRuns.contextSnapshot}->'paperclipSkillTest') else null end`, +}; + +export const runIdentityBrokerFields = { + ...runIdentityFields, invocationSource: heartbeatRuns.invocationSource, + contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' + then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId', + 'projectId', ${heartbeatRuns.contextSnapshot}->'projectId', 'routineId', ${heartbeatRuns.contextSnapshot}->'routineId', + 'responsibleUserId', ${heartbeatRuns.contextSnapshot}->'responsibleUserId', + 'responsible_user_id', ${heartbeatRuns.contextSnapshot}->'responsible_user_id', + 'paperclipIssue', case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->'paperclipIssue') = 'object' + then jsonb_build_object('id', ${heartbeatRuns.contextSnapshot}->'paperclipIssue'->'id', + 'projectId', ${heartbeatRuns.contextSnapshot}->'paperclipIssue'->'projectId', + 'responsibleUserId', ${heartbeatRuns.contextSnapshot}->'paperclipIssue'->'responsibleUserId', + 'responsible_user_id', ${heartbeatRuns.contextSnapshot}->'paperclipIssue'->'responsible_user_id') else null end) + else null end`, +}; + +const runIdentityProjectFields = { + ...runIdentityTaskFields, runtimeMode: heartbeatRuns.runtimeMode, + runnerProfileJson: sql | null>`case when jsonb_typeof(${heartbeatRuns.runnerProfileJson}) = 'object' + then case when ${heartbeatRuns.runnerProfileJson} ? 'nativeExecutionInput' + then jsonb_build_object('nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput') + else '{}'::jsonb end else null end`, +}; + +type FullIdentityRun = typeof heartbeatRuns.$inferSelect; +type IdentityRun = Pick; +type TaskIdentityRun = Pick; +type BrokerIdentityRun = Pick; +type ProjectIdentityRun = Pick; +export type RunIdentityReadScope = "github_credentials" | "request_authentication" | "identity" | "task_authorization" | "token_broker" | "project_execution"; +export type RunIdentityCapture = { + run: Scope extends "identity" ? IdentityRun : Scope extends "task_authorization" ? TaskIdentityRun : + Scope extends "token_broker" ? BrokerIdentityRun : Scope extends "project_execution" ? ProjectIdentityRun : FullIdentityRun; + context: RunIdentityContext | null; +}; + +export async function captureRunIdentity( db: Db, input: { companyId: string; runId: string; agentId: string }, - options?: { readScope: "github_credentials" | "request_authentication" }, -) { + options?: { readScope: Scope }, +): Promise> { // Lock acquisition serializes with steering delivery and its durable acknowledgement. return db.transaction(async (tx) => { await lockIdentityTask(tx, input.companyId, input.runId); const [run] = await tx - .select(options?.readScope === "github_credentials" || options?.readScope === "request_authentication" ? { + .select(options?.readScope === "identity" ? runIdentityFields : + options?.readScope === "task_authorization" ? runIdentityTaskFields : + options?.readScope === "token_broker" ? runIdentityBrokerFields : + options?.readScope === "project_execution" ? runIdentityProjectFields : + options?.readScope === "github_credentials" || options?.readScope === "request_authentication" ? { ...getTableColumns(heartbeatRuns), contextSnapshot: sql | null>`case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' then jsonb_build_object('issueId', ${heartbeatRuns.contextSnapshot}->'issueId', @@ -528,7 +583,7 @@ export async function captureRunIdentity( ), ) : []; - return { run, context: context ?? null }; + return { run, context: context ?? null } as RunIdentityCapture; }); } diff --git a/server/src/services/tool-access/token-broker/admission.ts b/server/src/services/tool-access/token-broker/admission.ts index 5b30a3fd915..b4a98042001 100644 --- a/server/src/services/tool-access/token-broker/admission.ts +++ b/server/src/services/tool-access/token-broker/admission.ts @@ -1,5 +1,5 @@ import type { DbTransaction } from "../types.js"; -import { captureRunIdentity } from "../../run-identity.js"; +import { captureRunIdentity, runIdentityBrokerFields } from "../../run-identity.js"; import { and, eq, gte, inArray, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { connectionTokenIssuances, companyMemberships, heartbeatRuns, toolConnections, toolProfileBindings, toolProfileEntries, toolProfiles } from "@paperclipai/db"; @@ -75,11 +75,11 @@ export function createTokenBrokerAdmission(db: Db, now: () => Date) { runId: string; }) { const [initialRun] = await db - .select() + .select(runIdentityBrokerFields) .from(heartbeatRuns) .where(eq(heartbeatRuns.id, input.runId)); const run = initialRun?.activeIdentityContextId - ? (await captureRunIdentity(db, input)).run + ? (await captureRunIdentity(db, input, { readScope: "token_broker" })).run : initialRun; if ( !run || diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index ff5e1028ffc..57763f8b8d5 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -2199,7 +2199,7 @@ export function createToolGatewayService( companyId: session.companyId, agentId: session.agentId, runId: session.runId, - }); + }, { readScope: "identity" }); return { ...session, identityContextId: captured.context?.id, diff --git a/server/src/services/voice/speko-agent-tools.ts b/server/src/services/voice/speko-agent-tools.ts index 6332329fe35..7f0ce74b78f 100644 --- a/server/src/services/voice/speko-agent-tools.ts +++ b/server/src/services/voice/speko-agent-tools.ts @@ -19,7 +19,7 @@ export function registerSpekoVoiceRuntime(db: Db, start: (binding: Binding, invo async function resolveAuthority(db: Db, session: SpekoTaskSession, endpointId: string) { if (!(await instanceSettingsService(db).getExperimental()).enableChatConnectors) throw forbidden("Voice connections are disabled"); if (!runtimes.has(db) || !session.agentId || !session.runId || !session.issueId) throw forbidden("Speko tools require an active task run"); - const { run, context } = await captureRunIdentity(db, { companyId: session.companyId, agentId: session.agentId, runId: session.runId }); + const { run, context } = await captureRunIdentity(db, { companyId: session.companyId, agentId: session.agentId, runId: session.runId }, { readScope: "task_authorization" }); if (boundRunIssueId(run) !== session.issueId || !run.responsibleUserId || session.identityContextId && session.identityContextId !== context?.id) throw forbidden("The accepted requester or task changed"); const [endpoint] = await db.select().from(chatEndpoints).where(and(eq(chatEndpoints.companyId, session.companyId), eq(chatEndpoints.id, endpointId), eq(chatEndpoints.provider, "speko"), eq(chatEndpoints.assignedAgentId, session.agentId), eq(chatEndpoints.status, "active"))); if (!endpoint) throw forbidden("This agent has no active Speko connection"); From 3d07e06be7ff24c04e8143eda1e893cf54349652 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:31:40 -0500 Subject: [PATCH 12/41] perf(chat): project reply lineage run context (#17932) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A run can continue a task that began in an external chat. > - The reply router follows durable lineage to find the authorized conversation. > - Each lineage hop reads the full run context, although it uses only nine keys. > - Wide prompts and workspace data make that read transfer far more data than the router needs. > - This pull request selects those keys inside PostgreSQL and preserves their exact JSON values. > - The benefit is less database transfer with the same reply destination and authorization checks. ## Linked Issues or Issue Description Refs: #17255, #17894. The first owns related chat review delivery. The second is an independent result-projection optimization. This PR has no dependency on another open PR. A public issue and PR search found no duplicate change. The roadmap does not list this query optimization as planned core work. **What existing behavior does this improve?** The server resolves the conversation that may receive a reply after one or more task continuations. **Current behavior** Each hop reads agent identity, responsible user identity, and the complete context snapshot. The snapshot can contain large prompts and workspace data that the router never reads. **Proposed behavior** Select only the nine context keys that this function consumes. Preserve missing keys, explicit JSON nulls, arrays, objects, and scalar values. Keep non-object contexts unchanged. Keep all durable authorization queries and the cycle-safe traversal. **Reason and benefit** A synthetic wide context needs 262,402 DataRow bytes before the change and 244 after it. A valid 24-continuation lineage needs 6,560,338 bytes before and 6,388 after it. The reduction occurs before PostgreSQL sends the result. **Breaking changes** None. The public API, stored context, reply bindings, tenant checks, and admission proofs retain their existing behavior. ## What Changed - Project an object context through `jsonb_each` and `jsonb_object_agg`. Retain only the nine consumed keys. Preserve all selected JSON values and key presence. - Keep the agent and responsible-user columns. Keep the company and run predicates. - Add 22 real PostgreSQL tests for routing, malformed JSON, strict tool-action receipt comparisons, tenant boundaries, 24 valid continuation hops, and an authorized cycle. - Add a synthetic benchmark that loads the pinned baseline service, captures its generated SQL, compares complete bindings and selected columns, and measures text-format PostgreSQL DataRow frames. - Run owned resource cleanup in nested `finally` blocks so fixture deletion or client close failures cannot skip cluster and scratch cleanup. Preserve the original benchmark failure alongside a cleanup failure. The complete consumer-key audit is: | Context keys | Consumer | | --- | --- | | `issueId`, `taskId` | Task identity fence, with the existing fallback order | | `wakeCommentIds`, `wakeCommentId`, `commentId` | Board reply and root comment provenance | | `source`, `sourceRunId`, `interactionId` | Durable continuation branches and tool-action receipt checks | | `endpointId` | Root destination fence | Missing `interactionId` stays absent. It does not become JSON null. This matters because the tool-action branch compares it strictly with the durable wake payload. The tests cover absent, null, string, number, and object values. They also remove the delivery receipt to prove that context alone cannot authorize a reply. ## Verification Baseline: `3cd3af5d75f200185ed4d9703a3145594a489b13`. Treatment: `3439ba7b714090d9f251816efec26622b3df223b`. PostgreSQL 17.11, Node 26.4.0. All data is synthetic. The benchmark holds imported helpers and fixtures identical and loads only the baseline `issues.ts` from history. It records generated SQL, parameters, plans, source SHA-256, and the initial worktree state before scratch creation. The committed final measurement reports a clean worktree and source SHA-256 `cd7e955179118fda12f6b2e981a17efff54137cbf241019785a3f8d16a7dcbb8`. Reproduce the benchmark: ```sh node --import ./server/node_modules/tsx/dist/loader.mjs scripts/benchmark-chat-reply-lineage.ts --baseline-ref=3cd3af5d75f200185ed4d9703a3145594a489b13 ``` The default creates disposable embedded PostgreSQL. `PAPERCLIP_CHAT_LINEAGE_BENCHMARK_DATABASE_URL` can name a dedicated external database that ends in `_test`. The harness applies migrations and removes its own fixture rows. It never uses an application database by default. Each wide run has 192 KiB of random bytes encoded as base64. This produces a 262,144-byte incompressible body. Two warmups precede seven timed samples. Baseline and treatment alternate order. Every scenario returns the same complete bindings. Every retained SQL column matches its baseline value, and the context matches the exact key projection. The acceptance check requires removal of every unrelated synthetic body byte. | Scenario | Baseline DataRow bytes | Changed DataRow bytes | Baseline SQL median ms (range) | Changed SQL median ms (range) | | --- | ---: | ---: | ---: | ---: | | small | 282 | 244 | 0.927 (0.900–1.455) | 1.027 (0.801–1.063) | | wide | 262,402 | 244 | 2.618 (2.440–4.831) | 1.490 (1.400–1.815) | | wide-24-continuations | 6,560,338 | 6,388 | 87.450 (80.103–130.197) | 43.103 (38.549–52.864) | These bytes are exact text-format DataRow frame sizes. They exclude row descriptions, command metadata, TLS, and TCP. Timings cover the captured lineage SQL, including all hops. They exclude the other authorization queries. The small control is neutral and timing ranges overlap on wider cases. An earlier loaded run had a small-control median of 1.15 ms before and 1.33 ms after. No deployed latency or billing improvement is claimed. The byte reduction is the primary result. Local checks: - `pnpm run preflight:workspace-links`: passed. - `pnpm exec vitest run server/src/services/chat-reply-lineage-context.test.ts`: 22 passed on the committed rebased head. - The existing `chat-interaction-continuation.test.ts` file: all 28 tests passed. These cover confirmation, verdict, connection, and invalid-authority paths. - Five selected cases in `chat-channels.integration.test.ts`: passed in a fresh dedicated database. They cover closed-choice question delivery and steering, Telegram confirmation, ambiguous close control, and both fresh-after-close authorization paths. - `pnpm -r typecheck` and `pnpm build`: passed again after the master rebase. - `node scripts/check-module-boundaries.mjs` and `git diff --check`: passed. - Cleanup repair at `3439ba7b714090d9f251816efec26622b3df223b`: all 22 focused PostgreSQL tests, the benchmark above, and server `tsc --noEmit` passed. Six controlled cleanup-failure cases confirmed later finalizers run; benchmark and cleanup errors remain visible together. The unchanged baseline failed the wide-result regression: 262,383 value bytes exceeded its 1 KiB bound. The treatment passes. Initial fixture setup needed a required conversation label and explicit foreign-key cleanup. The first full typecheck found a row-type mismatch in the new capture helper; that helper was corrected. The complete typecheck then passed. No production logic changed for those setup repairs. A prior assembled run reused the interaction test database. Its retained running fixtures caused a global milestone-count assertion to fail. Later reuse exposed retained bot registration and identity state. An attempted manual reset named a nonexistent table and did not complete. These failed attempts are not acceptance evidence. The final five-case assembled proof used a newly created and migrated database and passed every assertion. No existing assertion was changed. The full local `pnpm test:run` was not run. Focused tests used the coordinated disposable PostgreSQL server to avoid uncontrolled embedded-cluster concurrency on this host. The full final-head CI matrix passes on `3439ba7b714090d9f251816efec26622b3df223b`: 55 successful checks, one neutral check, and two optional Storybook skips. Greptile completed at 5/5 on that exact head, with zero unresolved findings. The existing interaction and assembled cases ran before the unrelated master rebase; the measured service source is unchanged. Focused tests, benchmark, full typecheck, and build passed on the rebased production-change head. The current cleanup-only head reran focused tests, benchmark, and server TypeScript validation; production service source is unchanged. ## Risks - Future lineage consumers must add any new context key to the SQL projection and the correctness oracle. - Selected values retain their complete JSON content. Non-object contexts pass through unchanged. Savings depend on unrelated object data being present. - PostgreSQL must still read the stored object to make the projection. The change reduces returned data, rather than removing all server-side JSON work. - Question delivery, tool action, connection, monitor, publication, company, and agent proofs retain their existing owners. No hop limit, cache, schema change, startup task, or deployment is added. - Benchmark usage and the result contract are documented here. No public command or API contract changes. ## Model Used OpenAI GPT-6 through Codex. The exact runtime model ID and context window are not exposed by this environment. Capabilities used: source analysis, code editing, shell tools, real PostgreSQL experiments, and automated tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- scripts/benchmark-chat-reply-lineage.ts | 110 +++++++++++++++++ .../__tests__/helpers/chat-reply-lineage.ts | 67 +++++++++++ .../chat-reply-lineage-context.test.ts | 111 ++++++++++++++++++ server/src/services/issues.ts | 8 +- 4 files changed, 295 insertions(+), 1 deletion(-) create mode 100644 scripts/benchmark-chat-reply-lineage.ts create mode 100644 server/src/__tests__/helpers/chat-reply-lineage.ts create mode 100644 server/src/services/chat-reply-lineage-context.test.ts diff --git a/scripts/benchmark-chat-reply-lineage.ts b/scripts/benchmark-chat-reply-lineage.ts new file mode 100644 index 00000000000..353b92de5be --- /dev/null +++ b/scripts/benchmark-chat-reply-lineage.ts @@ -0,0 +1,110 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { promises as fs, readFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { applyPendingMigrations, createDb, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js"; +import { resolveChatOriginPublicationBindings } from "../server/src/services/issues.js"; +import { captureLineageReads, chatReplyLineageFixture, cleanupChatReplyLineageFixture, expectedLineageContext, type LineageQuery, type LineageResolver } from "../server/src/__tests__/helpers/chat-reply-lineage.js"; +import { resolveExternalTestDatabaseUrl } from "../server/src/__tests__/helpers/external-test-database.js"; + +const repoRoot = fileURLToPath(new URL("../", import.meta.url)); +const ref = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15); +assert.ok(ref, "Pass a pinned pre-fix --baseline-ref"); +const git = (args: string[]) => execFileSync("git", args, { cwd: repoRoot, encoding: "utf8" }).trim(); +const sourcePath = "server/src/services/issues.ts"; +const baselineRevision = git(["rev-parse", `${ref}^{commit}`]), treatmentRevision = git(["rev-parse", "HEAD"]); +const uncommittedChanges = git(["status", "--porcelain"]), treatmentDiff = git(["diff", "HEAD", "--", sourcePath]); +const sourceHash = () => createHash("sha256").update(readFileSync(path.join(repoRoot, sourcePath))).digest("hex"); +const treatmentSourceSha256 = sourceHash(); +const keepAlive = setInterval(() => {}, 1000); +let db: Db | undefined, temporary: Awaited> | undefined, baselineDirectory: string | undefined; +const companyIds: string[] = []; +let benchmarkFailure: { error: unknown } | undefined; +try { + const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_CHAT_LINEAGE_BENCHMARK_DATABASE_URL"); + if (externalUrl) await applyPendingMigrations(externalUrl); + else temporary = await startEmbeddedPostgresTestDatabase("chat-lineage-benchmark-"); + db = createDb(externalUrl ?? temporary!.connectionString); + baselineDirectory = await fs.mkdtemp(path.join(repoRoot, "server/scripts/.chat-lineage-baseline-")); + const original = pathToFileURL(path.join(repoRoot, sourcePath)); + const baselineSource = execFileSync("git", ["show", `${baselineRevision}:${sourcePath}`], { cwd: repoRoot, encoding: "utf8" }); + await fs.writeFile(path.join(baselineDirectory, "issues.ts"), baselineSource.replace(/(from\s+|import\()(["'])(\.{1,2}\/[^"']+)\2/g, + (_match, prefix: string, quote: string, specifier: string) => `${prefix}${quote}${fileURLToPath(new URL(specifier, original))}${quote}`)); + const baseline = (await import(pathToFileURL(path.join(baselineDirectory, "issues.ts")).href)).resolveChatOriginPublicationBindings as LineageResolver; + const measurements = []; + for (const scenario of [{ name: "small", bytes: 16, hops: 0 }, { name: "wide", bytes: 192 * 1024, hops: 0 }, { name: "wide-24-continuations", bytes: 192 * 1024, hops: 24 }]) { + const f = await chatReplyLineageFixture(db, scenario.bytes); companyIds.push(f.companyId); + let runId = f.root.id; + for (let hop = 0; hop < scenario.hops; hop++) runId = (await f.continueFrom(runId)).run.id; + await db.$client.unsafe("ANALYZE"); + async function capture(resolver: LineageResolver) { + const reads: LineageQuery[] = []; + const result = await resolver(captureLineageReads(db!, reads), f.companyId, f.issueId, runId); + assert.deepEqual(result, f.expected); + assert.equal(reads.length, scenario.hops + 1); + return { result, reads }; + } + const before = await capture(baseline), after = await capture(resolveChatOriginPublicationBindings); + assert.deepEqual(after.result, before.result); + for (let index = 0; index < before.reads.length; index++) { + const row = before.reads[index].rows[0]; + assert.deepEqual(after.reads[index].rows, [{ ...row, contextSnapshot: expectedLineageContext(row.contextSnapshot) }]); + } + const variants = [before, after], samples: number[][] = [[], []], bytes: number[] = [0, 0]; + for (let iteration = 0; iteration < 9; iteration++) { + for (const variant of iteration % 2 ? [1, 0] : [0, 1]) { + const started = performance.now(); let returned = 0; + for (const query of variants[variant].reads) { + const rows = await db.$client.unsafe(query.sql, query.params as never[]).raw(); + returned += rows.reduce((sum, row) => sum + 7 + row.reduce((n: number, value: Buffer | null) => n + 4 + (value?.length ?? 0), 0), 0); + } + if (iteration >= 2) samples[variant].push(performance.now() - started); + bytes[variant] = returned; + } + } + const minimumSaving = Buffer.byteLength(f.context.prompt) * (scenario.hops + 1); + assert.ok(bytes[0] - bytes[1] >= minimumSaving, "All unrelated synthetic body bytes must be removed before transfer"); + const plans = []; + for (const variant of variants) { + const query = variant.reads[0]; + const [plan] = await db.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]); + plans.push(plan["QUERY PLAN"]); + } + const timing = (values: number[]) => { const sorted = [...values].sort((a, b) => a - b); return { medianMs: sorted[3], minMs: sorted[0], maxMs: sorted[6], samplesMs: values }; }; + measurements.push({ scenario, bodyBytesPerRun: Buffer.byteLength(f.context.prompt), completeBindingsEqual: true, selectedColumnsEqual: true, + baselineDataRowBytes: bytes[0], treatmentDataRowBytes: bytes[1], minimumSaving, baselineTiming: timing(samples[0]), treatmentTiming: timing(samples[1]), + baselineQueries: before.reads.map(({ rows: _rows, ...query }) => query), treatmentQueries: after.reads.map(({ rows: _rows, ...query }) => query), plans }); + } + assert.equal(sourceHash(), treatmentSourceSha256, "Measured service source changed during the benchmark"); + console.log(JSON.stringify({ baselineRevision, treatmentRevision, sourcePath, treatmentSourceSha256, uncommittedChanges, treatmentDiff, + byteBoundary: "Exact text-format PostgreSQL DataRow frames; excludes row descriptions, command metadata, TLS and TCP. Timings cover captured lineage SQL only, not the complete authorization service.", + dependencyScope: "Only the pinned baseline issues.ts is loaded from history; imported helpers and synthetic fixtures are held identical.", + postgres: await db.$client.unsafe("select version()"), settings: await db.$client.unsafe("select name,setting,unit from pg_settings where name in ('shared_buffers','work_mem','jit') order by name"), + node: process.version, measurements }, null, 2)); +} catch (error) { + benchmarkFailure = { error }; + throw error; +} finally { + try { + try { + if (db) for (const companyId of companyIds) await cleanupChatReplyLineageFixture(db, companyId); + } finally { + try { + await db?.$client.end({ timeout: 0 }); + } finally { + try { + await temporary?.cleanup(); + } finally { + try { + if (baselineDirectory) await fs.rm(baselineDirectory, { recursive: true, force: true }); + } finally { clearInterval(keepAlive); } + } + } + } + } catch (error) { + if (benchmarkFailure) throw new AggregateError([benchmarkFailure.error, error], "Benchmark and cleanup failed"); + throw error; + } +} diff --git a/server/src/__tests__/helpers/chat-reply-lineage.ts b/server/src/__tests__/helpers/chat-reply-lineage.ts new file mode 100644 index 00000000000..5a5b65baedc --- /dev/null +++ b/server/src/__tests__/helpers/chat-reply-lineage.ts @@ -0,0 +1,67 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { agents, agentWakeupRequests, chatConversations, chatEndpoints, chatMessageLinks, companies, heartbeatRuns, issueComments, issues, issueThreadInteractions, toolApplications, toolConnections, toolActionRequests, toolActionDeliveries, toolInvocations, type Db } from "@paperclipai/db"; +import { resolveChatOriginPublicationBindings } from "../../services/issues.js"; + +export const lineageContextKeys = ["issueId", "taskId", "wakeCommentIds", "wakeCommentId", "commentId", "source", "sourceRunId", "interactionId", "endpointId"] as const; +export type LineageResolver = typeof resolveChatOriginPublicationBindings; +export type LineageQuery = { sql: string; params: unknown[]; rows: Array> }; + +export function captureLineageReads(db: Db, queries: LineageQuery[]): Db { + return new Proxy(db, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const query = from(...args), execute = query.execute.bind(query); + if (args[0] === heartbeatRuns) query.execute = async (...executeArgs: Parameters) => { + const statement = query.toSQL(); + const rows = await execute(...executeArgs); + if (fields[0] && Object.keys(fields[0]).join(",") === "agentId,responsibleUserId,contextSnapshot") queries.push({ ...statement, rows }); + return rows; + }; + return query; + }) as typeof builder.from; + return builder; + }; + } }); +} + +export function expectedLineageContext(value: unknown): unknown { + if (value === null || typeof value !== "object" || Array.isArray(value)) return value; + return Object.fromEntries(lineageContextKeys.filter(key => Object.hasOwn(value, key)).map(key => [key, (value as Record)[key]])); +} + +export async function chatReplyLineageFixture(db: Db, bytes = 192 * 1024) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Synthetic reply lineage", issuePrefix: `L${companyId.replaceAll("-", "").slice(0, 7)}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Synthetic worker", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Synthetic external reply", status: "in_progress", assigneeAgentId: agentId }); + const [application] = await db.insert(toolApplications).values({ companyId, name: "Synthetic chat", applicationKey: randomUUID(), type: "chat" }).returning(); + const [connection] = await db.insert(toolConnections).values({ companyId, applicationId: application.id, name: "Synthetic chat", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel" }).returning(); + const [endpoint] = await db.insert(chatEndpoints).values({ companyId, connectionId: connection.id, assignedAgentId: agentId, provider: "slack", publicId: randomUUID(), status: "active" }).returning(); + const [conversation] = await db.insert(chatConversations).values({ companyId, endpointId: endpoint.id, issueId, externalConversationId: randomUUID(), externalThreadId: randomUUID(), externalLabel: "Synthetic conversation" }).returning(); + const [comment] = await db.insert(issueComments).values({ companyId, issueId, authorType: "user", authorUserId: "synthetic-requester", body: "Please continue" }).returning(); + await db.insert(chatMessageLinks).values({ companyId, endpointId: endpoint.id, conversationId: conversation.id, commentId: comment.id, providerMessageId: randomUUID(), direction: "inbound" }); + const context = { source: "chat:slack", issueId, endpointId: endpoint.id, wakeCommentIds: [comment.id], prompt: randomBytes(bytes).toString("base64") }; + const insertRun = async (snapshot: unknown = context, overrides: Partial = {}) => { + const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "succeeded", contextSnapshot: snapshot as never, ...overrides }).returning(); + return run; + }; + const root = await insertRun(); + async function continueFrom(sourceRunId: string) { + const [interaction] = await db.insert(issueThreadInteractions).values({ companyId, issueId, sourceRunId, kind: "request_confirmation", status: "accepted", payload: { version: 1 } as never }).returning(); + const run = await insertRun({ issueId, source: "issue.interaction.accept", sourceRunId, interactionId: interaction.id, prompt: context.prompt }); + const [wake] = await db.insert(agentWakeupRequests).values({ companyId, agentId, source: "automation", reason: "issue_commented", status: "claimed", runId: run.id, idempotencyKey: `interaction:${interaction.id}:accepted` }).returning(); + return { run, interaction, wake }; + } + return { companyId, agentId, issueId, endpoint, conversation, comment, context, root, insertRun, continueFrom, + expected: [{ companyId, endpointId: endpoint.id, conversationId: conversation.id }] }; +} + +export async function cleanupChatReplyLineageFixture(db: Db, companyId: string) { + for (const table of [agentWakeupRequests, toolActionDeliveries, toolActionRequests, toolInvocations, issueThreadInteractions, chatMessageLinks, chatConversations, chatEndpoints, heartbeatRuns, issueComments, issues, toolConnections, toolApplications, agents]) { + await db.delete(table).where(eq(table.companyId, companyId)); + } + await db.delete(companies).where(eq(companies.id, companyId)); +} diff --git a/server/src/services/chat-reply-lineage-context.test.ts b/server/src/services/chat-reply-lineage-context.test.ts new file mode 100644 index 00000000000..41ad66d3f3e --- /dev/null +++ b/server/src/services/chat-reply-lineage-context.test.ts @@ -0,0 +1,111 @@ +import { randomUUID } from "node:crypto"; +import { eq } from "drizzle-orm"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { applyPendingMigrations, createDb, heartbeatRuns, agentWakeupRequests, toolInvocations, toolActionRequests, toolActionDeliveries } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js"; +import { captureLineageReads, chatReplyLineageFixture, cleanupChatReplyLineageFixture, expectedLineageContext, type LineageQuery } from "../__tests__/helpers/chat-reply-lineage.js"; +import { resolveChatOriginPublicationBindings } from "./issues.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("chat reply lineage context projection", () => { + let db: ReturnType; + let temporary: Awaited> | undefined; + const companyIds: string[] = []; + async function fixture(bytes?: number) { const f = await chatReplyLineageFixture(db, bytes); companyIds.push(f.companyId); return f; } + beforeAll(async () => { + if (externalUrl) await applyPendingMigrations(externalUrl); + else temporary = await startEmbeddedPostgresTestDatabase("chat-lineage-projection-"); + db = createDb(externalUrl ?? temporary!.connectionString); + }, 90_000); + afterAll(async () => { + try { + if (db) for (const companyId of companyIds) await cleanupChatReplyLineageFixture(db, companyId); + } finally { + try { await db?.$client.end({ timeout: 0 }); } + finally { await temporary?.cleanup(); } + } + }); + + it("projects the complete routing context before a wide body crosses PostgreSQL", async () => { + const f = await fixture(), reads: LineageQuery[] = []; + expect(await resolveChatOriginPublicationBindings(captureLineageReads(db, reads), f.companyId, f.issueId, f.root.id)).toEqual(f.expected); + expect(reads).toHaveLength(1); + const rows = await db.$client.unsafe(reads[0].sql, reads[0].params as never[]).raw(); + expect(rows.reduce((total, row) => total + row.reduce((n: number, value: Buffer | null) => n + (value?.length ?? 0), 0), 0)).toBeLessThan(1024); + expect(reads[0].rows).toEqual([{ agentId: f.agentId, responsibleUserId: null, contextSnapshot: expectedLineageContext(f.context) }]); + }); + + it.each([null, false, true, 0, 12, "", "malformed", [], ["source", { source: "chat:slack" }], {}, + { issueId: null, taskId: false, source: 3, interactionId: null, wakeCommentIds: { id: "not-array" } }, + ])("preserves malformed JSON and key absence for %j", async snapshot => { + const f = await fixture(1), run = await f.insertRun(snapshot), reads: LineageQuery[] = []; + expect(await resolveChatOriginPublicationBindings(captureLineageReads(db, reads), f.companyId, f.issueId, run.id)).toEqual([]); + expect(reads[0].rows[0].contextSnapshot).toEqual(expectedLineageContext(snapshot)); + }); + + it("keeps all nine keys and their JSON values, including explicit nulls and arrays", async () => { + const f = await fixture(1); + const snapshot = { issueId: null, taskId: f.issueId, wakeCommentIds: [f.comment.id, 1, null, f.comment.id], wakeCommentId: f.comment.id, commentId: f.comment.id, + source: "chat:slack", sourceRunId: { invalid: true }, interactionId: null, endpointId: f.endpoint.id, omitted: "wide unrelated value" }; + const run = await f.insertRun(snapshot), reads: LineageQuery[] = []; + expect(await resolveChatOriginPublicationBindings(captureLineageReads(db, reads), f.companyId, f.issueId, run.id)).toEqual(f.expected); + expect(reads[0].rows[0].contextSnapshot).toEqual(expectedLineageContext(snapshot)); + expect(Object.hasOwn(reads[0].rows[0].contextSnapshot as object, "interactionId")).toBe(true); + expect(Object.hasOwn(reads[0].rows[0].contextSnapshot as object, "omitted")).toBe(false); + }); + + it("preserves explicit JSON null as well as SQL null", async () => { + const f = await fixture(1), run = await f.insertRun(null), reads: LineageQuery[] = []; + await db.$client.unsafe("update heartbeat_runs set context_snapshot = 'null'::jsonb where id = $1::uuid", [run.id]); + expect(await resolveChatOriginPublicationBindings(captureLineageReads(db, reads), f.companyId, f.issueId, run.id)).toEqual([]); + expect(reads[0].rows[0].contextSnapshot).toBeNull(); + }); + + it.each([ + { name: "missing", snapshot: {}, wake: {}, allowed: true }, + { name: "explicit null", snapshot: { interactionId: null }, wake: { interactionId: null }, allowed: true }, + { name: "missing versus null", snapshot: { interactionId: null }, wake: {}, allowed: false }, + { name: "equal strings", snapshot: { interactionId: "synthetic-interaction" }, wake: { interactionId: "synthetic-interaction" }, allowed: true }, + { name: "number versus text", snapshot: { interactionId: 12 }, wake: { interactionId: "12" }, allowed: false }, + { name: "separate objects", snapshot: { interactionId: { id: "same" } }, wake: { interactionId: { id: "same" } }, allowed: false }, + ])("retains strict tool-action interaction semantics for $name and requires the durable receipt", async ({ snapshot, wake, allowed }) => { + const f = await fixture(1), continued = await f.continueFrom(f.root.id); + const [invocation] = await db.insert(toolInvocations).values({ companyId: f.companyId, issueId: f.issueId, agentId: f.agentId, runId: f.root.id, toolName: "synthetic-operation" }).returning(); + const [request] = await db.insert(toolActionRequests).values({ companyId: f.companyId, issueId: f.issueId, invocationId: invocation.id, + interactionId: continued.interaction.id, requestedByAgentId: f.agentId, status: "executed", canonicalArgumentsHash: "synthetic", canonicalArgumentsSummary: { summary: "Synthetic operation" } }).returning(); + await db.insert(toolActionDeliveries).values({ companyId: f.companyId, issueId: f.issueId, actionRequestId: request.id, interactionId: continued.interaction.id }); + await db.update(heartbeatRuns).set({ contextSnapshot: { issueId: f.issueId, source: "tool_action_review", sourceRunId: f.root.id, ...snapshot } }).where(eq(heartbeatRuns.id, continued.run.id)); + await db.insert(agentWakeupRequests).values({ companyId: f.companyId, agentId: f.agentId, source: "automation", reason: "issue_commented", status: "claimed", runId: continued.run.id, + idempotencyKey: `tool-action-response:${request.id}`, payload: { sourceRunId: f.root.id, toolActionRequestIds: [request.id], ...wake } }); + expect(await resolveChatOriginPublicationBindings(db, f.companyId, f.issueId, continued.run.id)).toEqual(allowed ? f.expected : []); + await db.delete(toolActionDeliveries).where(eq(toolActionDeliveries.actionRequestId, request.id)); + expect(await resolveChatOriginPublicationBindings(db, f.companyId, f.issueId, continued.run.id)).toEqual([]); + }); + + it("retains valid lineage beyond sixteen hops and rejects a durable cycle", async () => { + const f = await fixture(1); + let previous = f.root.id; + for (let index = 0; index < 24; index++) previous = (await f.continueFrom(previous)).run.id; + expect(await resolveChatOriginPublicationBindings(db, f.companyId, f.issueId, previous)).toEqual(f.expected); + const cycle = await f.continueFrom(previous); + await db.update(heartbeatRuns).set({ contextSnapshot: { source: "issue.interaction.accept", issueId: f.issueId, interactionId: cycle.interaction.id, sourceRunId: previous } }).where(eq(heartbeatRuns.id, f.root.id)); + await db.update(agentWakeupRequests).set({ runId: f.root.id }).where(eq(agentWakeupRequests.id, cycle.wake.id)); + const reads: LineageQuery[] = []; + expect(await resolveChatOriginPublicationBindings(captureLineageReads(db, reads), f.companyId, f.issueId, previous)).toEqual([]); + expect(reads).toHaveLength(25); + }); + + it("rejects mismatched company, task, agent, and endpoint bindings", async () => { + const f = await fixture(1), other = await fixture(1); + expect(await resolveChatOriginPublicationBindings(db, other.companyId, f.issueId, f.root.id)).toEqual([]); + for (const change of [{ issueId: other.issueId }, { endpointId: other.endpoint.id }, { source: false }, { wakeCommentIds: [randomUUID()], wakeCommentId: 12 }]) { + const run = await f.insertRun({ ...f.context, ...change }); + expect(await resolveChatOriginPublicationBindings(db, f.companyId, f.issueId, run.id)).toEqual([]); + } + const continued = await f.continueFrom(f.root.id); + await db.update(heartbeatRuns).set({ agentId: other.agentId }).where(eq(heartbeatRuns.id, continued.run.id)); + expect(await resolveChatOriginPublicationBindings(db, f.companyId, f.issueId, continued.run.id)).toEqual([]); + }); +}); diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index c6e54798555..391d08c660f 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1170,7 +1170,13 @@ export async function resolveChatOriginPublicationBindings( .select({ agentId: heartbeatRuns.agentId, responsibleUserId: heartbeatRuns.responsibleUserId, - contextSnapshot: heartbeatRuns.contextSnapshot, + contextSnapshot: sql | null>` + case when jsonb_typeof(${heartbeatRuns.contextSnapshot}) = 'object' then + (select coalesce(jsonb_object_agg(entry.key, entry.value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.contextSnapshot}) as entry + where entry.key in ('issueId', 'taskId', 'wakeCommentIds', 'wakeCommentId', + 'commentId', 'source', 'sourceRunId', 'interactionId', 'endpointId')) + else ${heartbeatRuns.contextSnapshot} end`, }) .from(heartbeatRuns) .where( From dc8cbdb81953bd295ca2ab635414412af729e458 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:31:55 -0500 Subject: [PATCH 13/41] perf(server): narrow native session resume database reads (#17934) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/94/changes/paperclip/94 ## Thinking Path > - Paperclip manages AI agents and their work. > - Native agents resume prior provider sessions from saved run checkpoints. > - Resume selection must respect every newer progress barrier. > - The prior query returned large context and profile JSON for up to 32 candidates. > - Most candidates need only process, status, binding, and checkpoint-presence checks. > - This PR selects the same source in one SQL snapshot and returns only its required payloads. > - This reduces database response bytes while preserving session authority and saved task context. ## Linked Issues or Issue Description **What existing behavior does this improve?** Native task/session resume selection in `findNativeSessionResumeRun`. **Current behavior** At baseline `bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e`, a resume lookup reads complete context and profile JSON for up to 32 runs, then reads provider-event evidence. A synthetic 15-run history returns about 4.44 MB even though one checkpoint is selected. **Proposed behavior** A bounded CTE finds the first checkpoint or progress barrier. The same SQL statement returns the selected source's native input, checkpoint, tool fingerprint, wake envelope, canonical task brief, and process identity. All fallback barriers and the 32-run limit stay in place. **Reason and benefit** Rejected candidates no longer transfer saved prompts and execution inputs to Node. A single snapshot also prevents a gap between candidate selection and payload hydration. **Breaking changes** None to public APIs or stored history. This private service returns the fields consumed by runtime selection and checkpoint rebinding. It keeps complete required nested values and distinguishes missing brief fields from explicit JSON nulls. **Additional context** Checked `ROADMAP.md`. This is a targeted improvement to existing resume behavior. Public searches for native session resume projections found no duplicate. Related work includes #6573 (durable agent sessions) and #17011 (managed Claude resume opt-in); neither changes this database read. Provider-refusal recovery in #17676 is a separate concern and is unchanged. ## What Changed - Select the newest progress barrier within the existing exact company/agent/task/session and 32-run window in one SQL snapshot. - Return three required profile fields, the full wake envelope, and the saved task's ID, title, and description only for the selected source. - Preserve invalid non-null checkpoint barriers, provider evidence, process authority, terminal-status rules, timestamp precision, and exact string binding checks. - Add real PostgreSQL regression coverage and a reproducible baseline/treatment benchmark. Document the read and benchmark command. ## Verification - `pnpm -r typecheck` passed. `pnpm build` passed. The final server `tsc --noEmit` also passed. The later benchmark-only relocation passed `pnpm check:module-boundaries` and the real PostgreSQL proof; runtime and regression-test source are unchanged. - Related suites: `native-session-resume-projection.test.ts`, `native-session-resume.test.ts`, `heartbeat/retries.test.ts`, and `heartbeat-retry-scheduling.test.ts`: 78 passed and 54 skipped across 3 passed files and 1 skipped file. Embedded PostgreSQL/platform gates caused the skips. Resume orchestration and the 10 new SQL tests ran against isolated PostgreSQL 17.11. The final 10-test projection rerun passed. - New SQL cases cover all seven provider evidence kinds, each process authority field, all terminal statuses, wrong bindings and runtime, invalid checkpoints, null/scalar/array/object JSON, missing versus null brief fields, the 32-row bound, ordering ties, future barriers, submillisecond timestamps, and a change committed before the one selection statement. - `pnpm test:run` was attempted. The host's embedded PostgreSQL startup failed after five attempts. The broad run recorded failures/skips in embedded-database suites and was stopped after that concrete host limitation (exit 143). It is not a local full-suite pass. The full final-head CI matrix passes on `dcde4e3dac2173321d7be2406e7775af8411826a`: 55 successful checks, one neutral check, and two optional Storybook skips. Greptile completed at 5/5 on that exact head, with zero unresolved findings. The initial benchmark-location policy failure was fixed before this final run. - Reproduce the benchmark with `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-native-session-resume-reads.ts --baseline-ref=bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e`. The optional `PAPERCLIP_SESSION_RESUME_BENCHMARK_DATABASE_URL` must name an isolated database ending in `_test`. - The harness imports the pinned baseline module, invokes both actual services, captures complete generated SQL and parameters, compares every retained consumer field, and captures `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)`. Its incompressible wide fixture keeps a complete 96 KiB native input in the selected result. Raw PostgreSQL text DataRow frames are measured exactly; RowDescription, protocol command metadata, TLS, and TCP are excluded. | Synthetic local case | Baseline DataRow bytes | Treatment DataRow bytes | Reduction | | --- | ---: | ---: | ---: | | 15 runs, one selected checkpoint | 4,441,143 | 104,599 | 97.64% | | 15 unused bootstraps | 4,435,590 | 0 | 100% | | One small selected source | 6,395 | 6,319 | 1.19% | The committed-head benchmark uses two warmups and seven alternating samples per variant. Local round-trip medians and ranges were: | Synthetic local case | Baseline median (range), ms | Treatment median (range), ms | | --- | ---: | ---: | | Wide history | 20.32 (18.63–24.37) | 1.63 (1.40–2.21) | | Unused history | 18.52 (17.63–20.35) | 1.20 (1.13–1.56) | | Small selected source | 1.05 (0.95–1.70) | 0.78 (0.73–1.30) | The small case has overlapping ranges and a 1.19% byte change, so it is neutral. `EXPLAIN` server execution time was higher for the treatment (wide: 0.402 ms versus 0.028 ms across the two baseline statements). `EXPLAIN` does not serialize result rows; the new JSON presence checks do extra server work while avoiding large result transfer. These are local synthetic round-trip measurements, not production latency claims. The final proof ran on `dcde4e3dac2173321d7be2406e7775af8411826a` with no runtime source diff. ## Risks - Selection policy now has a SQL form. Regression tests preserve the pure selector's barrier semantics, including invalid non-null checkpoints and JavaScript Date precision. - Checkpoint-presence checks can still detoast candidate profiles inside PostgreSQL. The change reduces returned bytes; it does not eliminate all database JSON work. - The selected checkpoint, native input, and wake envelope can still be large because consumers require them in full. No new payload cap is introduced. - There is no migration, cache, background polling, ownership lock change, deployment, or merge in this PR. ## Model Used OpenAI GPT-6 through Codex. The host did not expose an exact runtime model ID or context-window size. The agent used reasoning, code execution, real PostgreSQL, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DATABASE.md | 12 ++ .../benchmark-native-session-resume-reads.ts | 127 +++++++++++ .../native-session-resume-projection.test.ts | 203 ++++++++++++++++++ .../native-runtime/native-session-resume.ts | 119 +++++++--- 4 files changed, 429 insertions(+), 32 deletions(-) create mode 100644 scripts/benchmark-native-session-resume-reads.ts create mode 100644 server/src/services/native-runtime/native-session-resume-projection.test.ts diff --git a/doc/DATABASE.md b/doc/DATABASE.md index 24e06d59830..bf9f4a03333 100644 --- a/doc/DATABASE.md +++ b/doc/DATABASE.md @@ -49,6 +49,18 @@ and leaves all existing cases passive. Source identities are not grants. ### Task query performance +Native session resume selects its newest progress barrier within the existing +32-run task/session window in one SQL snapshot. Only the chosen checkpoint source +returns its saved native input, checkpoint, tool fingerprint, wake envelope, and +canonical task brief. Unused bootstrap candidates do not return full context or +profiles. Wrong runtime, nonterminal status, process identity, provider events, +invalid but non-null checkpoints, and future checkpoint barriers retain their +existing selection rules. Stored run payloads and history reads stay complete. +Run the synthetic PostgreSQL comparison with a pinned pre-change revision: +`pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-native-session-resume-reads.ts --baseline-ref=`. +An optional `PAPERCLIP_SESSION_RESUME_BENCHMARK_DATABASE_URL` must name a dedicated +test database ending in `_test`; otherwise the harness starts disposable PostgreSQL. + Agent request authentication uses a compact run-identity read. It omits saved execution payloads, results, and log excerpts from the query response. The read still locks the task before the run and reconciles pending steering from durable diff --git a/scripts/benchmark-native-session-resume-reads.ts b/scripts/benchmark-native-session-resume-reads.ts new file mode 100644 index 00000000000..df302f1ba71 --- /dev/null +++ b/scripts/benchmark-native-session-resume-reads.ts @@ -0,0 +1,127 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { randomBytes, randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { agents, applyPendingMigrations, companies, createDb, heartbeatRuns, issues, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js"; +import { findNativeSessionResumeRun } from "../server/src/services/native-runtime/native-session-resume.js"; +import { resolveExternalTestDatabaseUrl } from "../server/src/__tests__/helpers/external-test-database.js"; + +type Query = { sql: string; params: unknown[] }; +const root = fileURLToPath(new URL("../", import.meta.url)); +const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15); +assert.ok(baselineRef, "Pass a pinned pre-fix --baseline-ref"); +const baselineRevision = execFileSync("git", ["rev-parse", "--verify", `${baselineRef}^{commit}`], { cwd: root, encoding: "utf8" }).trim(); +const sourcePath = "server/src/services/native-runtime/native-session-resume.ts"; +const treatmentRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: root, encoding: "utf8" }).trim(); +const treatmentDiff = execFileSync("git", ["diff", "HEAD", "--", sourcePath], { cwd: root, encoding: "utf8" }); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_SESSION_RESUME_BENCHMARK_DATABASE_URL"); +const database = externalUrl ? { connectionString: externalUrl, cleanup: async () => {} } : await startEmbeddedPostgresTestDatabase("native-session-resume-benchmark-"); +const db = createDb(database.connectionString); +const baselineDir = await fs.mkdtemp(path.join(root, "server/scripts/.session-resume-baseline-")); +const companyIds: string[] = []; +try { + if (externalUrl) await applyPendingMigrations(externalUrl); + const moduleUrl = new URL(`../${sourcePath}`, import.meta.url); + const source = execFileSync("git", ["show", `${baselineRevision}:${sourcePath}`], { cwd: root, encoding: "utf8" }); + await fs.writeFile(path.join(baselineDir, "resume.ts"), source.replace(/from "(\.{1,2}\/[^\"]+)"/g, + (_match, specifier: string) => `from ${JSON.stringify(fileURLToPath(new URL(specifier, moduleUrl)))}`)); + const baseline = await import(pathToFileURL(path.join(baselineDir, "resume.ts")).href) as { findNativeSessionResumeRun: typeof findNativeSessionResumeRun }; + async function capture(invoke: typeof findNativeSessionResumeRun, input: Parameters[1]) { + const queries: Query[] = []; + const wrap = (target: Db): Db => new Proxy(target, { get(target, key) { + if (key === "with") return (...args: Parameters) => wrap(target.with(...args) as Db); + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const query = from(...args), execute = query.execute.bind(query); + query.execute = async (...executeArgs: Parameters) => { + queries.push(query.toSQL()); + return execute(...executeArgs); + }; + return query; + }) as typeof builder.from; + return builder; + }; + }}); + return { result: await invoke(wrap(db), input), queries }; + } + const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]; + const measurements = []; + for (const scenario of ["wide_history", "unused_history", "small_selected"] as const) { + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), normalizedSessionId = randomUUID(); + companyIds.push(companyId); + await db.insert(companies).values({ id: companyId, name: "Synthetic resume benchmark", issuePrefix: `B${companyId.replaceAll("-", "")}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Synthetic runner", adapterType: "paperclip_runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Retain canonical task" }); + const runCount = scenario === "small_selected" ? 1 : 15; + const prompt = randomBytes(scenario === "small_selected" ? 16 : 144 * 1024).toString("base64"); + const profilePadding = randomBytes(scenario === "small_selected" ? 16 : 72 * 1024).toString("base64"); + const neededInput = { task: { title: "Prior task", description: "Canonical description", prompt: profilePadding }, runtimeContext: { instructions: "Keep complete input" } }; + const checkpoint = { sessionId: normalizedSessionId, state: randomBytes(4096).toString("base64") }; + await db.insert(heartbeatRuns).values(Array.from({ length: runCount }, (_, n) => ({ + id: randomUUID(), companyId, agentId, nativeIssueId: issueId, nativeSessionId: normalizedSessionId, + runtimeMode: "native", status: "failed", createdAt: new Date(Date.UTC(2026, 8, 8, 0, 0, n)), + runnerProfileJson: { padding: "Unused profile metadata", nativeExecutionInput: neededInput, nativeToolContractFingerprint: "fixture", + ...(scenario !== "unused_history" && n === 0 ? { sessionCheckpoint: checkpoint } : {}) }, + contextSnapshot: { prompt, paperclipWake: { externalChatProvider: "slack", githubReviewConfigurations: [{ key: "review" }] }, + paperclipIssue: { id: issueId, title: "Prior canonical title", description: "Prior canonical description" } }, + }))); + await db.$client.unsafe("ANALYZE heartbeat_runs"); + const input = { companyId, agentId, issueId, normalizedSessionId, currentRunId: randomUUID(), beforeCreatedAt: new Date("2026-09-09T00:00:00Z") }; + const before = await capture(baseline.findNativeSessionResumeRun, input), after = await capture(findNativeSessionResumeRun, input); + const contract = (run: typeof after.result) => run && ({ ...run, + runnerProfileJson: { nativeExecutionInput: run.runnerProfileJson?.nativeExecutionInput ?? null, + nativeToolContractFingerprint: run.runnerProfileJson?.nativeToolContractFingerprint ?? null, + sessionCheckpoint: run.runnerProfileJson?.sessionCheckpoint ?? null }, + contextSnapshot: { paperclipWake: run.contextSnapshot?.paperclipWake ?? null, paperclipIssue: run.contextSnapshot?.paperclipIssue ?? null } }); + assert.deepEqual(contract(after.result), contract(before.result)); + const variants = [before.queries, after.queries], samples: number[][] = [[], []], dataRowBytes = [0, 0], resultRows = [0, 0]; + for (let iteration = 0; iteration < 9; iteration++) { + for (const variant of iteration % 2 ? [1, 0] : [0, 1]) { + const start = performance.now(); + let bytes = 0, count = 0; + for (const query of variants[variant]) { + const rows = await db.$client.unsafe(query.sql, query.params as never[]).raw(); + count += rows.length; + bytes += rows.reduce((total, row) => total + 7 + row.reduce((size: number, value: Buffer | null) => size + 4 + (value?.length ?? 0), 0), 0); + } + if (iteration >= 2) samples[variant].push(performance.now() - start); + dataRowBytes[variant] = bytes; resultRows[variant] = count; + } + } + const plans = []; + for (const queries of variants) { + const variantPlans = []; + for (const query of queries) { + const [plan] = await db.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]); + variantPlans.push(plan["QUERY PLAN"]); + } + plans.push(variantPlans); + } + if (!process.argv.includes("--baseline-only") && scenario !== "small_selected") assert.ok(dataRowBytes[1] < dataRowBytes[0] / 10); + measurements.push({ scenario, runCount, equalConsumerFields: true, baselineQueryCount: before.queries.length, treatmentQueryCount: after.queries.length, + dataRowBytes, resultRows, baselineMedianMs: median(samples[0]), treatmentMedianMs: median(samples[1]), samples, queries: variants, plans }); + } + console.log(JSON.stringify({ baselineRevision, treatmentRevision, treatmentDiff, + acceptance: "Equal selected consumer fields; wide and unused histories remove at least 90% of DataRow bytes; no timing threshold", + byteBoundary: "Exact text-format PostgreSQL DataRow frames (7 bytes framing + 4 bytes length per column + raw field bytes); excludes RowDescription, command metadata, TLS and TCP", + postgres: await db.$client.unsafe("select version()"), measurements }, null, 2)); +} finally { + try { + for (const companyId of companyIds) { + await db.$client.unsafe("DELETE FROM heartbeat_runs WHERE company_id = $1::uuid", [companyId]); + await db.$client.unsafe("DELETE FROM issues WHERE company_id = $1::uuid", [companyId]); + await db.$client.unsafe("DELETE FROM agents WHERE company_id = $1::uuid", [companyId]); + await db.$client.unsafe("DELETE FROM companies WHERE id = $1::uuid", [companyId]); + } + } finally { + try { await db.$client.end({ timeout: 0 }); } + finally { + try { await database.cleanup(); } + finally { await fs.rm(baselineDir, { recursive: true, force: true }); } + } + } +} diff --git a/server/src/services/native-runtime/native-session-resume-projection.test.ts b/server/src/services/native-runtime/native-session-resume-projection.test.ts new file mode 100644 index 00000000000..08242f2a690 --- /dev/null +++ b/server/src/services/native-runtime/native-session-resume-projection.test.ts @@ -0,0 +1,203 @@ +import { randomUUID } from "node:crypto"; +import { and, eq, sql } from "drizzle-orm"; +import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js"; +import { findNativeSessionResumeRun } from "./native-session-resume.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL"); +const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport()).supported; + +(supported ? describe : describe.skip)("native session resume SQL projection", () => { + let database: Awaited>; + let db: Db; + const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), normalizedSessionId = randomUUID(); + const currentRunId = randomUUID(); + const at = (second: number) => new Date(Date.UTC(2026, 8, 8, 0, 0, second)); + const input = { companyId, agentId, issueId, normalizedSessionId, currentRunId, beforeCreatedAt: at(30) }; + const checkpoint = { sessionId: "provider-session", identity: { runId: "prior" }, complete: [false, 0, "", null] }; + const nativeExecutionInput = { binding: { issueId }, task: { prompt: "Complete native input", description: "Description" }, runtimeContext: { nested: [1, 2] } }; + const paperclipWake = { externalChatProvider: "slack", githubReviewConfigurations: [{ key: "configuration", nested: { retained: true } }], other: ["preserved"] }; + const paperclipIssue = { id: issueId, title: "Saved canonical title", description: "Saved canonical description" }; + const profile = { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint", unrelated: "Omit" }; + const run = (second: number, overrides: Partial = {}) => ({ + id: randomUUID(), companyId, agentId, nativeIssueId: issueId, nativeSessionId: normalizedSessionId, + runtimeMode: "native", status: "failed", createdAt: at(second), runnerProfileJson: {}, ...overrides, + }); + const select = () => findNativeSessionResumeRun(db, input); + const source = () => run(0, { runnerInstanceId: randomUUID(), processPid: 17, processGroupId: 19, processStartedAt: at(0), + runnerProfileJson: profile, contextSnapshot: { paperclipWake, paperclipIssue: { ...paperclipIssue, unrelated: "Omit" }, prompt: "Omit" } }); + const writeProfile = (id: string, value: unknown) => db.execute(sql`update ${heartbeatRuns} + set runner_profile_json = ${JSON.stringify(value)}::jsonb where ${heartbeatRuns.id} = ${id}`); + + beforeAll(async () => { + database = externalUrl ? { connectionString: externalUrl, cleanup: async () => {} } : await startEmbeddedPostgresTestDatabase("native-resume-projection-"); + db = createDb(database.connectionString); + await db.insert(companies).values({ id: companyId, name: "Resume projection", issuePrefix: `N${companyId.replaceAll("-", "")}` }); + await db.insert(agents).values({ id: agentId, companyId, name: "Runner" }); + await db.insert(issues).values({ id: issueId, companyId, title: "Resume task" }); + }); + afterEach(async () => { + await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId)); + }); + afterAll(async () => { + if (!db) return; + try { + await db.delete(issues).where(eq(issues.companyId, companyId)); + await db.delete(agents).where(eq(agents.companyId, companyId)); + await db.delete(companies).where(eq(companies.id, companyId)); + } finally { + try { await db.$client.end({ timeout: 0 }); } + finally { await database.cleanup(); } + } + }); + + it("retains full required input, checkpoint, wake, canonical brief and process identity only for the selected source", async () => { + const saved = source(); + await db.insert(heartbeatRuns).values([saved, run(1), run(2)]); + const result = await select(); + expect(result).toEqual({ ...saved, runnerProfileJson: { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint" }, + contextSnapshot: { paperclipWake, paperclipIssue } }); + const stored = await db.select({ profile: heartbeatRuns.runnerProfileJson, context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.id)); + expect(stored[0].profile).toEqual(profile); + expect(stored[0].context).toEqual(saved.contextSnapshot); + for (const status of ["succeeded", "interrupted", "cancelled", "timed_out"]) { + await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, saved.id)); + expect((await select())?.id).toBe(saved.id); + } + }); + + it("treats every non-null checkpoint as a barrier, including invalid false, zero, empty string, array and object values", async () => { + const saved = source(), newer = run(1); + await db.insert(heartbeatRuns).values([saved, newer]); + for (const value of [false, 0, "", "invalid", [], {}]) { + await writeProfile(newer.id, { sessionCheckpoint: value }); + const selected = await select(); + expect(selected?.id).toBe(newer.id); + expect(selected?.runnerProfileJson.sessionCheckpoint).toEqual(value); + } + }); + + it("skips only unused SQL-null, JSON-null, scalar, array and checkpoint-null profiles", async () => { + const saved = source(), newer = run(1, { runnerProfileJson: null }); + await db.insert(heartbeatRuns).values([saved, newer]); + expect((await select())?.id).toBe(saved.id); + for (const value of [null, false, 0, "sessionCheckpoint", [], ["sessionCheckpoint"], {}, { sessionCheckpoint: null }]) { + await writeProfile(newer.id, value); + expect((await select())?.id).toBe(saved.id); + } + }); + + it("preserves malformed selected context values and missing brief fields", async () => { + const saved = source(); + await db.insert(heartbeatRuns).values(saved); + for (const value of [null, false, 0, "invalid", [], {}, { paperclipWake: [], paperclipIssue: { id: issueId, title: false } }, + { paperclipIssue: { id: issueId, title: "Valid title without description" } }, + { paperclipIssue: { id: issueId, title: "Valid title", description: null } }]) { + await db.execute(sql`update ${heartbeatRuns} set context_snapshot = ${JSON.stringify(value)}::jsonb where ${heartbeatRuns.id} = ${saved.id}`); + const result = await select(); + const object = typeof value === "object" && value !== null && !Array.isArray(value) ? value : {}; + expect(result?.contextSnapshot.paperclipWake).toEqual("paperclipWake" in object ? object.paperclipWake : null); + expect(result?.contextSnapshot.paperclipIssue).toEqual("paperclipIssue" in object ? object.paperclipIssue : null); + } + }); + + it("stops at wrong runtime, nonterminal status and each process authority field", async () => { + await db.insert(heartbeatRuns).values(source()); + for (const override of [{ runtimeMode: "legacy" }, { status: "running" }, { processPid: 0 }, { processGroupId: 0 }, { processStartedAt: at(0) }]) { + const newer = run(1, override); + await db.insert(heartbeatRuns).values(newer); + expect(await select()).toBeNull(); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, newer.id)); + } + }); + + it("checks every provider evidence event and ignores unrelated event kinds", async () => { + const saved = source(), newer = run(1); + await db.insert(heartbeatRuns).values([saved, newer]); + for (const eventType of ["harness.ready", "session.started", "session.resumed", "session.updated", "turn.started", "provider.event", "provider.rpc_result", "log"]) { + await db.insert(heartbeatRunEvents).values({ companyId, agentId, runId: newer.id, seq: 1, eventType }); + if (eventType === "log") expect((await select())?.id).toBe(saved.id); + else expect(await select()).toBeNull(); + await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, newer.id)); + } + }); + + it("excludes current and mismatched agent, company, task and session rows before applying barriers", async () => { + const saved = source(), otherCompany = randomUUID(), otherAgent = randomUUID(), otherIssue = randomUUID(); + await db.insert(companies).values({ id: otherCompany, name: "Other", issuePrefix: `D${otherCompany.replaceAll("-", "")}` }); + await db.insert(agents).values({ id: otherAgent, companyId: otherCompany, name: "Other" }); + await db.insert(issues).values({ id: otherIssue, companyId, title: "Other" }); + try { + await db.insert(heartbeatRuns).values([saved, run(1, { id: currentRunId, status: "running" }), + run(2, { companyId: otherCompany, status: "running" }), run(3, { agentId: otherAgent, status: "running" }), + run(4, { nativeIssueId: otherIssue, status: "running" }), run(5, { nativeSessionId: randomUUID(), status: "running" })]); + expect((await select())?.id).toBe(saved.id); + } finally { + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, otherCompany)); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.agentId, otherAgent)); + await db.delete(issues).where(eq(issues.id, otherIssue)); + await db.delete(agents).where(eq(agents.id, otherAgent)); + await db.delete(companies).where(eq(companies.id, otherCompany)); + } + expect(await findNativeSessionResumeRun(db, { ...input, normalizedSessionId: "invalid" })).toBeNull(); + for (const key of ["companyId", "agentId", "issueId", "normalizedSessionId"] as const) { + expect(await findNativeSessionResumeRun(db, { ...input, [key]: input[key].toUpperCase() })).toBeNull(); + } + }); + + it("retains the 32-row bound and deterministic created-at/id ordering", async () => { + const saved = source(), unused = Array.from({ length: 32 }, () => run(1)); + await db.insert(heartbeatRuns).values([saved, ...unused]); + expect(await select()).toBeNull(); + await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, unused[0].id)); + expect((await select())?.id).toBe(saved.id); + const tied = [run(2, { runnerProfileJson: { sessionCheckpoint: false } }), run(2, { status: "running" })].sort((a, b) => b.id.localeCompare(a.id)); + await db.insert(heartbeatRuns).values(tied); + if (tied[0].status === "running") expect(await select()).toBeNull(); + else expect((await select())?.id).toBe(tied[0].id); + }); + + it("does not skip future checkpoint barriers and preserves Date millisecond comparison", async () => { + const saved = source(), future = run(31); + await db.insert(heartbeatRuns).values([saved, future]); + expect((await select())?.id).toBe(saved.id); + await writeProfile(future.id, { sessionCheckpoint: false }); + expect(await select()).toBeNull(); + await db.execute(sql`update ${heartbeatRuns} set created_at = '2026-09-08T00:00:30.000999Z'::timestamptz where ${heartbeatRuns.id} = ${future.id}`); + expect((await select())?.id).toBe(future.id); + }); + + it("sees a newly committed barrier and changed source binding in its single selection snapshot", async () => { + const saved = source(); + await db.insert(heartbeatRuns).values(saved); + let reads = 0; + const changing = new Proxy(db, { get(target, key) { + if (key !== "with") return Reflect.get(target, key); + return (...ctes: Parameters) => { + const builder = target.with(...ctes), original = builder.select.bind(builder); + builder.select = ((...fields: Parameters) => { + const selection = original(...fields), from = selection.from.bind(selection); + selection.from = ((...args: Parameters) => { + const query = from(...args), execute = query.execute.bind(query); + query.execute = async (...executeArgs: Parameters) => { + reads++; + await db.insert(heartbeatRuns).values(run(1, { runnerProfileJson: { sessionCheckpoint: 0 } })); + await db.update(heartbeatRuns).set({ nativeSessionId: randomUUID() }).where(and(eq(heartbeatRuns.id, saved.id), eq(heartbeatRuns.companyId, companyId))); + return execute(...executeArgs); + }; + return query; + }) as typeof selection.from; + return selection; + }) as typeof builder.select; + return builder; + }; + }}); + const selected = await findNativeSessionResumeRun(changing, input); + expect(reads).toBe(1); + expect(selected?.id).not.toBe(saved.id); + expect(selected?.runnerProfileJson.sessionCheckpoint).toBe(0); + }); +}); diff --git a/server/src/services/native-runtime/native-session-resume.ts b/server/src/services/native-runtime/native-session-resume.ts index a271601ce0b..25463b68c78 100644 --- a/server/src/services/native-runtime/native-session-resume.ts +++ b/server/src/services/native-runtime/native-session-resume.ts @@ -1,6 +1,6 @@ import { createHash, randomUUID } from "node:crypto"; import { isDeepStrictEqual } from "node:util"; -import { and, desc, eq, inArray, ne } from "drizzle-orm"; +import { and, desc, eq, exists, inArray, isNotNull, ne, not, or, sql } from "drizzle-orm"; import { heartbeatRunEvents, heartbeatRuns, type Db } from "@paperclipai/db"; import type { NativeExecutionInput, @@ -186,6 +186,16 @@ export function selectNativeSessionResumeRun< return null; } +const NATIVE_SESSION_PROVIDER_EVENT_TYPES = [ + "harness.ready", + "session.started", + "session.resumed", + "session.updated", + "turn.started", + "provider.event", + "provider.rpc_result", +]; + export async function nativeSessionProviderEvidence( db: Pick, runIds: string[], @@ -197,15 +207,7 @@ export async function nativeSessionProviderEvidence( .where( and( inArray(heartbeatRunEvents.runId, runIds), - inArray(heartbeatRunEvents.eventType, [ - "harness.ready", - "session.started", - "session.resumed", - "session.updated", - "turn.started", - "provider.event", - "provider.rpc_result", - ]), + inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES), ), ) .groupBy(heartbeatRunEvents.runId); @@ -224,7 +226,50 @@ export async function findNativeSessionResumeRun( }, ) { if (!isNativeSessionId(input.normalizedSessionId)) return null; - const runs = await db + const candidates = db.$with("native_resume_candidates").as(db + .select({ + id: heartbeatRuns.id, + runtimeMode: heartbeatRuns.runtimeMode, + status: heartbeatRuns.status, + createdAt: heartbeatRuns.createdAt, + processPid: heartbeatRuns.processPid, + processGroupId: heartbeatRuns.processGroupId, + processStartedAt: heartbeatRuns.processStartedAt, + hasCheckpoint: sql`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' <> 'null'::jsonb, false)`.as("has_checkpoint"), + }) + .from(heartbeatRuns) + .where( + and( + ne(heartbeatRuns.id, input.currentRunId), + eq(heartbeatRuns.companyId, input.companyId), + eq(heartbeatRuns.agentId, input.agentId), + eq(heartbeatRuns.nativeIssueId, input.issueId), + eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId), + ), + ) + .orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)) + .limit(32)); + const terminalStatuses = [...LEGACY_RETRY_SOURCE_TERMINAL_STATUSES]; + const barrier = db.$with("native_resume_barrier").as(db + .select() + .from(candidates) + .where(or( + ne(candidates.runtimeMode, "native"), + not(inArray(candidates.status, terminalStatuses)), + candidates.hasCheckpoint, + isNotNull(candidates.processPid), + isNotNull(candidates.processGroupId), + isNotNull(candidates.processStartedAt), + exists(db.select({ runId: heartbeatRunEvents.runId }) + .from(heartbeatRunEvents) + .where(and( + eq(heartbeatRunEvents.runId, candidates.id), + inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES), + ))), + )) + .orderBy(desc(candidates.createdAt), desc(candidates.id)) + .limit(1)); + const [run] = await db.with(candidates, barrier) .select({ id: heartbeatRuns.id, companyId: heartbeatRuns.companyId, @@ -238,29 +283,39 @@ export async function findNativeSessionResumeRun( processPid: heartbeatRuns.processPid, processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt, - runnerProfileJson: heartbeatRuns.runnerProfileJson, - contextSnapshot: heartbeatRuns.contextSnapshot, + runnerProfileJson: sql<{ + nativeToolContractFingerprint: unknown; + nativeExecutionInput: unknown; + sessionCheckpoint: unknown; + }>`jsonb_build_object( + 'nativeToolContractFingerprint', ${heartbeatRuns.runnerProfileJson}->'nativeToolContractFingerprint', + 'nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput', + 'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' + )`, + contextSnapshot: sql<{ + paperclipWake: unknown; + paperclipIssue: unknown; + }>`jsonb_build_object( + 'paperclipWake', ${heartbeatRuns.contextSnapshot}->'paperclipWake', + 'paperclipIssue', case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->'paperclipIssue') = 'object' + then (select coalesce(jsonb_object_agg(brief.key, brief.value), '{}'::jsonb) + from jsonb_each(${heartbeatRuns.contextSnapshot}->'paperclipIssue') as brief + where brief.key in ('id', 'title', 'description')) else ${heartbeatRuns.contextSnapshot}->'paperclipIssue' end + )`, }) .from(heartbeatRuns) - .where( - and( - ne(heartbeatRuns.id, input.currentRunId), - eq(heartbeatRuns.companyId, input.companyId), - eq(heartbeatRuns.agentId, input.agentId), - eq(heartbeatRuns.nativeIssueId, input.issueId), - eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId), - ), - ) - .orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id)) - .limit(32); - return selectNativeSessionResumeRun({ - ...input, - runs, - providerEvidenceRunIds: await nativeSessionProviderEvidence( - db, - runs.map((run) => run.id), - ), - }); + .innerJoin(barrier, eq(heartbeatRuns.id, barrier.id)) + .where(and( + eq(barrier.runtimeMode, "native"), + inArray(barrier.status, terminalStatuses), + eq(barrier.hasCheckpoint, true), + sql`${heartbeatRuns.companyId}::text = ${input.companyId}`, + sql`${heartbeatRuns.agentId}::text = ${input.agentId}`, + sql`${heartbeatRuns.nativeIssueId}::text = ${input.issueId}`, + sql`${heartbeatRuns.nativeSessionId}::text = ${input.normalizedSessionId}`, + sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`, + )); + return run ?? null; } /** A preassigned session id may rotate only before immutable native admission. */ From 09d4ca533e5f84ddc7abb93660a72d1c0264d51e Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:32:09 -0500 Subject: [PATCH 14/41] perf(server): return only sidebar failure badge candidates (#17935) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/95/changes/paperclip/95 ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The sidebar badge shows failures that need attention. > - The service reads the newest 200 runs before it selects one run per agent. > - Most of those rows cannot contribute to the failure count. > - This pull request selects failure candidates in SQL and returns four required fields. > - The benefit is less database response data with the same badge and access rules. ## Linked Issues or Issue Description Refs: #12770. Refs: #17756. Refs: #17683. These are related load, inbox, and badge reports. This implementation was written independently from the current service. It does not change which issue states count as failures. **What existing behavior does this improve?** The run read used by the company sidebar badge. **Current behavior** Every call transfers up to 200 rows with eight scalar fields. The application then selects the latest run per agent and applies status, Mine, dismissal, and access rules. **Proposed behavior** SQL selects the same latest run per agent from the same 200-row window. It then applies status and Mine rules. It returns only the run ID, scope, issue ID, and creation time. The service still checks dismissals and current issue access. **Reason and benefit** A mixed synthetic fixture reduces the response from 30,689 to 1,297 value bytes. The complete badge response stays equal. **Breaking changes** None. The endpoint response and the 200-run window stay the same. ## What Changed - Keep company and active-agent filters before the existing 200-row limit. - Rank each agent by the full creation timestamp in descending order, then by the observation ordinal. The timestamp order explicitly selects the newest run. Exact ties retain the first observed row. No ID tie-break is added. - Apply failure and Mine filters after ranking. A shared agent's success cannot expose an older failure. - Return four fields for dismissal and fresh issue authorization. Supply the already constrained company ID in the service. - Add real PostgreSQL regression fixtures and a service-generated SQL comparison harness. Document the command in `doc/DEVELOPING.md`. ## Verification - `pnpm -r typecheck` passed. `pnpm build` then passed in a separate sequential step. The normal Runner verifier checked 1,509 inputs and 1,555 outputs before reuse. - `PAPERCLIP_SIDEBAR_TEST_DATABASE_URL= pnpm --filter @paperclipai/server exec vitest run src/__tests__/sidebar-badges.test.ts` passed all 14 tests on PostgreSQL 17.11. - The 10 unchanged `inbox-dismissals.test.ts` route tests passed against the same real PostgreSQL server through a temporary local adapter for database startup and cleanup. The adapter was removed. The standard host run passed the 13 new tests and skipped those 10 embedded-PostgreSQL tests. - The first focused attempt could not import the unbuilt plugin SDK. The normal build supplied that package before the passing reruns. - `pnpm test:run` was attempted. It recorded 16 embedded-PostgreSQL startup failures in the unrelated native-session executor suite. Other database suites also reported startup failures and skipped. The owned command was then stopped with exit 143. This is incomplete local full-suite evidence, not a pass. Full remote CI passed on final head `df5b2c81af43c282d031cc37e6bc867054217df5`: 54 successful checks, 2 neutral checks, and 2 optional Storybook skips; no pending or failed checks. Fresh exact-head Greptile review is 5/5 with zero unresolved threads. - A fixture rollback test reproduced the review finding before the fix: a failed final approval insert left 2 companies, 44 agents, 2 issues, and 657 runs. After the fix, the same PostgreSQL constraint error leaves zero owned rows in all five tables. The test also confirms that more than 600 runs were written before the deliberate failure. - Fixture seeding now uses one transaction, including the submillisecond SQL insert. The 14 focused tests and all 10 benchmark cases pass after this repair. The server typecheck also passes. Production SQL is unchanged. - The final ordering repair passed all 14 real-PostgreSQL tests, all 10 pinned-baseline benchmark cases, and affected server typecheck and build, run sequentially. The plans now explicitly sort by agent ID, full creation time descending, then observation ordinal. Complete badge responses and candidate bindings remain equal. - One server-test shard was interrupted by its CI runner receiving a shutdown signal. Its test step was canceled without an assertion failure. A single targeted retry passed; the aggregate `ci / verify` gate also passed. No source change was needed for that interruption. - `git diff --check` passed. No schema, workflow, or lockfile changed. Reproduce the benchmark with current schema migrations and a dedicated synthetic database whose name ends in `_test`: ```sh PAPERCLIP_SIDEBAR_TEST_DATABASE_URL= PAPERCLIP_LOG_LEVEL=error SIDEBAR_REQUIRE_BYTE_REDUCTION=1 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-sidebar-badges.ts --baseline-ref=bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e ``` The harness loads the baseline service from that commit. Its authorization, privacy, and shared helper dependencies are unchanged. It captures the SQL each service executes. It compares complete badge responses and every returned candidate field. Fixtures cover more than 200 runs, shared agents, superseding successes, exact and submillisecond ties, Mine user variants, dismissal equality, private and deleted issues, missing issue bindings, terminated agents, and cross-company rows. Tests also cover absent actors, grants, revocation, deletion after a prior read, and index versus sort plans. Synthetic local SQL results from PostgreSQL 17.11 on Docker, Node 24.22.1, and postgres.js 3.4.9 follow. Each timing is the median of seven alternating warm samples after two warmups. The report includes all samples, ranges, settings, SQL, parameters, and `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` plans. | Fixture | Rows before → after | Value bytes before → after | Reduction | Median ms before → after | | --- | --- | --- | --- | --- | | Mixed, company-wide | 200 → 18 | 30,689 → 1,297 | 95.8% | 1.361 → 1.305 | | Mixed, user-1 | 200 → 9 | 30,689 → 676 | 97.8% | 1.785 → 1.391 | | Mixed, local-board | 200 → 6 | 30,689 → 414 | 98.7% | 1.463 → 1.232 | | All failed, 200 distinct latest agents | 200 → 200 | 30,298 → 13,778 | 54.5% | 1.277 → 1.237 | | All succeeded | 200 → 0 | 30,898 → 0 | 100% | 1.095 → 0.898 | | Empty | 0 → 0 | 0 → 0 | Neutral | 0.703 → 1.002 | Value bytes mean UTF-8 non-null cell values before Drizzle decoding. They exclude PostgreSQL protocol framing and are not wire bytes. The primary claim is less response data. Warm timing ranges overlap. Earlier all-failed samples were slower, and the empty control remains slower. SQL adds ranking work. No production latency improvement is claimed. The plans keep the limit below both window operations. ## Risks - SQL adds two bounded window operations. The all-failed control transfers fewer bytes but can take more SQL time. - Per-agent ranking explicitly orders by full creation time descending. Exact timestamp ties retain the observation ordinal. Real PostgreSQL tests compare complete candidates with the original helper under index and sort plans. - Mine rules now also have a SQL form. Tests cover undefined, null, empty, named, and local-board users, including empty responsible-user IDs. - Issue access stays fresh and fail closed. No authorization result is cached. No migration or stored run change is required. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI GPT-6 through Codex. The exact serving model ID and context window are not exposed in this session. The model used reasoning, code execution, SQL inspection, and GitHub tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/DEVELOPING.md | 29 ++++ scripts/benchmark-sidebar-badges.ts | 104 +++++++++++++ .../helpers/sidebar-badges-fixture.ts | 127 ++++++++++++++++ server/src/__tests__/sidebar-badges.test.ts | 139 ++++++++++++++++++ server/src/services/sidebar-badges.ts | 53 +++++-- 5 files changed, 443 insertions(+), 9 deletions(-) create mode 100644 scripts/benchmark-sidebar-badges.ts create mode 100644 server/src/__tests__/helpers/sidebar-badges-fixture.ts create mode 100644 server/src/__tests__/sidebar-badges.test.ts diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 6f7b2b181cc..20f874a17ae 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -3382,6 +3382,35 @@ pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-prepar It compares actual preparation guards with synthetic history. Local timings are separate from Git materialization, environment acquisition, and provider startup. +### Sidebar badge run-query benchmark + +The badge query keeps the newest 200 runs of nonterminated same-company agents. +It selects each agent's first observed run before status or Mine filters. An +observation ordinal preserves equal-time input order. Per-agent ranking orders +by full creation time first, then that ordinal for exact ties. SQL returns only failure +IDs, issue bindings, and creation times. Dismissals and fresh issue authorization +still run in the service. + +Run the synthetic PostgreSQL comparison with a pinned pre-change revision: + +```sh +PAPERCLIP_LOG_LEVEL=error SIDEBAR_REQUIRE_BYTE_REDUCTION=1 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-sidebar-badges.ts --baseline-ref=bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e +``` + +The harness starts disposable embedded PostgreSQL by default. To use an existing +isolated test database with current migrations, set +`PAPERCLIP_SIDEBAR_TEST_DATABASE_URL` to a dedicated database whose name ends in +`_test`. The harness removes only its own synthetic company fixtures. Do not use +a development or hosted company database. + +The report includes service-generated SQL, complete badge equality, candidate +bindings, alternating warm samples, plans, and UTF-8 response-value bytes before +Drizzle decoding. This byte count excludes protocol framing. The mixed fixture +covers shared agents, superseding successes, timestamp ties, Mine users, +dismissal equality, private and deleted issues, and more than 200 runs. The +all-failed, all-succeeded, and empty controls expose neutral or slower paths. +Timings describe local SQL work and do not establish hosted request latency. + ### Budget accounting performance Budget admission and settlement share the company accounting lock. They evaluate diff --git a/scripts/benchmark-sidebar-badges.ts b/scripts/benchmark-sidebar-badges.ts new file mode 100644 index 00000000000..6e45bb45a65 --- /dev/null +++ b/scripts/benchmark-sidebar-badges.ts @@ -0,0 +1,104 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { promises as fs } from "node:fs"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { createDb, startEmbeddedPostgresTestDatabase, type Db } from "../packages/db/src/index.js"; +import { sidebarBadgeService } from "../server/src/services/sidebar-badges.js"; +import { captureSidebarQueries, cleanupSidebarFixture, observedSidebarFailures, seedSidebarFixture, type CapturedQuery } from "../server/src/__tests__/helpers/sidebar-badges-fixture.js"; + +const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15); +assert.ok(baselineRef, "Pass --baseline-ref="); +const baselineRevision = execFileSync("git", ["rev-parse", "--verify", `${baselineRef}^{commit}`], { encoding: "utf8" }).trim(); +const treatmentRevision = execFileSync("git", ["rev-parse", "HEAD"], { encoding: "utf8" }).trim(); +const externalUrl = process.env.PAPERCLIP_SIDEBAR_TEST_DATABASE_URL; +assert.ok(!externalUrl || new URL(externalUrl).pathname.endsWith("_test"), "Use a dedicated synthetic *_test database"); +const keepAlive = setInterval(() => {}, 1000); +let database: Awaited> | undefined; +let baselineDir: string | undefined; +let db: Db | undefined; +const fixtures: Awaited>[] = []; +let failure: unknown; +try { + if (!externalUrl) database = await startEmbeddedPostgresTestDatabase("sidebar-benchmark-"); + db = createDb(externalUrl ?? database!.connectionString); + const modulePath = "server/src/services/sidebar-badges.ts"; + const moduleUrl = new URL(`../${modulePath}`, import.meta.url); + const source = execFileSync("git", ["show", `${baselineRevision}:${modulePath}`], { encoding: "utf8" }); + const rewritten = source.replace(/from "(\.{1,2}\/[^\"]+)"/g, + (_match, specifier: string) => `from ${JSON.stringify(fileURLToPath(new URL(specifier, moduleUrl)))}`); + baselineDir = await fs.mkdtemp(path.join(fileURLToPath(new URL("../server/scripts/", import.meta.url)), ".sidebar-baseline-")); + const baselineFile = path.join(baselineDir, "sidebar-badges.ts"); + await fs.writeFile(baselineFile, rewritten); + const baseline = await import(pathToFileURL(baselineFile).href) as typeof import("../server/src/services/sidebar-badges.js"); + const [{ version }] = await db.$client.unsafe("select version()"); + const settings = await db.$client.unsafe("select name, setting from pg_settings where name in ('work_mem', 'shared_buffers', 'jit', 'max_parallel_workers_per_gather')"); + const results = []; + for (const mode of ["mixed", "all-failed", "all-succeeded", "empty"] as const) { + const fixture = await seedSidebarFixture(db, mode); + fixtures.push(fixture); + await db.$client.unsafe("ANALYZE heartbeat_runs"); + await db.$client.unsafe("ANALYZE agents"); + const users = mode === "mixed" ? [undefined, null, "", "user-1", "user-2", "local-board", "nobody"] : [undefined]; + for (const currentUserId of users) { + const queries = { baseline: [] as CapturedQuery[], treatment: [] as CapturedQuery[] }; + const before = baseline.sidebarBadgeService(captureSidebarQueries(db, queries.baseline)); + const after = sidebarBadgeService(captureSidebarQueries(db, queries.treatment)); + const options = { ...fixture.options, currentUserId }; + const baselineResponse = await before.get(fixture.companyId, options); + const treatmentResponse = await after.get(fixture.companyId, options); + assert.deepEqual(treatmentResponse, baselineResponse); + assert.equal(queries.baseline.length, 1); + assert.equal(queries.treatment.length, 1); + const expected = await observedSidebarFailures(db, fixture.companyId, currentUserId); + const normalize = (rows: unknown[][]) => rows.map(row => JSON.stringify(row)).sort(); + const baselineQuery = queries.baseline[0]!, treatmentQuery = queries.treatment[0]!; + const treatmentRows = await db.$client.unsafe(treatmentQuery.sql, treatmentQuery.params as never[]).values(); + if (treatmentQuery.sql !== baselineQuery.sql) { + assert.deepEqual(normalize(treatmentRows.map(row => [...row.slice(0, 3), new Date(String(row[3])).toISOString()])), normalize(expected.map(row => [row.id, row.scopeKind, row.issueId, row.createdAt.toISOString()]))); + } + const samples = { baseline: [] as number[], treatment: [] as number[] }; + const rowsByVersion: Record = {}; + for (let iteration = 0; iteration < 9; iteration++) { + for (const label of iteration % 2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const) { + const query = label === "baseline" ? baselineQuery : treatmentQuery; + const start = performance.now(); + rowsByVersion[label] = await db.$client.unsafe(query.sql, query.params as never[]).values(); + if (iteration >= 2) samples[label].push(performance.now() - start); + } + } + const measurements = []; + for (const label of ["baseline", "treatment"] as const) { + const query = label === "baseline" ? baselineQuery : treatmentQuery; + const rows = rowsByVersion[label]!; + const responseValueBytes = rows.flat().reduce((sum, value) => sum + (value == null ? 0 : Buffer.byteLength(typeof value === "string" ? value : String(value))), 0); + const [plan] = await db.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${query.sql}`, query.params as never[]); + const sorted = [...samples[label]].sort((a, b) => a - b); + measurements.push({ label, query, rows: rows.length, responseValueBytes, samplesMs: samples[label], medianMs: sorted[3], minMs: sorted[0], maxMs: sorted[6], plan: plan!["QUERY PLAN"] }); + } + if (process.env.SIDEBAR_REQUIRE_BYTE_REDUCTION === "1" && mode !== "empty") { + assert.ok(measurements[1]!.responseValueBytes < measurements[0]!.responseValueBytes); + } + results.push({ mode, currentUserId: currentUserId === undefined ? "" : currentUserId, equalBadgeResponse: treatmentResponse, equalCandidateRows: expected.length, measurements }); + } + } + console.log(JSON.stringify({ baselineRevision, treatmentRevision, + treatmentDiff: execFileSync("git", ["diff", "--", modulePath], { encoding: "utf8" }), + dependencyPolicy: "Baseline imports unchanged current authorization, privacy, and shared helpers", version, settings, + acceptance: "Identical complete badges and candidate bindings; fewer response-value bytes for nonempty fixtures. Timings are descriptive with no latency threshold.", + byteBoundary: "UTF-8 non-null cell values before Drizzle decoding, excluding PostgreSQL protocol framing; not wire bytes", + results }, null, 2)); +} catch (error) { + failure = error; +} finally { + const errors: unknown[] = failure === undefined ? [] : [failure]; + async function cleanup(action: () => Promise | undefined) { + try { await action(); } catch (error) { errors.push(error); } + } + if (db) for (const fixture of fixtures) await cleanup(() => cleanupSidebarFixture(db!, fixture)); + await cleanup(() => db?.$client.end({ timeout: 0 })); + await cleanup(() => database?.cleanup()); + await cleanup(() => baselineDir ? fs.rm(baselineDir, { recursive: true, force: true }) : undefined); + clearInterval(keepAlive); + if (errors.length) throw new AggregateError(errors, "Sidebar benchmark or cleanup failed"); +} diff --git a/server/src/__tests__/helpers/sidebar-badges-fixture.ts b/server/src/__tests__/helpers/sidebar-badges-fixture.ts new file mode 100644 index 00000000000..7effeb74757 --- /dev/null +++ b/server/src/__tests__/helpers/sidebar-badges-fixture.ts @@ -0,0 +1,127 @@ +import { randomUUID } from "node:crypto"; +import { and, desc, eq, inArray, not, sql } from "drizzle-orm"; +import { agents, approvals, companies, heartbeatRuns, issues, type Db } from "@paperclipai/db"; +import { isHeartbeatRunVisibleInMine, latestFailedRunsByAgent } from "@paperclipai/shared"; +import type { sidebarBadgeService } from "../../services/sidebar-badges.js"; + +export type BadgeOptions = Parameters["get"]>[1]; +export type CapturedQuery = { sql: string; params: unknown[] }; + +export function captureSidebarQueries(db: Db, queries: CapturedQuery[]): Db { + return new Proxy(db, { get(target, key) { + if (key !== "select") return Reflect.get(target, key); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const selection = from(...args), execute = selection.execute.bind(selection); + selection.execute = async () => { + const query = selection.toSQL(); + if (query.sql.includes('"heartbeat_runs"')) queries.push(query); + return execute(); + }; + return selection; + }) as typeof builder.from; + return builder; + }; + }}); +} + +export async function observedSidebarFailures(db: Db, companyId: string, currentUserId?: string | null) { + const rows = await db.select({ + id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, companyId: heartbeatRuns.companyId, + scopeKind: heartbeatRuns.scopeKind, issueId: heartbeatRuns.issueId, status: heartbeatRuns.status, + responsibleUserId: heartbeatRuns.responsibleUserId, createdAt: heartbeatRuns.createdAt, + }).from(heartbeatRuns).innerJoin(agents, eq(heartbeatRuns.agentId, agents.id)).where(and( + eq(heartbeatRuns.companyId, companyId), eq(agents.companyId, companyId), not(eq(agents.status, "terminated")), + )).orderBy(desc(heartbeatRuns.createdAt)).limit(200); + return latestFailedRunsByAgent(rows).filter(row => currentUserId === undefined || isHeartbeatRunVisibleInMine(row, currentUserId)); +} + +export async function seedSidebarFixture(db: Db, mode: "mixed" | "all-failed" | "all-succeeded" | "empty" = "mixed") { + return db.transaction(async tx => { + const companyId = randomUUID(), foreignCompanyId = randomUUID(); + await tx.insert(companies).values([ + { id: companyId, name: "Synthetic sidebar", issuePrefix: `B${companyId.slice(0, 7)}` }, + { id: foreignCompanyId, name: "Foreign sidebar", issuePrefix: `F${foreignCompanyId.slice(0, 7)}` }, + ]); + const actorId = randomUUID(); + await tx.insert(agents).values({ id: actorId, companyId, name: "Reader", role: "engineer" }); + const actor = { type: "agent" as const, agentId: actorId, companyId }; + const privateIssueId = randomUUID(), publicIssueId = randomUUID(), deletedIssueId = randomUUID(); + await tx.insert(issues).values([ + { id: privateIssueId, companyId, title: "Private", visibility: "private", privacyRootIssueId: privateIssueId, responsibleUserId: "other-user" }, + { id: publicIssueId, companyId, title: "Public" }, + { id: deletedIssueId, companyId, title: "Deleted" }, + ]); + const origin = Date.parse("2026-10-01T00:00:00.000Z"); + const agentIds = Array.from({ length: mode === "all-failed" ? 240 : 30 }, () => randomUUID()); + await tx.insert(agents).values(agentIds.map((id, index) => ({ id, companyId, name: `Agent ${index}`, role: "engineer" }))); + const runs: (typeof heartbeatRuns.$inferInsert)[] = []; + if (mode !== "empty") { + for (let index = 0; index < 420; index++) { + const agentIndex = index % agentIds.length; + runs.push({ id: randomUUID(), companyId, agentId: agentIds[agentIndex]!, + status: mode === "all-failed" ? "failed" : mode === "all-succeeded" ? "succeeded" : ["failed", "timed_out", "succeeded", "running", "queued", "cancelled"][agentIndex % 6], + responsibleUserId: ["user-1", "user-2", null, "", "local-board"][agentIndex % 5], + createdAt: new Date(origin + index), invocationSource: "manual" }); + } + } + const special: Record = {}; + if (mode === "mixed") { + async function add(name: string, values: Partial = {}) { + const agentId = randomUUID(), id = randomUUID(); + await tx.insert(agents).values({ id: agentId, companyId, name, role: "engineer" }); + runs.push({ id, companyId, agentId, status: "failed", responsibleUserId: "user-1", createdAt: new Date(origin + 1000 + runs.length), ...values }); + special[name] = id; + return agentId; + } + await add("outside-window", { createdAt: new Date(origin - 1000) }); + const shared = await add("superseded"); + runs.push({ companyId, agentId: shared, status: "succeeded", responsibleUserId: "user-2", createdAt: new Date(origin + 2000) }); + const foreignOwner = await add("other-user-latest"); + runs.push({ companyId, agentId: foreignOwner, status: "failed", responsibleUserId: "user-2", createdAt: new Date(origin + 2001) }); + await add("private", { scopeKind: "issue", issueId: privateIssueId }); + await add("public", { scopeKind: "issue", issueId: publicIssueId }); + await add("deleted", { scopeKind: "issue", issueId: deletedIssueId }); + await add("missing", { scopeKind: "issue", issueId: null }); + await add("dismissed"); + for (const [name, status] of [["tied-failure-first", "failed"], ["tied-success-first", "succeeded"]] as const) { + const agentId = await add(name, { status, createdAt: new Date(origin + 3000) }); + runs.push({ companyId, agentId, status: status === "failed" ? "succeeded" : "failed", responsibleUserId: "user-1", createdAt: new Date(origin + 3000) }); + } + const terminated = randomUUID(), crossCompanyAgent = randomUUID(); + await tx.insert(agents).values([ + { id: terminated, companyId, name: "Terminated", status: "terminated" }, + { id: crossCompanyAgent, companyId: foreignCompanyId, name: "Foreign" }, + ]); + for (let index = 0; index < 220; index++) runs.push({ companyId, agentId: index % 2 ? terminated : crossCompanyAgent, + status: "failed", responsibleUserId: "user-1", createdAt: new Date(origin + 10000 + index) }); + runs.push({ companyId: foreignCompanyId, agentId: crossCompanyAgent, status: "failed", createdAt: new Date(origin + 20000) }); + } + if (runs.length) await tx.insert(heartbeatRuns).values(runs); + if (mode === "mixed") { + await tx.delete(issues).where(eq(issues.id, deletedIssueId)); + const microAgent = randomUUID(); + await tx.insert(agents).values({ id: microAgent, companyId, name: "Submillisecond" }); + await tx.execute(sql`insert into heartbeat_runs (company_id, agent_id, status, responsible_user_id, created_at) values + (${companyId}, ${microAgent}, 'failed', 'user-1', '2026-10-01 00:00:04.000100+00'), + (${companyId}, ${microAgent}, 'succeeded', 'user-2', '2026-10-01 00:00:04.000900+00')`); + } + const approvalId = randomUUID(); + await tx.insert(approvals).values({ id: approvalId, companyId, type: "hire_agent", status: "pending", payload: {}, updatedAt: new Date(origin) }); + const dismissals = new Map([[`approval:${approvalId}`, origin]]); + const dismissed = runs.find(row => row.id === special.dismissed); + if (dismissed) dismissals.set(`run:${dismissed.id}`, (dismissed.createdAt as Date).getTime()); + return { companyId, foreignCompanyId, actor, actorId, privateIssueId, publicIssueId, special, dismissals, + options: { dismissals, joinRequests: [{ id: "join", createdAt: new Date(origin), updatedAt: null }], unreadTouchedIssues: 2, actor } satisfies BadgeOptions }; + }); +} + +export async function cleanupSidebarFixture(db: Db, fixture: Awaited>) { + const companyIds = [fixture.companyId, fixture.foreignCompanyId]; + await db.delete(heartbeatRuns).where(inArray(heartbeatRuns.companyId, companyIds)); + await db.delete(approvals).where(inArray(approvals.companyId, companyIds)); + await db.delete(issues).where(inArray(issues.companyId, companyIds)); + await db.delete(agents).where(inArray(agents.companyId, companyIds)); + await db.delete(companies).where(inArray(companies.id, companyIds)); +} diff --git a/server/src/__tests__/sidebar-badges.test.ts b/server/src/__tests__/sidebar-badges.test.ts new file mode 100644 index 00000000000..9830424dde9 --- /dev/null +++ b/server/src/__tests__/sidebar-badges.test.ts @@ -0,0 +1,139 @@ +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { count, eq, inArray } from "drizzle-orm"; +import { agents, approvals, companies, createDb, getEmbeddedPostgresTestSupport, heartbeatRuns, issueAccessGrants, issues, startEmbeddedPostgresTestDatabase, type Db } from "@paperclipai/db"; +import { sidebarBadgeService } from "../services/sidebar-badges.js"; +import { authorizationService } from "../services/authorization.js"; +import { getHeartbeatRunReadDecisions } from "../services/heartbeat-run-privacy.js"; +import { captureSidebarQueries, cleanupSidebarFixture, observedSidebarFailures, seedSidebarFixture, type BadgeOptions, type CapturedQuery } from "./helpers/sidebar-badges-fixture.js"; + +const externalUrl = process.env.PAPERCLIP_SIDEBAR_TEST_DATABASE_URL; +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const describePostgres = support.supported ? describe : describe.skip; +if (!support.supported) console.warn(`Skipping sidebar badge SQL tests: ${support.reason}`); + +describePostgres("sidebar badge failure candidates", () => { + let db: ReturnType; + let database: Awaited> | undefined; + const fixtures: Awaited>[] = []; + let fixture: Awaited>; + + beforeAll(async () => { + if (externalUrl && !new URL(externalUrl).pathname.endsWith("_test")) throw new Error("Use a dedicated synthetic *_test database"); + if (!externalUrl) database = await startEmbeddedPostgresTestDatabase("sidebar-badges-"); + db = createDb(externalUrl ?? database!.connectionString, { maxConnections: 1 }); + fixture = await seedSidebarFixture(db); + fixtures.push(fixture); + }, 90_000); + + afterAll(async () => { + const errors: unknown[] = []; + async function cleanup(action: () => Promise | undefined) { + try { await action(); } catch (error) { errors.push(error); } + } + for (const entry of fixtures) await cleanup(() => cleanupSidebarFixture(db, entry)); + await cleanup(() => db?.$client.end({ timeout: 0 })); + await cleanup(() => database?.cleanup()); + if (errors.length) throw new AggregateError(errors, "Sidebar test cleanup failed"); + }); + + async function verify(options?: BadgeOptions) { + const queries: CapturedQuery[] = []; + const actual = await sidebarBadgeService(captureSidebarQueries(db, queries)).get(fixture.companyId, options); + const expectedCandidates = await observedSidebarFailures(db, fixture.companyId, options?.currentUserId); + const candidates = await db.$client.unsafe(queries[0]!.sql, queries[0]!.params as never[]).values(); + const normalize = (rows: unknown[][]) => rows.map(row => JSON.stringify(row)).sort(); + expect(normalize(candidates.map(row => [...row.slice(0, 3), new Date(String(row[3])).toISOString()]))).toEqual(normalize( + expectedCandidates.map(row => [row.id, row.scopeKind, row.issueId, row.createdAt.toISOString()]), + )); + expect(expectedCandidates.map(row => row.id)).not.toContain(fixture.special.superseded); + expect(expectedCandidates.map(row => row.id)).not.toContain(fixture.special["outside-window"]); + const undismissed = expectedCandidates.filter(row => (options?.dismissals?.get(`run:${row.id}`) ?? -Infinity) < row.createdAt.getTime()); + const failedRuns = options?.actor + ? (await getHeartbeatRunReadDecisions(db, authorizationService(db), options.actor, undismissed)).filter(Boolean).length + : undismissed.length; + const approvals = options?.dismissals ? 0 : 1; + const joinRequests = options?.joinRequests?.length ?? 0; + expect(actual).toEqual({ inbox: approvals + failedRuns + joinRequests + (options?.unreadTouchedIssues ?? 0), approvals, failedRuns, joinRequests }); + return actual; + } + + it("rolls back all fixture rows when the final approval insert fails", async () => { + let companyIds: string[] = []; + let writtenRuns = 0; + function failLastInsert>(connection: T): T { + return new Proxy(connection, { get(target, key) { + if (key !== "insert") return Reflect.get(target, key); + return (table: Parameters[0]) => { + if (table === companies) return { values: (values: (typeof companies.$inferInsert)[]) => { + companyIds = values.map(row => row.id!); + fixtures.push({ ...fixture, companyId: companyIds[0]!, foreignCompanyId: companyIds[1]! }); + return target.insert(companies).values(values); + }}; + if (table === approvals) return { values: async (values: typeof approvals.$inferInsert) => { + const [row] = await target.select({ value: count() }).from(heartbeatRuns).where(inArray(heartbeatRuns.companyId, companyIds)); + writtenRuns = row!.value; + return target.insert(approvals).values({ ...values, payload: null as never }); + }}; + return target.insert(table); + }; + }}); + } + const failingDb = new Proxy(failLastInsert(db), { get(target, key) { + if (key !== "transaction") return Reflect.get(target, key); + return (callback: Parameters[0]) => db.transaction(tx => callback(failLastInsert(tx))); + }}); + await expect(seedSidebarFixture(failingDb)).rejects.toMatchObject({ cause: { code: "23502" } }); + expect(companyIds).toHaveLength(2); + expect(writtenRuns).toBeGreaterThan(600); + const rows = await Promise.all([ + db.select({ value: count() }).from(companies).where(inArray(companies.id, companyIds)), + db.select({ value: count() }).from(agents).where(inArray(agents.companyId, companyIds)), + db.select({ value: count() }).from(issues).where(inArray(issues.companyId, companyIds)), + db.select({ value: count() }).from(heartbeatRuns).where(inArray(heartbeatRuns.companyId, companyIds)), + db.select({ value: count() }).from(approvals).where(inArray(approvals.companyId, companyIds)), + ]); + expect(rows.map(([row]) => row!.value)).toEqual([0, 0, 0, 0, 0]); + }); + + it.each([undefined, null, "", "user-1", "user-2", "local-board", "nobody"])("keeps the bounded latest run before Mine filtering for %s", async currentUserId => { + await verify({ ...fixture.options, currentUserId }); + }); + + it("retains company-wide behavior without an actor or options", async () => { + await verify(); + await verify({ ...fixture.options, actor: undefined }); + await verify({ ...fixture.options, actor: { type: "none" } }); + }); + + it("rechecks private issue grants and deletion on every request", async () => { + const service = sidebarBadgeService(db); + const options = { ...fixture.options, currentUserId: "user-1" }; + const denied = await service.get(fixture.companyId, options); + const [grant] = await db.insert(issueAccessGrants).values({ issueId: fixture.privateIssueId, subjectType: "agent", subjectId: fixture.actorId, source: "explicit" }).returning(); + const allowed = await service.get(fixture.companyId, options); + expect(allowed.failedRuns).toBe(denied.failedRuns + 1); + await db.update(issueAccessGrants).set({ revokedAt: new Date() }).where(eq(issueAccessGrants.id, grant!.id)); + expect(await service.get(fixture.companyId, options)).toEqual(denied); + await db.delete(issues).where(eq(issues.id, fixture.publicIssueId)); + expect((await service.get(fixture.companyId, options)).failedRuns).toBe(denied.failedRuns - 1); + await verify(options); + }); + + it("preserves equal-time input order with index and sort plans", async () => { + await db.$client.unsafe("SET enable_indexscan = off"); + await db.$client.unsafe("SET enable_bitmapscan = off"); + try { await verify(fixture.options); } + finally { + await db.$client.unsafe("RESET enable_indexscan"); + await db.$client.unsafe("RESET enable_bitmapscan"); + } + }); + + it.each(["all-failed", "all-succeeded", "empty"] as const)("handles the %s control", async mode => { + const control = await seedSidebarFixture(db, mode); + fixtures.push(control); + const response = await sidebarBadgeService(db).get(control.companyId); + const failedRuns = mode === "all-failed" ? 200 : 0; + expect(response).toEqual({ inbox: failedRuns + 1, approvals: 1, failedRuns, joinRequests: 0 }); + }); +}); diff --git a/server/src/services/sidebar-badges.ts b/server/src/services/sidebar-badges.ts index 4f0249043c2..80348af686a 100644 --- a/server/src/services/sidebar-badges.ts +++ b/server/src/services/sidebar-badges.ts @@ -1,7 +1,7 @@ -import { and, desc, eq, inArray, not, type SQL } from "drizzle-orm"; +import { and, desc, eq, inArray, isNull, not, or, sql, type SQL } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, approvals, heartbeatRuns } from "@paperclipai/db"; -import { isHeartbeatRunVisibleInMine, latestFailedRunsByAgent, type SidebarBadges } from "@paperclipai/shared"; +import type { SidebarBadges } from "@paperclipai/shared"; import { authorizationService, type AuthorizationActor } from "./authorization.js"; import { getHeartbeatRunReadDecisions } from "./heartbeat-run-privacy.js"; @@ -51,11 +51,10 @@ export function sidebarBadgeService(db: Db) { rows.filter((row) => !isDismissed(extra?.dismissals ?? new Map(), `approval:${row.id}`, row.updatedAt)).length ); - const recentRuns = await db + const recentRuns = db .select({ id: heartbeatRuns.id, agentId: heartbeatRuns.agentId, - companyId: heartbeatRuns.companyId, scopeKind: heartbeatRuns.scopeKind, issueId: heartbeatRuns.issueId, status: heartbeatRuns.status, @@ -72,14 +71,50 @@ export function sidebarBadgeService(db: Db) { ), ) .orderBy(desc(heartbeatRuns.createdAt)) - .limit(200); + .limit(200) + .as("recent_runs"); - const failures = latestFailedRunsByAgent(recentRuns).filter((row) => - (extra?.currentUserId === undefined || isHeartbeatRunVisibleInMine(row, extra.currentUserId)) - && !isDismissed(extra?.dismissals ?? new Map(), `run:${row.id}`, row.createdAt), + const observedRuns = db.select({ + id: recentRuns.id, + agentId: recentRuns.agentId, + scopeKind: recentRuns.scopeKind, + issueId: recentRuns.issueId, + status: recentRuns.status, + responsibleUserId: recentRuns.responsibleUserId, + createdAt: recentRuns.createdAt, + observedOrder: sql`row_number() over ()`.as("observed_order"), + }).from(recentRuns).as("observed_runs"); + const rankedRuns = db.select({ + id: observedRuns.id, + scopeKind: observedRuns.scopeKind, + issueId: observedRuns.issueId, + status: observedRuns.status, + responsibleUserId: observedRuns.responsibleUserId, + createdAt: observedRuns.createdAt, + agentRunRank: sql`row_number() over (partition by ${observedRuns.agentId} order by ${observedRuns.createdAt} desc, ${observedRuns.observedOrder})`.as("agent_run_rank"), + }).from(observedRuns).as("ranked_runs"); + const mineCondition = extra?.currentUserId === undefined + ? undefined + : !extra.currentUserId + ? sql`false` + : extra.currentUserId === "local-board" + ? or(eq(rankedRuns.responsibleUserId, "local-board"), isNull(rankedRuns.responsibleUserId), eq(rankedRuns.responsibleUserId, "")) + : eq(rankedRuns.responsibleUserId, extra.currentUserId); + const candidates = await db.select({ + id: rankedRuns.id, + scopeKind: rankedRuns.scopeKind, + issueId: rankedRuns.issueId, + createdAt: rankedRuns.createdAt, + }).from(rankedRuns).where(and( + eq(rankedRuns.agentRunRank, 1), + inArray(rankedRuns.status, ["failed", "timed_out"]), + mineCondition, + )); + const failures = candidates.filter((row) => + !isDismissed(extra?.dismissals ?? new Map(), `run:${row.id}`, row.createdAt), ); const failedRuns = extra?.actor - ? (await getHeartbeatRunReadDecisions(db, authorization, extra.actor, failures)).filter(Boolean).length + ? (await getHeartbeatRunReadDecisions(db, authorization, extra.actor, failures.map((row) => ({ ...row, companyId })))).filter(Boolean).length : failures.length; const joinRequests = (extra?.joinRequests ?? []).filter((row) => From f6bc653f7f745adaafa26393852fb2f5a99256a4 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:33:30 -0500 Subject: [PATCH 15/41] fix(server): schedule interrupted tool claims by their deadlines (#17458) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip governs tool execution and keeps durable approval records. > - An interrupted execution may have produced an external effect without a saved final request state. > - The maintenance tick currently scans executing requests even when none exist. > - Every executing claim already has a ten-minute stale deadline. > - This PR schedules that deadline from committed claim changes and clears it after settlement. > - Recovery preserves saved provider outcomes and quarantines unknown outcomes without another provider call. ## Linked Issues or Issue Description Refs #17454. This is the sixth recurring-server sleep slice, stacked on `codex/sleep-05-approved-tool-recovery`. **Problem** The aggregate maintenance tick repeatedly searches for stale executing tool requests. Its settlement can overwrite a saved successful invocation with an unknown-outcome failure after a crash between invocation and request writes. **Proposed Solution** Use the existing review notification boundary and shared scheduler. Read the earliest executing claim deadline through a partial index. At that deadline, lock invocation then request, recheck claim identity, and atomically settle from the saved outcome or quarantine the unknown outcome. **Alternatives Considered** A longer polling interval still queries empty work. A new queue would duplicate existing records. Replaying an interrupted call can duplicate an external effect. Treating every stale request as unknown discards already saved outcomes. ## What Changed - Coalesce sequential review notifications for up to 30 seconds while preserving already loaded recovery deadlines. Do not schedule scans when no notification or execution deadline remains. - Extract interrupted-execution deadline lookup and settlement into the owning tool-review service. - Register its worker at startup. Use committed claim/result changes to update the existing ten-minute deadline; retry failed settlement after 30 seconds. - Preserve terminal invocation results, reasons, and completion times when repairing request state. - Settle uncertain request/invocation state atomically with the existing unknown-outcome reason, without dispatching a provider call. - Notify all gateway claim and settlement paths through the existing review transaction helper. Keep framework wiring and the shared scheduler free of provider-specific behavior. - Add an idempotent concurrent partial index for executing deadlines. Preserve the combined manual sweep and scheduler-disabled startup pass. - Remove interrupted scanning from the aggregate tick. Interaction projection repair remains the next separate duty. ## Verification Current head: `648d8a9a64ded15deeeb331545b92a12d69e277d`. The final commit changes no files and requests CI after the master retarget. A preceding run appeared while that trigger was being published; it was observational delay, not an established CI outage. No test rerun or bypass was used to turn a failure green. Source remains identical to the 198-test passing integration, apart from the already recorded ledger update. Fresh checks and both automated reviews must qualify this current head before merge. Current master-targeted head: `d55e44b376555e4926bb2c2a0e4dcfb431324162`. Parent #17454 landed as `77295faebc8`; a normal merge reconciles that squash. Every conflicted master file was checked against the already-integrated parent, and all matched byte-for-byte. Each interruption-only hunk was retained; the complete integrated source tree matched the 198-test passing `848f1b87b3` exactly before the sole final documentation update. All master migration journal entries are unchanged, with only this PR's executing-deadline index appended. Server types pass. Fresh CI and current-head Greptile/Ponytail are required before the next authorized merge. Latest published integration: `848f1b87b3a6646ae4e56e8ab142f64889d7a69c`, normally merging approved-recovery parent `23868c1751`. All 198 gateway/interruption/startup/deadline cases and server production typechecking pass. Both parent approval fixes and all interruption tests are retained. The two merge conflicts preserved the interruption import alongside the parent's claim-bound hash helper and preserved the parameterized historical test helpers. This remains stacked on #17454; fresh CI and both automated review gates are required after integration. - Review documentation correction at `c985eb9d1b48a481909de6a1c2d89d7af11bfc03`: the focused real-PostgreSQL interruption-worker selection passes all 16 cases; 124 unrelated cases were filtered out. The docs now correctly name 16 cases and six saved terminal statuses. No production behavior changed in the correction. Fresh remote CI/review is pending after integrating the verified parent. Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/f6a5949fb726f71c247950714adecb275655284f/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Current head: `f6a5949fb726f71c247950714adecb275655284f`. This remains a single-duty PR stacked on #17454. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/eaac8d343e5ef4d2c13cdf7cb7607cb5a498e9a2/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the existing single-active-writer database owner. Independent writers and replicas need an explicit wake contract. - The ten-minute policy is unchanged. A provider call that outlives its claim remains an uncertain outcome; this worker never retries the external call. - Interaction projection follows authoritative settlement outside its transaction. Its separate existing repair duty still handles an interrupted projection. - Non-executing review writes can cause one empty reconciliation. Without an executing claim or failed attempt, no recurring deadline remains. - Cloud eligibility and external-alarm coverage remain conservative. Worker quiet does not prove actual host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-interrupted-tool-recovery.md | 116 ++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 52 +++- packages/db/src/client.ts | 1 + .../src/migrations/0385_open_madame_web.sql | 1 + ...{0380_snapshot.json => 0385_snapshot.json} | 101 ++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/tool_access.ts | 1 + .../server-startup-feedback-export.test.ts | 2 + .../__tests__/tool-gateway-service.test.ts | 253 ++++++++++++++++++ server/src/index.ts | 11 +- .../services/delivery-work-notifications.ts | 1 + server/src/services/tool-gateway.ts | 90 ++----- .../services/tool-review-interruption-work.ts | 16 ++ .../src/services/tool-review-interruption.ts | 61 +++++ .../tool-review-work-notifications.ts | 1 + 15 files changed, 643 insertions(+), 71 deletions(-) create mode 100644 doc/idle-interrupted-tool-recovery.md create mode 100644 packages/db/src/migrations/0385_open_madame_web.sql rename packages/db/src/migrations/meta/{0380_snapshot.json => 0385_snapshot.json} (99%) create mode 100644 server/src/services/tool-review-interruption-work.ts create mode 100644 server/src/services/tool-review-interruption.ts diff --git a/doc/idle-interrupted-tool-recovery.md b/doc/idle-interrupted-tool-recovery.md new file mode 100644 index 00000000000..c02ecdd97ec --- /dev/null +++ b/doc/idle-interrupted-tool-recovery.md @@ -0,0 +1,116 @@ +# Interrupted tool-action recovery + +An executing tool-action request has an existing ten-minute stale deadline from +its `updatedAt` claim timestamp. Recovery now registers the earliest actual +claim deadline with the shared scheduler. It no longer scans executing requests +on every aggregate maintenance tick. + +## Ownership and producers + +The gateway owns execution claims, provider calls, and result settlement. The +new interruption service owns only the deadline query and stale-row settlement. +The worker uses the existing deadline registration and exposes no provider +protocol or signed argument representation to the scheduler. + +The review write helper signals interruption alongside approved recovery and +pending expiry. All three agent claim paths and the Test-tab claim use that +helper. Test-tab outcomes already commit both rows inside it. Agent pre-dispatch +and execution failure settlement uses the same notification transaction, and +successful or failed direct execution request updates now notify before writing. +Administrative review transitions continue to use the same existing boundary. + +Unrelated pending review changes can cause a bounded empty reconciliation. They +do not create a periodic deadline. Process-local signals cover the existing +single-active-writer database owner; independent writers require a wake contract. + +## Settlement and quiet states + +The query uses a partial executing-request index and reads one earliest row. +There is no deadline after all executing claims settle. Retained terminal +requests and configured-but-idle tools produce no recurring worker queries. + +At the deadline, settlement locks invocation then request, matching the existing +execution-failure lock order. It rechecks the claim status and timestamp after +locking. A changed or completed claim cannot be settled from an earlier scan. +Unknown outcomes atomically mark both rows failed with +`tool_execution_outcome_unknown`. Recovery never calls the external provider. + +A saved terminal invocation is authoritative. Recovery repairs its request to +executed or failed without replacing the saved result, failure reason, or +completion time with an unknown-outcome failure. This covers a crash between +legacy invocation completion and request settlement. Interaction projection +runs after the transaction through the existing gateway owner. A failed +projection still relies on the separate projection-repair duty. + +A database failure rolls back both outcome rows and retains a 30-second retry. +Startup reconstructs saved deadlines. Drain pauses dispatch without forgetting +the first claim or failed retry; resume processes overdue claims. Completion +notifications cancel unnecessary future scans. The scheduler-disabled path keeps +one startup pass. The public combined review sweep retains its existing duties. + +## Verification and limits + +Sixteen new real-PostgreSQL cases cover real foreground execution activation +and completion, configured idle and terminal history, restart/deadline timing, +six saved terminal invocation states, first-write drain/resume, outer commit and +rollback, transactional failure/retry, concurrent claim change, and multi-page +settlement. The service/startup selection passes 136 tests on the combined parent-review +head. Workspace typecheck/build and migration safety checks pass on that head. +Migration application and reapplication passed. The combined run also exercised +469 tool-access tests: 468 passed and an unchanged connection-archive case timed +out under concurrent build load. That case passes in isolation. The separate full tool-access suite passes all +469 tests and gateway acceptance passes all 92 (697 across the final selections). +Fresh full CI/review remains required. + +The first typecheck caught optional approval IDs losing narrowing in transaction +callbacks; capturing them before the callback fixes it. Two new fault/transaction +cases initially waited for idle before advancing their next fake-clock retry. +They now advance time while waiting for the authoritative outcome, and the full +selection passes. These are fixture corrections, not permission to clear an +unresolved work hold. + +The ten-minute policy is unchanged. Active provider execution, unknown external +outcomes, and other worker obligations still prevent unsafe sleep. No Cloud +eligibility or external-alarm coverage changes here. Worker quiet is not evidence +of whole-instance or database suspension. + +## Sequential-write cost correction (October 27) + +Review found that sequential tenant writes could each request a global +interruption pass, even when no execution was stale. A real-SQL regression +created ten tool review requests through the gateway and allowed each preceding +pass to finish. It measured 11 stale-claim scans including startup. Concurrent +notification coalescing alone did not cover this sequence. + +The worker now uses the existing deadline helper's 30-second notification +coalescing window, matching the former default maintenance cadence. The same +regression measures one startup scan and one follow-up scan for those ten +requests, then no worker SQL through an additional idle hour. A loaded stale +claim deadline only 500 milliseconds away still preempts the window and +quarantines the unknown outcome without invoking the provider. Foreground +execution and result settlement remain immediate. Newly discovered work can +wait up to this window; the ten-minute stale-claim policy is unchanged. + +This is the bounded-frequency alternative raised in review. Discovery is still +instance-wide, and real stale backlogs still drain through the existing pages. +It does not claim company-isolated scans or constant cost for a real backlog. +No notification and no saved execution deadline means no periodic scan. No new +table, index, or scheduler is added. Startup, drain, retry and shutdown retain +the existing owners. + +The first full verification passed 175 cases and failed three fake-clock waits: +settlement writes leave a notification pending for its coalesced follow-up, so +waiting for the idle hold without advancing that armed deadline cannot finish. +An initial shared clock adjustment then advanced an unrelated five-second +expiry test too far (177 passed, one failed). The adjustment is now confined to +the interruption tests. It lets real SQL finish and advances only to an actually +armed short follow-up. The imminent-deadline test checks quarantine before that +30-second window and separately waits for eventual quiet. Product assertions +were retained. + +The final complete gateway-service and startup-wiring suites pass all 178 cases +without skips, and server typechecking passes, on owner source based on +`e255d8eff3363cf0bb63541bdc7ef055d4321f83`. The attempted third test selector +`deadline-work.test.ts` did not exist at this owner; no separate helper-suite +pass is claimed. The shared helper is exercised by the full gateway suites. +Fresh combined-source checks and remote review remain required. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index b699f1b242d..b8976fb34fb 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -18,7 +18,7 @@ Do not mark an item done from a timer removal or a green check alone. | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | -| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), historical argument identity repair verified and awaiting fresh CI/review; other review duties pending | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), merged after current-head CI and both 5/5 reviews; interrupted claims: [#17458](https://github.com/paperclipai/paperclip/pull/17458), integration verified and awaiting fresh CI/review; projection: [#17468](https://github.com/paperclipai/paperclip/pull/17468), interaction-link index verified and awaiting fresh CI/review | | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | @@ -230,3 +230,53 @@ tool-access service tests pass after enabling the existing external database path independently of embedded support. Its initial browser shard failed during unrelated plan-task fixture company deletion with a concurrent activity-log FK violation. Inspect fresh-head CI before deciding whether that needs a repair. + + +## Slice 06: interrupted tool executions + +Branch `codex/sleep-06-interrupted-tool-recovery` is based on #17454 and includes +its one-shot outcome-write repair. The next deadline is the minimum executing +claim timestamp plus the existing ten minutes. The new service locks invocation +then request, rechecks claim identity, preserves saved terminal outcomes, and +atomically quarantines unknown outcomes without calling a provider. All claim +producers and request settlements use the existing review notification boundary. +A partial index supports the first deadline. The aggregate tick now excludes +interrupted claims; projection repair remains its own pending slice. + +Sixteen new worker cases and the existing service/startup selection pass +(136 total on the combined head). Workspace typecheck/build and migration checks +pass. Two fake-clock fixture ordering failures were corrected to wait for real +settlement while advancing retry time. The combined tool-access run had one +unchanged connection-archive timeout; it passed in isolation. The full suite +and gateway acceptance are running separately before readiness is claimed. +No PR is claimed ready until fresh CI and review pass. + +## Next owner audit: tool-review interaction projection + +The remaining aggregate tool-review duty selects terminal requests whose +interaction result does not match the authoritative request state. Recovery +currently reads up to 100 joined rows. Move this duty to its own transaction- +notified worker, with retries only while projection debt remains. Do not add +an outbox table merely to migrate the scan. + +Audit the final two direct request writers in the gateway as well as existing +review helpers, expiry, interruption, and Test-tab outcome settlement. Preserve +the existing reflection owner: it locks issue/interaction/request, checks the +current request state before writing the card, then publishes activity and +requests continuation. A no-op reflection can mean an ownership or lifecycle +condition, not successful reconciliation. Retained terminal requests with +already matching cards must be quiet. Unprojectable/mismatched durable state +must not be silently dropped. Paginate without starving later candidates. + +Continuation delivery already has its own coordinator. A projection write is +not proof that downstream delivery or activity publication completed. Identify +the exact existing recovery owner before removing any backstop, and keep +provider/private payload interpretation out of the scheduler. + +Final slice-06 local verification: the isolated tool-access rerun passes all +469 tests, gateway acceptance passes 92, and the combined service/startup +selection passed 136. All 697 selected tests pass on the combined head. The +initial concurrent-run timeout is retained above as evidence, not hidden. +Workspace typecheck/build and migration safety/numbering pass. The first four +PRs are mergeable and have all checks passed at their recorded heads. #17454 +is mergeable with fresh checks and review in progress at `7a1cf03e62`. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index b3d66e14855..3e4cce15e2e 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -503,6 +503,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0369_late_logan.sql", "0381_mean_scream.sql", "0382_narrow_the_executioner.sql", + "0385_open_madame_web.sql", "0384_funny_adam_warlock.sql", "0383_tool_review_company_expiry.sql", ]); diff --git a/packages/db/src/migrations/0385_open_madame_web.sql b/packages/db/src/migrations/0385_open_madame_web.sql new file mode 100644 index 00000000000..19ce52b5cdb --- /dev/null +++ b/packages/db/src/migrations/0385_open_madame_web.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "tool_action_requests_executing_deadline_idx" ON "tool_action_requests" USING btree ("updated_at","id") WHERE "tool_action_requests"."status" = 'executing'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0380_snapshot.json b/packages/db/src/migrations/meta/0385_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0380_snapshot.json rename to packages/db/src/migrations/meta/0385_snapshot.json index 5e635e8be52..5564d696be5 100644 --- a/packages/db/src/migrations/meta/0380_snapshot.json +++ b/packages/db/src/migrations/meta/0385_snapshot.json @@ -1,6 +1,6 @@ { - "id": "bfda87fa-662b-47bb-87c0-dabc814aebd7", - "prevId": "2a9439e3-4a2b-4a20-ab2b-2aee3db8cb03", + "id": "9965d300-e355-4f47-ba0a-951ed3b4b8a2", + "prevId": "4f96f70a-fd19-4d35-ad8a-8d5559b83cbd", "version": "7", "dialect": "postgresql", "tables": { @@ -52812,6 +52812,88 @@ "method": "btree", "with": {} }, + "tool_action_requests_pending_expiry_idx": { + "name": "tool_action_requests_pending_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_company_pending_expiry_idx": { + "name": "tool_action_requests_company_pending_expiry_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_executing_deadline_idx": { + "name": "tool_action_requests_executing_deadline_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'executing'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_approved_idx": { + "name": "tool_action_requests_approved_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'approved'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_invocation_idx": { "name": "tool_action_requests_invocation_idx", "columns": [ @@ -55222,6 +55304,21 @@ "method": "btree", "with": {} }, + "tool_gateway_sessions_expiry_idx": { + "name": "tool_gateway_sessions_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_gateway_sessions_run_idx": { "name": "tool_gateway_sessions_run_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 6b79f74c167..f4cfed69f11 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2682,6 +2682,13 @@ "when": 1793060791728, "tag": "0384_funny_adam_warlock", "breakpoints": true + }, + { + "idx": 385, + "version": "7", + "when": 1793060791729, + "tag": "0385_open_madame_web", + "breakpoints": true } ] } diff --git a/packages/db/src/schema/tool_access.ts b/packages/db/src/schema/tool_access.ts index f1e788626e3..9f79743d6c3 100644 --- a/packages/db/src/schema/tool_access.ts +++ b/packages/db/src/schema/tool_access.ts @@ -836,6 +836,7 @@ export const toolActionRequests = pgTable( .where(sql`${table.status} = 'pending'`), index("tool_action_requests_company_pending_expiry_idx").on(table.companyId, table.expiresAt) .where(sql`${table.status} = 'pending'`), + index("tool_action_requests_executing_deadline_idx").on(table.updatedAt, table.id).where(sql`${table.status} = 'executing'`), index("tool_action_requests_approved_idx").on(table.id).where(sql`${table.status} = 'approved'`), index("tool_action_requests_invocation_idx").on(table.invocationId), index("tool_action_requests_issue_idx").on(table.companyId, table.issueId), diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 658f5c418ff..1a9fcf51576 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -69,6 +69,8 @@ const { stop: vi.fn(async () => {}), }, toolGateway: { + settleInterruptedActionReviews: vi.fn(async () => ({ scanned: 0 })), + nextActionReviewInterruptionAt: vi.fn(async () => null), recoverApprovedActionReviews: vi.fn(async () => ({ scanned: 0, retry: false })), approvedActionReviewCompanies: vi.fn(async () => []), sweepActionReviews: vi.fn(async () => ({ scanned: 0 })), diff --git a/server/src/__tests__/tool-gateway-service.test.ts b/server/src/__tests__/tool-gateway-service.test.ts index 4a014b22ab0..35dabccbe22 100644 --- a/server/src/__tests__/tool-gateway-service.test.ts +++ b/server/src/__tests__/tool-gateway-service.test.ts @@ -1,3 +1,5 @@ +import { registerToolReviewInterruptionWork } from "../services/tool-review-interruption-work.js"; +import { writeToolReviewChange } from "../services/tool-review-work-notifications.js"; import { deleteCompany } from "../services/company-deletion.js"; import { subscribeDeliveryWork } from "../services/delivery-work-notifications.js"; import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; @@ -318,6 +320,257 @@ describeEmbeddedPostgres("tool gateway service", () => { expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); } + async function testRequest(provider: NonNullable, rateLimited = false) { + await db.insert(companyMemberships).values({ companyId: fixture.company.id, principalType: "user", + principalId: "board", status: "active", membershipRole: "owner" }); + const remote = await createRemoteMcpToolFixture(db, fixture.company.id); + await db.update(toolCatalogEntries).set({ isReadOnly: false, riskLevel: "write" }) + .where(eq(toolCatalogEntries.id, remote.catalogEntry.id)); + await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "Review connected writes", + policyType: "require_approval", priority: 10, selectors: { connectionId: remote.connection.id } }); + if (rateLimited) await db.insert(toolPolicies).values({ companyId: fixture.company.id, name: "One reviewed test call", + policyType: "rate_limit", priority: 1, selectors: { connectionId: remote.connection.id }, + config: { limit: 1, windowSeconds: 3600, keyBy: ["agent", "tool"] } }); + gateway = createTestToolGatewayService(db, { remoteHttpRequest: provider }); + const result = await gateway.executeTestCall({ companyId: fixture.company.id, connectionId: remote.connection.id, + agentId: fixture.agent.id, userId: "board", toolName: "needs_input", parameters: { value: "reviewed" } }); + expect(result.decision).toBe("ask_first"); + const [pending] = await db.select().from(toolActionRequests); + return { pending, remote }; + } + function providerFixture() { + let release!: () => void, entered!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const started = new Promise(resolve => { entered = resolve; }); + const provider = vi.fn(async (_url, init) => { + const payload = JSON.parse(String(init.body)); + expect(payload.method).toBe("tools/call"); + entered(); await gate; + return Response.json({ jsonrpc: "2.0", id: payload.id, + result: { content: [{ type: "text", text: "Saved once" }] } }); + }); + return { provider, release, started }; + } + + describe("interrupted review recovery worker", () => { + async function startInterruption() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerToolReviewInterruptionWork(coordinator, gateway, () => !draining).ready; + } + async function settle(ms = 60_000) { + await vi.advanceTimersByTimeAsync(ms); + if (!draining) await vi.waitFor(async () => { + // Finish real SQL before following an armed coalescing/retry deadline. + // Do not jump across a future business deadline while waiting for I/O. + if (idleWorkSnapshot().active > 0) { + await db.execute(sql`select pg_sleep(0.01)`); + const next = coordinator!.nextWakeAt(); + if (next !== null && next <= Date.now() + 30_000) { + await vi.advanceTimersByTimeAsync(Math.max(0, next - Date.now())); + } + } + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + } + async function quiet() { + await settle(); expect(coordinator!.nextWakeAt()).toBeNull(); + const selects = vi.spyOn(db, "select"), executes = vi.spyOn(db, "execute"); + await settle(60 * 60_000); + expect(selects).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + selects.mockRestore(); executes.mockRestore(); + } + async function savedClaim(source = db, age = 0) { + const pending = await request(); + const claimedAt = new Date(Date.now() - age); + await writeToolReviewChange(source, fixture.company.id, async tx => { + await tx.update(toolInvocations).set({ status: "executing", startedAt: claimedAt, updatedAt: claimedAt }) + .where(eq(toolInvocations.id, pending.invocationId)); + await tx.update(toolActionRequests).set({ status: "executing", updatedAt: claimedAt }) + .where(eq(toolActionRequests.id, pending.id)); + }); + return { ...pending, updatedAt: claimedAt }; + } + async function unknown(pending: Awaited>) { + expect((await db.select().from(toolActionRequests).where(eq(toolActionRequests.id, pending.id)))[0].status).toBe("failed"); + expect((await db.select().from(toolInvocations).where(eq(toolInvocations.id, pending.invocationId)))[0]) + .toMatchObject({ status: "failed", errorCode: "tool_execution_outcome_unknown" }); + } + + it("coalesces sequential review writes into one interruption scan per window", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + const previous = db.$client.options.debug; + const scans: string[] = []; + db.$client.options.debug = (_connection, query) => { + if (query.includes('from "tool_action_requests"') && query.includes('"updated_at" <=')) scans.push(query); + }; + try { + await startInterruption(); + const firstPassAt = Date.now(); + expect(scans).toHaveLength(1); + for (let i = 0; i < 10; i++) { + await request(); + await vi.advanceTimersByTimeAsync(1); + // Let each previous pass finish; overlapping calls already coalesce. + await db.execute(sql`select pg_sleep(0.01)`); + } + expect(scans).toHaveLength(1); + expect(coordinator!.nextWakeAt()).toBe(firstPassAt + 30_000); + await vi.advanceTimersByTimeAsync(30_000); + await vi.waitFor(() => { + expect(scans).toHaveLength(2); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBeNull(); + }); + await quiet(); + expect(scans).toHaveLength(2); + } finally { db.$client.options.debug = previous; } + }); + + it("keeps a loaded stale-claim deadline ahead of review notification coalescing", async () => { + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + const pending = await savedClaim(db, 10 * 60_000 - 500); + const deadline = pending.updatedAt.getTime() + 10 * 60_000; + await startInterruption(); + expect(coordinator!.nextWakeAt()).toBe(deadline); + await request(); + await vi.advanceTimersByTimeAsync(1); + await db.execute(sql`select pg_sleep(0.01)`); + expect(coordinator!.nextWakeAt()).toBe(deadline); + await vi.advanceTimersByTimeAsync(499); + await vi.waitFor(async () => { await unknown(pending); }); + expect(Date.now()).toBeLessThan(deadline + 1000); + await quiet(); + }); + + it("arms from a real execution claim and disarms after the provider result", async () => { + await startInterruption(); await quiet(); + const remote = providerFixture(); + const { pending } = await testRequest(remote.provider); + const running = gateway.approveActionRequest({ companyId: fixture.company.id, + actionRequestId: pending.id, actor: { userId: "board" } }); + await Promise.race([remote.started, running]); + try { + await settle(); + const [claim] = await db.select().from(toolActionRequests); + expect(claim.status).toBe("executing"); + expect(coordinator!.nextWakeAt()).toBe(claim.updatedAt.getTime() + 10 * 60_000); + const reads = vi.spyOn(db, "select"); + await settle(60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + } finally { remote.release(); } + await running; await quiet(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("executed"); + expect(remote.provider).toHaveBeenCalledTimes(1); + }); + + it("reconstructs the saved deadline and quarantines only when due", async () => { + const pending = await savedClaim(); await startInterruption(); + expect(coordinator!.nextWakeAt()).toBe(pending.updatedAt.getTime() + 10 * 60_000); + await settle(9 * 60_000); + expect((await db.select().from(toolActionRequests))[0].status).toBe("executing"); + await settle(2 * 60_000); await unknown(pending); await quiet(); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); + }); + + it.each(["succeeded", "failed", "denied", "timed_out", "rate_limited", "cancelled"] as const)("preserves a saved %s invocation while repairing the request", async status => { + const pending = await savedClaim(db, 11 * 60_000); + const completedAt = new Date(Date.now() - 5 * 60_000); + await db.update(toolInvocations).set({ status, completedAt, resultSummary: { summary: "Saved provider result" }, + errorCode: status === "succeeded" ? null : "saved_provider_failure" }).where(eq(toolInvocations.id, pending.invocationId)); + await startInterruption(); await quiet(); + expect((await db.select().from(toolActionRequests))[0]).toMatchObject({ + status: status === "succeeded" ? "executed" : "failed", resolvedAt: completedAt }); + expect((await db.select().from(toolInvocations))[0]).toMatchObject({ status, completedAt, + errorCode: status === "succeeded" ? null : "saved_provider_failure", resultSummary: { summary: "Saved provider result" } }); + }); + + it("retains a first claim committed during drain without passive database scans", async () => { + await startInterruption(); await quiet(); draining = true; + const pending = await savedClaim(); await settle(); + const reads = vi.spyOn(db, "select"); + await settle(60 * 60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; await settle(); await unknown(pending); await quiet(); + }); + + it.each([false, true])("waits for outer execution commit (rollback %s)", async rollback => { + const pending = await request(); await startInterruption(); await quiet(); + let entered!: () => void, release!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const aborted = new Error("fixture rollback"); + const running = db.transaction(async tx => { + await writeToolReviewChange(tx as unknown as typeof db, fixture.company.id, async nested => { + await nested.update(toolActionRequests).set({ status: "executing", updatedAt: new Date() }) + .where(eq(toolActionRequests.id, pending.id)); + }); + entered(); await gate; if (rollback) throw aborted; + }).catch(error => { if (error !== aborted) throw error; }); + await ready; + try { + await vi.advanceTimersByTimeAsync(11 * 60_000); + expect((await db.select().from(toolActionRequests))[0].status).toBe("pending"); + } finally { release(); await running; } + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + if (rollback) expect((await db.select().from(toolActionRequests))[0].status).toBe("pending"); + else await unknown(pending); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + await quiet(); + }); + + it("rolls back both outcome rows on failure and retries after drain", async () => { + const pending = await savedClaim(db, 11 * 60_000); + await db.execute(sql`create function sleep_review_fail() returns trigger language plpgsql as $$ + begin raise exception 'interruption fixture failure'; end $$`); + await db.execute(sql`create trigger sleep_review_fail before update on tool_invocations + for each row when (new.error_code = 'tool_execution_outcome_unknown') execute function sleep_review_fail()`); + await startInterruption(); expect(errors.length).toBeGreaterThan(0); errors.length = 0; + expect((await db.select().from(toolActionRequests))[0].status).toBe("executing"); + expect((await db.select().from(toolInvocations))[0].status).toBe("executing"); + draining = true; + await db.execute(sql`drop trigger sleep_review_fail on tool_invocations`); + await settle(60 * 60_000); + draining = false; + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + await unknown(pending); + }, { timeout: 5000 }); + await quiet(); + }); + + it("rechecks a changed claim after selecting a stale candidate", async () => { + const pending = await savedClaim(db, 11 * 60_000); + const transaction = db.transaction.bind(db); + let entered!: () => void, release!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const intercepted = vi.spyOn(db, "transaction").mockImplementationOnce(async callback => { + entered(); await gate; return transaction(callback); + }); + const sweep = gateway.settleInterruptedActionReviews(); + await ready; + const updatedAt = new Date(); + try { + await writeToolReviewChange(db, fixture.company.id, tx => tx.update(toolActionRequests).set({ updatedAt }) + .where(eq(toolActionRequests.id, pending.id))); + } finally { release(); } + await sweep; intercepted.mockRestore(); + expect((await db.select().from(toolActionRequests))[0].status).toBe("executing"); + await startInterruption(); + expect(coordinator!.nextWakeAt()).toBe(updatedAt.getTime() + 10 * 60_000); + }); + + it("settles more than one page of interrupted claims", async () => { + const pending = await savedClaim(db, 11 * 60_000); + await db.insert(toolActionRequests).values(Array.from({ length: 101 }, () => ({ ...pending, + id: randomUUID(), interactionId: null, status: "executing" as const }))); + await startInterruption(); await quiet(); + const rows = await db.select().from(toolActionRequests); + expect(rows).toHaveLength(102); expect(rows.every(row => row.status === "failed")).toBe(true); + }); + }); + describe("approved review recovery worker", () => { async function startRecovery() { coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, diff --git a/server/src/index.ts b/server/src/index.ts index e9fc24fcf0b..50627307a54 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerToolReviewInterruptionWork } from "./services/tool-review-interruption-work.js"; import { registerToolReviewRecoveryWork } from "./services/tool-review-recovery-work.js"; import { registerToolReviewExpiryWork } from "./services/tool-review-expiry-work.js"; import { registerGatewayCredentialExpiryWork } from "./services/gateway-credential-expiry-work.js"; @@ -1479,7 +1480,13 @@ async function startServerWithDatabaseTeardown( } else { await app.locals.toolGateway.recoverApprovedActionReviews(); } - return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false }); + if (config.heartbeatSchedulerEnabled) { + await registerToolReviewInterruptionWork(deliveryWork, app.locals.toolGateway, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready; + } else { + await app.locals.toolGateway.settleInterruptedActionReviews(); + } + return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false, includeInterrupted: false }); }).catch((err: unknown) => logger.error({ err }, "startup tool review recovery failed")); await timeStartupRecovery("gateway_sessions", () => config.heartbeatSchedulerEnabled ? registerGatewayCredentialExpiryWork(deliveryWork, app.locals.toolGateway, @@ -1910,7 +1917,7 @@ async function startServerWithDatabaseTeardown( logger.error({ err }, "periodic tool connection health sweep failed"); })); - trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); + trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false, includeInterrupted: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 923ae4b7979..9c943677dac 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + toolReviewInterruption: "tool-review-interruption", toolReviewRecovery: "tool-review-recovery", toolReviewExpiry: "tool-review-expiry", gatewayCredentialExpiry: "gateway-credential-expiry", diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 57763f8b8d5..2d94a361de2 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1,7 +1,8 @@ +import { toolReviewInterruptionService } from "./tool-review-interruption.js"; import { bindVerifiedToolApprovalArguments, claimToolTestApproval, releaseToolApproval, persistToolTestOutcome, startToolInvocation, type ToolApprovalClaim } from "./tool-approval-execution.js"; import { approvedToolReviewCompanies, prepareApprovedToolReview, recoverApprovedToolReviews } from "./tool-review-recovery.js"; import { toolReviewExpiryService } from "./tool-review-expiry.js"; -import { writeToolReviewChange } from "./tool-review-work-notifications.js"; +import { notifyToolReviewChange, writeToolReviewChange } from "./tool-review-work-notifications.js"; import { restConnectorFor } from "./connectors/rest-registry.js"; import type { runPromptInjectionProvider } from "./prompt-injection-provider.js"; import { promptInjectionPreventionService } from "./prompt-injection-prevention.js"; @@ -46,7 +47,6 @@ import { asc, desc, eq, - gt, inArray, isNull, lte, @@ -7987,6 +7987,7 @@ export function createToolGatewayService( const [actionRequest] = await tx .select({ + companyId: toolActionRequests.companyId, status: toolActionRequests.status, updatedAt: toolActionRequests.updatedAt, }) @@ -8001,6 +8002,7 @@ export function createToolGatewayService( return false; } + await notifyToolReviewChange(tx, actionRequest.companyId); await tx .update(toolInvocations) .set({ @@ -8476,10 +8478,10 @@ export function createToolGatewayService( }) .where(eq(toolInvocations.id, invocation.id)); void emitConnectionInvoked(db, invocation.id); - await db + await writeToolReviewChange(db, claimed.companyId, tx => tx .update(toolActionRequests) .set({ status: "executed", resolvedAt: now, updatedAt: now }) - .where(eq(toolActionRequests.id, claimed.id)); + .where(eq(toolActionRequests.id, claimed.id))); await reflectToolActionInteractionLifecycle({ actionRequestId: claimed.id, status: "executed", @@ -8818,6 +8820,11 @@ export function createToolGatewayService( }; } + const reviewInterruption = toolReviewInterruptionService(db, async settled => { + void emitConnectionInvoked(db, settled.invocationId); + await reflectToolActionInteractionLifecycle(settled); + }); + const reviewExpiry = toolReviewExpiryService(db, async expired => { void emitConnectionInvoked(db, expired.invocationId); await reflectToolActionInteractionLifecycle(expired); @@ -9910,6 +9917,8 @@ export function createToolGatewayService( return buildTestCallStatus(actionRequest, invocation); }, + settleInterruptedActionReviews: reviewInterruption.sweep, + nextActionReviewInterruptionAt: reviewInterruption.next, sweepExpiredActionReviews: reviewExpiry.sweep, nextActionReviewExpiryAt: reviewExpiry.next, actionReviewExpiryDeadlines: reviewExpiry.deadlines, @@ -9941,7 +9950,7 @@ export function createToolGatewayService( }, companyId); }, - async sweepActionReviews(input: { includeExpiry?: boolean; includeApproved?: boolean } = {}) { + async sweepActionReviews(input: { includeExpiry?: boolean; includeApproved?: boolean; includeInterrupted?: boolean } = {}) { // A process may stop between persisting a provider outcome and updating the // feed projection. Reconcile from authoritative rows before delivering it. const unreflected = await db @@ -9976,62 +9985,9 @@ export function createToolGatewayService( errorMessage: invocation.errorMessage, resultSummary: invocation.resultSummary?.summary, }); - const now = new Date(); - const staleAt = new Date(now.getTime() - 10 * 60_000); - let cursor: string | undefined; - let scanned = input.includeExpiry === false ? 0 : (await reviewExpiry.sweep(now)).scanned; + let scanned = input.includeExpiry === false ? 0 : (await reviewExpiry.sweep()).scanned; if (input.includeApproved !== false) scanned += (await this.recoverApprovedActionReviews()).scanned; - for (;;) { - const rows = await db - .select() - .from(toolActionRequests) - .where( - and( - eq(toolActionRequests.status, "executing"), - lte(toolActionRequests.updatedAt, staleAt), - cursor ? gt(toolActionRequests.id, cursor) : undefined, - ), - ) - .orderBy(asc(toolActionRequests.id)) - .limit(100); - for (const row of rows) { - const status = "failed"; - const errorCode = "tool_execution_outcome_unknown"; - const errorMessage = "Execution was interrupted; the external outcome is unknown. Inspect the provider before retrying."; - const [changed] = await db - .update(toolActionRequests) - .set({ status, resolvedAt: now, updatedAt: now }) - .where( - and( - eq(toolActionRequests.id, row.id), - eq(toolActionRequests.status, row.status), - eq(toolActionRequests.updatedAt, row.updatedAt), - ), - ) - .returning(); - if (!changed) continue; - await db - .update(toolInvocations) - .set({ - status: "failed", - errorCode, - errorMessage, - completedAt: now, - updatedAt: now, - }) - .where(eq(toolInvocations.id, row.invocationId)); - void emitConnectionInvoked(db, row.invocationId); - await reflectToolActionInteractionLifecycle({ - actionRequestId: row.id, - status, - errorCode, - errorMessage, - }); - } - scanned += rows.length; - if (rows.length < 100) break; - cursor = rows[rows.length - 1].id; - } + if (input.includeInterrupted !== false) scanned += (await reviewInterruption.sweep()).scanned; return { scanned }; }, @@ -11251,7 +11207,8 @@ export function createToolGatewayService( }) .where(eq(toolInvocations.id, invocationId)); if (input.approvedActionRequestId) { - const [executedRequest] = await db + const actionRequestId = input.approvedActionRequestId; + const [executedRequest] = await writeToolReviewChange(db, session.companyId, tx => tx .update(toolActionRequests) .set({ status: "executed", @@ -11260,11 +11217,11 @@ export function createToolGatewayService( }) .where( and( - eq(toolActionRequests.id, input.approvedActionRequestId), + eq(toolActionRequests.id, actionRequestId), eq(toolActionRequests.status, "executing"), ), ) - .returning({ id: toolActionRequests.id }); + .returning({ id: toolActionRequests.id })); if (executedRequest) { await reflectToolActionInteractionLifecycle({ actionRequestId: executedRequest.id, @@ -11384,7 +11341,8 @@ export function createToolGatewayService( .where(eq(toolInvocations.id, invocationId)); void emitConnectionInvoked(db, invocationId); if (input.approvedActionRequestId) { - const [failedRequest] = await db + const actionRequestId = input.approvedActionRequestId; + const [failedRequest] = await writeToolReviewChange(db, session.companyId, tx => tx .update(toolActionRequests) .set({ status: "failed", @@ -11393,11 +11351,11 @@ export function createToolGatewayService( }) .where( and( - eq(toolActionRequests.id, input.approvedActionRequestId), + eq(toolActionRequests.id, actionRequestId), eq(toolActionRequests.status, "executing"), ), ) - .returning({ id: toolActionRequests.id }); + .returning({ id: toolActionRequests.id })); if (failedRequest) { await reflectToolActionInteractionLifecycle({ actionRequestId: failedRequest.id, diff --git a/server/src/services/tool-review-interruption-work.ts b/server/src/services/tool-review-interruption-work.ts new file mode 100644 index 00000000000..9053cb1c4dd --- /dev/null +++ b/server/src/services/tool-review-interruption-work.ts @@ -0,0 +1,16 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { createToolGatewayService } from "./tool-gateway.js"; + +export function registerToolReviewInterruptionWork( + coordinator: ReturnType, + gateway: Pick, "settleInterruptedActionReviews" | "nextActionReviewInterruptionAt">, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.toolReviewInterruption, { + retryMs: 30_000, notificationCoalesceMs: 30_000, canRun, + run: () => gateway.settleInterruptedActionReviews(), + next: () => gateway.nextActionReviewInterruptionAt(), + }); +} diff --git a/server/src/services/tool-review-interruption.ts b/server/src/services/tool-review-interruption.ts new file mode 100644 index 00000000000..de8769f3da1 --- /dev/null +++ b/server/src/services/tool-review-interruption.ts @@ -0,0 +1,61 @@ +import { and, asc, eq, gt, lte } from "drizzle-orm"; +import { toolActionRequests, toolInvocations, type Db } from "@paperclipai/db"; +import { writeToolReviewChange } from "./tool-review-work-notifications.js"; + +const STALE_EXECUTION_MS = 10 * 60_000; + +export function toolReviewInterruptionService(db: Db, onSettled: (input: { + actionRequestId: string; + invocationId: string; + status: "executed" | "failed"; + errorCode: string | null; + errorMessage: string | null; + resultSummary?: string; +}) => Promise) { + return { + async next(): Promise { + const [row] = await db.select({ at: toolActionRequests.updatedAt }).from(toolActionRequests) + .where(eq(toolActionRequests.status, "executing")).orderBy(asc(toolActionRequests.updatedAt)).limit(1); + return row ? row.at.getTime() + STALE_EXECUTION_MS : null; + }, + async sweep(now = new Date()) { + const staleAt = new Date(now.getTime() - STALE_EXECUTION_MS); + let cursor: string | undefined; + let scanned = 0; + for (;;) { + const rows = await db.select().from(toolActionRequests).where(and( + eq(toolActionRequests.status, "executing"), lte(toolActionRequests.updatedAt, staleAt), + cursor ? gt(toolActionRequests.id, cursor) : undefined, + )).orderBy(asc(toolActionRequests.id)).limit(100); + for (const row of rows) { + const settled = await writeToolReviewChange(db, row.companyId, async tx => { + const [invocation] = await tx.select().from(toolInvocations) + .where(eq(toolInvocations.id, row.invocationId)).for("update"); + const [request] = await tx.select().from(toolActionRequests) + .where(eq(toolActionRequests.id, row.id)).for("update"); + if (!invocation || request?.status !== "executing" || + request.updatedAt.getTime() !== row.updatedAt.getTime()) return null; + const completed = ["succeeded", "failed", "denied", "timed_out", "rate_limited", "cancelled"].includes(invocation.status); + const status = invocation.status === "succeeded" ? "executed" as const : "failed" as const; + const errorCode = completed ? invocation.errorCode : "tool_execution_outcome_unknown"; + const errorMessage = completed ? invocation.errorMessage + : "Execution was interrupted; the external outcome is unknown. Inspect the provider before retrying."; + if (!completed) { + await tx.update(toolInvocations).set({ status: "failed", errorCode, errorMessage, + completedAt: now, updatedAt: now }).where(eq(toolInvocations.id, row.invocationId)); + } + await tx.update(toolActionRequests).set({ status, resolvedAt: invocation.completedAt ?? now, updatedAt: now }) + .where(eq(toolActionRequests.id, row.id)); + return { actionRequestId: row.id, invocationId: row.invocationId, status, + errorCode, errorMessage, resultSummary: invocation.resultSummary?.summary }; + }); + if (settled) await onSettled(settled); + } + scanned += rows.length; + if (rows.length < 100) break; + cursor = rows[rows.length - 1].id; + } + return { scanned }; + }, + }; +} diff --git a/server/src/services/tool-review-work-notifications.ts b/server/src/services/tool-review-work-notifications.ts index 1fc52d02b56..44957fefc1b 100644 --- a/server/src/services/tool-review-work-notifications.ts +++ b/server/src/services/tool-review-work-notifications.ts @@ -4,6 +4,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio export async function notifyToolReviewChange(transaction: object, companyId: string) { await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewExpiry, companyId); await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewRecovery, companyId); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewInterruption, companyId); } export function writeToolReviewChange(db: Db, companyId: string, write: (transaction: Db) => Promise): Promise { From 2c70d04bf6e0c4f83e5472d30ce7aac8789779e8 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 16:38:23 -0500 Subject: [PATCH 16/41] feat(evals): account for explicit Daytona model requests (#17936) ## Thinking Path > - Paperclip evals need complete model-spend evidence. > - The bounded Product E2E controller already owns request admission and token receipts. > - Its loopback endpoints cannot serve a Daytona agent. > - Explicit remote cases therefore need a scoped transport to the same controller. > - This PR exposes only authenticated model and OTLP endpoints through one owned quick tunnel. > - Remote requests keep the same budget, request identity, usage journal and cancellation drain. ## Linked Issues or Issue Description Refs #17928, #17929, #17523. **What happened?** The operations manifest and fixture reject Daytona. The parent gateway and native trace relay bind loopback, so normal remote agent traffic cannot use their cost accounting. **Expected behavior** An explicit native marker or feedback case on Daytona can route requests through the bounded parent. Upstream provider and Braintrust keys stay on that controller. Every paid request keeps its usage receipt and the attempt retains ordinary feedback and cleanup evidence. **Steps to reproduce** Use the explicit `remoteGateway: "cloudflare"` manifest option for a native Codex or Claude Daytona marker/feedback case. Without that option, the remote selection remains rejected. The parent verifies its public health nonce before fixture setup. ## What Changed - Add a deadline-limited bridge with exact model/OTLP paths, separate scoped tokens, streamed responses and owned tunnel cleanup. - Start one HTTPS quick tunnel with a private home and no provider credentials or ambient Cloudflare configuration. Do not retry creation or paid work. - Admit only explicit native Daytona marker/feedback cells and require the provisioning credential before opening the journal. - Bind fixture model/OTLP endpoints to the assigned remote origin. Keep provisioning access in the fixture/server controller. - Preserve final token usage when a remote client disconnects during shutdown. - Document the manual route. Daily and weekly selections remain local. ## Verification - Full Product E2E unit suite: 4,370 passed and 2 skipped across 206 files, plus 137 Node checks. - Focused remote/usage/operations/trace checks: 71 tests passed; final bridge/drain checks: 15 passed. - Product E2E typecheck passed. - Tests cover missing tunnel executable, expiry, authentication, blocked paths, token separation, missing provisioning credentials, endpoint assignment and listener cleanup. - The cancellation regression uses the real gateway and ledger with a synthetic upstream. It retains the final receipt once after remote disconnect. - Latest source and architecture/cleanup checks: 167 passed; eval typecheck passed. Both reviews are 5/5 and exact-head CI is green. Live qualification follows merge. ## Risks The explicit route sends synthetic eval requests and scoped attempt tokens through Cloudflare. Those tokens expire with the existing campaign deadline and budget. Upstream keys stay on the controller. Public readiness has a finite wait; failure stops before fixture setup. The operator still needs the existing immutable Daytona image and verified native artifacts. Source and deterministic tests do not establish live remote acceptance. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/OPERATIONS.md | 39 ++++++ tests/runner-e2e/braintrust.test.ts | 10 ++ tests/runner-e2e/braintrust.ts | 7 +- tests/runner-e2e/eval-gateway-drain.test.ts | 17 +++ tests/runner-e2e/eval-remote-gateway.test.ts | 101 ++++++++++++++++ tests/runner-e2e/eval-remote-gateway.ts | 119 +++++++++++++++++++ tests/runner-e2e/harness-env.ts | 13 +- tests/runner-e2e/launch.ts | 3 + tests/runner-e2e/live-fixtures.ts | 6 +- tests/runner-e2e/operations.test.ts | 44 ++++++- tests/runner-e2e/operations.ts | 25 ++-- tests/runner-e2e/support.test.ts | 9 ++ 12 files changed, 377 insertions(+), 16 deletions(-) create mode 100644 tests/runner-e2e/eval-remote-gateway.test.ts create mode 100644 tests/runner-e2e/eval-remote-gateway.ts diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index 1481b4451b1..be61693fb07 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -909,3 +909,42 @@ reservations and model-readiness status separately. It does not release any historical hold or certify a full-lifecycle vendor cost bound. Missing model receipts, conflicts, active requests, excess commitments, and the existing product, cleanup, feedback and trace checks still fail qualification. + + +### Explicit bounded Daytona gateway qualification + +A manual manifest can set `remoteGateway: "cloudflare"` for the native Codex +or Claude marker/feedback Daytona cells. Install `cloudflared` on the controller +PATH, or set `PAPERCLIP_EVAL_CLOUDFLARED` to its executable path. Empty overrides +use the PATH default. The parent starts its existing model +gateway and native OTLP relay, then exposes only their authenticated endpoints +through one owned HTTPS quick tunnel. It verifies a fresh health nonce before +fixture setup, waits up to 120 seconds for registration and 120 seconds for +public readiness, and does not retry tunnel creation or provider work. + +Only attempt-scoped tokens leave the controller. Provider and Braintrust keys +stay in the parent; the fixture/server retains the existing Daytona provisioning +credential separately. The model gateway still enforces its campaign allowance, +request count, concurrency, deadline and receipt journal. The bridge streams +responses and preserves terminal-usage draining after a client disconnect. +The tunnel receives no ambient provider credentials or Cloudflare config. Its +exposure closes at the campaign deadline or four hours after startup, whichever +comes first, and on attempt cleanup. Owned gateway, +relay and tunnel cleanup failures remain failures. A configured remote route +is not live acceptance; retain the actual Daytona result, cleanup, feedback, +native traces and complete model receipts. + +The Daytona image runs Linux/x64. ARM or macOS controllers use the image's +verified daemon by default; only an explicit cross-compiled remote override +replaces it. + +Daily and weekly selections remain local. This explicit remote option adds no +scheduled coverage or public report publication. + +The worker generates the attempt's trace parent. The relay validates the project +in both the header and payload, but does not pin that worker-generated parent. +Treat the attempt OTLP token as authority to submit trace attribution within +this eval project. Keep it limited to the trusted harness/provider runtime; +never expose it to test prompts or unrelated clients. The project check is not +an attempt-isolation guarantee. Indexing verification is a separate retained +or scheduled check; the manual launcher alone does not perform it. diff --git a/tests/runner-e2e/braintrust.test.ts b/tests/runner-e2e/braintrust.test.ts index 118621d3ffe..da60c202173 100644 --- a/tests/runner-e2e/braintrust.test.ts +++ b/tests/runner-e2e/braintrust.test.ts @@ -211,3 +211,13 @@ it("retains finalization export failures without changing the recorded product r }, transport)).toMatchObject({ status: "failed", errorCode: "invalid_finalization_context" }); expect(transport).not.toHaveBeenCalled(); }); + + +it("binds remote OTLP to the explicitly assigned eval gateway origin", () => { + const origin = "https://synthetic-eval.trycloudflare.com"; + const env = { PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: projectId, BRAINTRUST_API_KEY: "synthetic-token", + PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN: origin, PAPERCLIP_E2E_OTLP_RELAY_URL: `${origin}/otel` }; + expect(evalBraintrustConfig(env)).toMatchObject({ remoteOrigin: origin, endpointBaseUrl: `${origin}/otel` }); + expect(() => evalBraintrustConfig({ ...env, PAPERCLIP_E2E_OTLP_RELAY_URL: "https://different-eval.trycloudflare.com/otel" })).toThrow(); + expect(buildPaperclipServerEnvironment(env).PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN).toBeUndefined(); +}); diff --git a/tests/runner-e2e/braintrust.ts b/tests/runner-e2e/braintrust.ts index adb5d94e4b9..83b7c2091f5 100644 --- a/tests/runner-e2e/braintrust.ts +++ b/tests/runner-e2e/braintrust.ts @@ -1,13 +1,15 @@ import { randomBytes } from "node:crypto"; import { createRequire } from "node:module"; +import { isEvalRemoteOrigin } from "./eval-remote-gateway.js"; import type { RunnerApi } from "./api.js"; import type { FailureClass, RunnerE2EResult } from "./types.js"; export const EVAL_OBSERVABILITY_SECRET_NAMES = ["BRAINTRUST_API_KEY"] as const; const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; -export interface EvalBraintrustConfig { projectId: string; apiKey: string; endpointBaseUrl?: string } +export interface EvalBraintrustConfig { projectId: string; apiKey: string; endpointBaseUrl?: string; remoteOrigin?: string } function evalOtlpEndpoint(config: EvalBraintrustConfig) { - if (config.endpointBaseUrl !== undefined && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(config.endpointBaseUrl)) { + if (config.remoteOrigin !== undefined && !isEvalRemoteOrigin(config.remoteOrigin)) throw new Error("Invalid eval remote OTLP origin"); + if (config.endpointBaseUrl !== undefined && !(config.remoteOrigin && config.endpointBaseUrl === `${config.remoteOrigin}/otel`) && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(config.endpointBaseUrl)) { throw new Error("Eval OTLP relay must use the parent-owned loopback endpoint"); } if (config.endpointBaseUrl && Number(new URL(config.endpointBaseUrl).port) > 65535) throw new Error("Invalid eval OTLP relay port"); @@ -20,6 +22,7 @@ export function evalBraintrustConfig(env: NodeJS.ProcessEnv): EvalBraintrustConf throw new Error("Eval Braintrust export requires an explicit project UUID and API key"); } const config = { projectId, apiKey: env.BRAINTRUST_API_KEY.trim(), + ...(env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN ? { remoteOrigin: env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN } : {}), ...(env.PAPERCLIP_E2E_OTLP_RELAY_URL ? { endpointBaseUrl: env.PAPERCLIP_E2E_OTLP_RELAY_URL } : {}) }; evalOtlpEndpoint(config); return config; diff --git a/tests/runner-e2e/eval-gateway-drain.test.ts b/tests/runner-e2e/eval-gateway-drain.test.ts index 24e6f43e776..9c5a73a06cc 100644 --- a/tests/runner-e2e/eval-gateway-drain.test.ts +++ b/tests/runner-e2e/eval-gateway-drain.test.ts @@ -3,6 +3,7 @@ import { tmpdir } from "node:os"; import path from "node:path"; import { afterEach, expect, it, vi } from "vitest"; import { openEvalCostLedger, readEvalCostLedger } from "./cost-ledger.js"; +import { startEvalRemoteBridge } from "./eval-remote-gateway.js"; import { forwardEvalRequest, startEvalGateway, type EvalGatewayReceipt, type EvalGatewayRoute } from "./eval-gateway.js"; const cleanups: Array<() => Promise> = []; @@ -181,3 +182,19 @@ it.each(["persist", "observe", "finish"] as const)("records a disconnect during expect(f.ledger.snapshot()).toMatchObject({ knownUsd: "0.000070000", heldUsd: "0.000000000", unknownRequests: 0 }); expect(transport).toHaveBeenCalledOnce(); }); + +it("retains terminal usage when a remote bridge client disconnects during shutdown", async () => { + const f = await fixture(), started = deferred(), release = deferred(); + const transport = vi.fn(async () => { started.resolve(); await release.promise; return finalResponse("openai"); }); + const gateway = await startEvalGateway({ ...f, transport }); cleanups.push(() => gateway.close()); + const bridge = await startEvalRemoteBridge({ gateway, deadlineAt: f.deadlineAt }); cleanups.push(() => bridge.close()); + const client = new AbortController(); + const result = fetch(bridge.baseUrl + "/responses", { method: "POST", signal: client.signal, + headers: { authorization: `Bearer ${gateway.token}` }, body: JSON.stringify({ model: "fixture", input: "hello" }) }).catch(error => error); + await started.promise; client.abort(); await bridge.close(); + const closing = gateway.close(); release.resolve(); await closing; + expect(await result).not.toBeInstanceOf(Response); + expect(transport).toHaveBeenCalledOnce(); + expect(f.receipts.at(-1)).toMatchObject({ clientDisconnected: true, final: true, usageComplete: true, costUsd: "0.000070000" }); + expect(f.ledger.snapshot()).toMatchObject({ knownUsd: "0.000070000", heldUsd: "0.000000000", unknownRequests: 0, inflight: 0 }); +}); diff --git a/tests/runner-e2e/eval-remote-gateway.test.ts b/tests/runner-e2e/eval-remote-gateway.test.ts new file mode 100644 index 00000000000..befe8d1e366 --- /dev/null +++ b/tests/runner-e2e/eval-remote-gateway.test.ts @@ -0,0 +1,101 @@ +import { mkdtemp, readFile, writeFile, rm, readdir } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { createServer } from "node:http"; +import { once } from "node:events"; +import { expect, it, vi } from "vitest"; +import { isEvalRemoteOrigin, startEvalRemoteBridge, startEvalRemoteGateway } from "./eval-remote-gateway.js"; + +it("forwards only authenticated model/OTLP routes and closes owned exposure", async () => { + const calls: Array<{ url: string; auth?: string; body: string }> = []; + const upstream = createServer(async (request, response) => { + let body = ""; for await (const chunk of request) body += chunk; + calls.push({ url: request.url!, auth: request.headers.authorization, body }); + response.writeHead(200, { "content-type": "text/event-stream" }); response.end("data: synthetic\n\n"); + }); + upstream.listen(0, "127.0.0.1"); await once(upstream, "listening"); + const address = upstream.address(); if (!address || typeof address === "string") throw new Error("Fixture did not bind"); + const url = `http://127.0.0.1:${address.port}`, modelToken = "a".repeat(64), otlpToken = "b".repeat(64); + const bridge = await startEvalRemoteBridge({ gateway: { baseUrl: url, token: modelToken }, + observability: { baseUrl: url + "/otel", token: otlpToken }, deadlineAt: Date.now() + 60_000 }); + try { + expect(await (await fetch(bridge.baseUrl + "/health")).json()).toEqual({ nonce: bridge.nonce }); + for (const path of ["/v1/responses", "/v1/messages?beta=true", "/otel/v1/traces", "/otel/v1/logs"]) { + expect((await fetch(bridge.baseUrl + path, { method: "POST" })).status).toBe(401); + const token = path.startsWith("/otel") ? otlpToken : modelToken; + const response = await fetch(bridge.baseUrl + path, { method: "POST", headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, body: "{}" }); + expect(response.status).toBe(200); expect(await response.text()).toBe("data: synthetic\n\n"); + } + expect(calls).toHaveLength(4); + expect(calls.map(call => call.url)).toEqual(["/v1/responses", "/v1/messages?beta=true", "/otel/v1/traces", "/otel/v1/logs"]); + expect(calls.map(call => call.auth)).toEqual([`Bearer ${modelToken}`, `Bearer ${modelToken}`, `Bearer ${otlpToken}`, `Bearer ${otlpToken}`]); + expect((await fetch(bridge.baseUrl + "/admin", { method: "POST", headers: { authorization: `Bearer ${modelToken}` } })).status).toBe(404); + expect((await fetch(bridge.baseUrl + "/otel/v1/traces", { method: "POST", headers: { authorization: `Bearer ${modelToken}` } })).status).toBe(401); + await bridge.close(); await expect(fetch(bridge.baseUrl + "/health")).rejects.toThrow(); + } finally { await bridge.close(); upstream.closeAllConnections(); await new Promise(resolve => upstream.close(() => resolve())); } +}); + +it("admits only HTTPS quick-tunnel origins and bounded loopback destinations", async () => { + expect(isEvalRemoteOrigin("https://synthetic-fixture.trycloudflare.com")).toBe(true); + for (const value of ["https://foreign.invalid", "http://synthetic.trycloudflare.com", "https://synthetic.trycloudflare.com/path", "https://synthetic.trycloudflare.com?token=private"]) expect(isEvalRemoteOrigin(value)).toBe(false); + await expect(startEvalRemoteBridge({ gateway: { baseUrl: "https://foreign.invalid", token: "a".repeat(64) }, deadlineAt: Date.now() + 60_000 })).rejects.toThrow("policy"); +}); + + +it("fails before remote admission when the tunnel executable is unavailable", async () => { + await expect(startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) }, + deadlineAt: Date.now() + 5000 }, { PAPERCLIP_EVAL_CLOUDFLARED: "/missing/synthetic-cloudflared", PATH: "" })) + .rejects.toThrow("Remote eval tunnel executable unavailable"); +}); + +it("closes remote exposure at its original campaign deadline", async () => { + const bridge = await startEvalRemoteBridge({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) }, deadlineAt: Date.now() + 50 }); + try { + await new Promise(resolve => setTimeout(resolve, 100)); + await expect(fetch(bridge.baseUrl + "/health")).rejects.toThrow(); + } finally { await bridge.close(); } +}); + + +it.each(["explicit", "empty-override"])("waits for registration and the correct public nonce, then terminates the owned tunnel: %s", async mode => { + const root = await mkdtemp(path.join(tmpdir(), "eval-fake-tunnel-")); + const executable = path.join(root, "cloudflared"), state = path.join(root, "state.json"), stopped = path.join(root, "stopped"); + const origin = "https://synthetic-owned.trycloudflare.com"; + const script = `#!${process.execPath} +const fs = require("node:fs"); +fs.writeFileSync(${JSON.stringify(state)}, JSON.stringify({baseUrl:process.argv.at(-1),pid:process.pid,env:process.env})); +process.stderr.write("${origin}\\n"); +setTimeout(() => process.stderr.write("Registered tunnel connection\\n"), 20); +process.on("SIGTERM", () => { fs.writeFileSync(${JSON.stringify(stopped)}, "stopped"); process.exit(0); }); +setInterval(() => {}, 1000); +`; + await writeFile(executable, script, { mode: 0o700 }); + const realFetch = globalThis.fetch; let healthChecks = 0; + const fetchSpy = vi.spyOn(globalThis, "fetch").mockImplementation(async url => { + expect(String(url)).toBe(origin + "/health"); + if (++healthChecks === 1) return new Response(JSON.stringify({ nonce: "foreign" }), { status: 200 }); + const { baseUrl } = JSON.parse(await readFile(state, "utf8")); + return realFetch(baseUrl + "/health"); + }); + let remote: Awaited> | undefined; + try { + remote = await startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) }, + deadlineAt: Date.now() + 10_000 }, { PAPERCLIP_EVAL_CLOUDFLARED: mode === "explicit" ? executable : "", PATH: root, OPENAI_API_KEY: "must-not-leave" }); + expect(remote.baseUrl).toBe(origin); expect(healthChecks).toBe(2); + const { baseUrl, pid, env } = JSON.parse(await readFile(state, "utf8")); + expect(env.OPENAI_API_KEY).toBeUndefined(); expect(env.HOME).not.toBe(process.env.HOME); + await remote.close(); + expect(await readFile(stopped, "utf8")).toBe("stopped"); + expect(() => process.kill(pid, 0)).toThrow(); + await expect(realFetch(baseUrl + "/health")).rejects.toThrow(); + await expect(readFile(path.join(env.HOME, "anything"))).rejects.toThrow(); + } finally { fetchSpy.mockRestore(); await remote?.close(); await rm(root, { recursive: true, force: true }); } +}, 15_000); + + +it("cleans the private tunnel home after synchronous spawn rejection", async () => { + const before = new Set((await readdir(tmpdir())).filter(name => name.startsWith("eval-cloudflare-"))); + await expect(startEvalRemoteGateway({ gateway: { baseUrl: "http://127.0.0.1:1", token: "a".repeat(64) }, + deadlineAt: Date.now() + 60_000 }, { PAPERCLIP_EVAL_CLOUDFLARED: "invalid\0executable", PATH: "" })).rejects.toThrow(); + expect((await readdir(tmpdir())).filter(name => name.startsWith("eval-cloudflare-") && !before.has(name))).toEqual([]); +}); diff --git a/tests/runner-e2e/eval-remote-gateway.ts b/tests/runner-e2e/eval-remote-gateway.ts new file mode 100644 index 00000000000..2d5d1d85054 --- /dev/null +++ b/tests/runner-e2e/eval-remote-gateway.ts @@ -0,0 +1,119 @@ +import { spawn } from "node:child_process"; +import { randomBytes, timingSafeEqual } from "node:crypto"; +import { once } from "node:events"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { createServer, request as httpRequest } from "node:http"; + +export const isEvalRemoteOrigin = (value: string) => /^https:\/\/[a-z0-9-]+\.trycloudflare\.com$/.test(value); + +export async function startEvalRemoteBridge(input: { + gateway: { baseUrl: string; token: string }; + observability?: { baseUrl: string; token: string }; + deadlineAt: number; +}) { + if (!Number.isFinite(input.deadlineAt) || input.deadlineAt <= Date.now() || input.deadlineAt > Date.now() + 4 * 3_600_000 + || !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}$/.test(input.gateway.baseUrl) + || input.observability && !/^http:\/\/127\.0\.0\.1:[1-9][0-9]{0,4}\/otel$/.test(input.observability.baseUrl) + || !/^[a-f0-9]{64}$/.test(input.gateway.token) + || input.observability && !/^[a-f0-9]{64}$/.test(input.observability.token)) throw new Error("Invalid remote eval bridge policy"); + try { new URL(input.gateway.baseUrl); if (input.observability) new URL(input.observability.baseUrl); } + catch { throw new Error("Invalid remote eval bridge policy"); } + const nonce = randomBytes(32).toString("hex"); + let closed = false; + const server = createServer((incoming, outgoing) => { + if (closed || Date.now() >= input.deadlineAt) { outgoing.writeHead(503).end(); return; } + if (incoming.method === "GET" && incoming.url === "/health") { + outgoing.writeHead(200, { "content-type": "application/json", "cache-control": "no-store" }).end(JSON.stringify({ nonce })); return; + } + const model = ["/responses", "/v1/responses", "/messages", "/messages?beta=true", "/v1/messages", "/v1/messages?beta=true"].includes(incoming.url ?? ""); + const otlp = ["/otel/v1/traces", "/otel/v1/logs"].includes(incoming.url ?? ""); + const destination = model ? input.gateway : otlp ? input.observability : undefined; + if (incoming.method !== "POST" || !destination) { outgoing.writeHead(404).end(); return; } + const auth = String(incoming.headers.authorization ?? "").replace(/^Bearer /, "") || String(incoming.headers["x-api-key"] ?? ""); + const presented = Buffer.from(auth), expected = Buffer.from(destination.token); + if (presented.length !== expected.length || !timingSafeEqual(presented, expected)) { outgoing.writeHead(401).end(); return; } + const suffix = otlp ? incoming.url!.slice("/otel".length) : incoming.url!; + const headers: Record = { authorization: `Bearer ${destination.token}` }; + for (const name of ["content-type", "content-encoding", "anthropic-beta", "anthropic-version", "x-bt-parent"]) { + const value = incoming.headers[name]; if (typeof value === "string") headers[name] = value; + } + const upstream = httpRequest(destination.baseUrl + suffix, { method: "POST", headers }, response => { + if (outgoing.destroyed) { response.destroy(); return; } + outgoing.writeHead(response.statusCode ?? 502, response.headers); response.pipe(outgoing); + response.on("error", () => outgoing.destroy()); + }); + upstream.on("error", () => { if (!outgoing.headersSent) outgoing.writeHead(502).end(); else outgoing.destroy(); }); + incoming.on("aborted", () => upstream.destroy()); + incoming.on("error", () => upstream.destroy()); + outgoing.on("error", () => upstream.destroy()); + outgoing.on("close", () => { if (!outgoing.writableEnded) upstream.destroy(); }); + incoming.pipe(upstream); + }); + server.requestTimeout = 15_000; server.headersTimeout = 10_000; + server.listen(0, "127.0.0.1"); await once(server, "listening"); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Remote eval bridge did not bind loopback"); + let timer: ReturnType; + let closing: Promise | undefined; + const close = () => closing ??= (async () => { + closed = true; clearTimeout(timer); + server.closeAllConnections(); await new Promise(resolve => server.close(() => resolve())); + })(); + timer = setTimeout(() => { void close().catch(() => {}); }, input.deadlineAt - Date.now()); timer.unref(); + return { baseUrl: `http://127.0.0.1:${address.port}`, nonce, close }; +} + +export async function startEvalRemoteGateway(input: Parameters[0], env: NodeJS.ProcessEnv) { + const bridge = await startEvalRemoteBridge(input); + let home: string; + try { home = await mkdtemp(path.join(tmpdir(), "eval-cloudflare-")); } + catch (error) { await bridge.close(); throw error; } + let tunnel: ReturnType | undefined; + let closing: Promise | undefined, deadlineTimer: ReturnType | undefined; + const close = () => closing ??= (async () => { + clearTimeout(deadlineTimer); + const errors: unknown[] = []; + try { await bridge.close(); } catch (error) { errors.push(error); } + try { + if (tunnel?.pid && tunnel.exitCode === null && tunnel.signalCode === null) { + const exited = once(tunnel, "exit").catch(() => {}); + tunnel.kill("SIGTERM"); const timer = setTimeout(() => tunnel?.kill("SIGKILL"), 5000); + try { await exited; } finally { clearTimeout(timer); } + } + } catch (error) { errors.push(error); } + try { await rm(home, { recursive: true, force: true }); } catch (error) { errors.push(error); } + if (errors.length) throw new AggregateError(errors, "Remote eval exposure cleanup failed"); + })(); + try { + tunnel = spawn(env.PAPERCLIP_EVAL_CLOUDFLARED?.trim() || "cloudflared", + ["tunnel", "--no-autoupdate", "--protocol", "http2", "--url", bridge.baseUrl], + { stdio: ["ignore", "pipe", "pipe"], env: { PATH: env.PATH, HOME: home, TMPDIR: home } }); + deadlineTimer = setTimeout(() => { void close().catch(() => {}); }, input.deadlineAt - Date.now()); deadlineTimer.unref(); + const child = tunnel; + const origin = await new Promise((resolve, reject) => { + let output = ""; + const timer = setTimeout(() => finish(new Error("Remote eval tunnel startup timed out")), Math.min(120_000, input.deadlineAt - Date.now())); + const finish = (error?: Error, value?: string) => { clearTimeout(timer); if (error) reject(error); else resolve(value!); }; + child.once("error", () => finish(new Error("Remote eval tunnel executable unavailable"))); + child.once("exit", () => finish(new Error("Remote eval tunnel exited"))); + const read = (chunk: Buffer) => { + output = (output + chunk.toString()).slice(-32768); + const value = output.match(/https:\/\/[a-z0-9-]+\.trycloudflare\.com/)?.[0]; + if (value && output.includes("Registered tunnel connection")) finish(undefined, value); + }; + child.stdout!.on("data", read); child.stderr!.on("data", read); + }); + const readyBy = Math.min(Date.now() + 120_000, input.deadlineAt); + while (Date.now() < readyBy) { + try { + const response = await fetch(origin + "/health", { redirect: "error", signal: AbortSignal.timeout(Math.min(5000, readyBy - Date.now())) }); + if (response.ok && (await response.json() as { nonce?: string }).nonce === bridge.nonce) return { baseUrl: origin, close }; + await response.body?.cancel(); + } catch {} + await new Promise(resolve => setTimeout(resolve, 1000)); + } + throw new Error("Remote eval tunnel not publicly reachable before admission"); + } catch (error) { await close(); throw error; } +} diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index 2ea3cd91a40..9aa865b0c1f 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -2,6 +2,7 @@ import { installedDaytonaPluginKeys } from "./installed-daytona-plugin.js"; import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js"; import { assertInstalledCliSelection, installedCliKeys, usesInstalledCli } from "./installed-cli.js"; import path from "node:path"; +import { isEvalRemoteOrigin } from "./eval-remote-gateway.js"; import { chatNeedsApiTools, isManagedHiringCase } from "./chat-cases.js"; import { CREDENTIAL_NAMES } from "./types.js"; import type { MatrixExecution } from "./types.js"; @@ -14,6 +15,7 @@ const AMBIENT_PAPERCLIP_CREDENTIAL_KEYS = [ "PAPERCLIP_E2E_GATEWAY_TOKEN", "PAPERCLIP_E2E_GATEWAY_PHASE_PATH", "PAPERCLIP_E2E_OTLP_RELAY_URL", + "PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN", "PAPERCLIP_SETUP_TOKEN", "PAPERCLIP_SECRETS_MASTER_KEY", "PAPERCLIP_SECRETS_MASTER_KEY_FILE", @@ -41,6 +43,8 @@ export function applyEvalGatewayEnvironment(env: NodeJS.ProcessEnv, credential: || PROVIDER_SECRET_KEY.test(name) && /KEY|TOKEN|SECRET|PASSWORD/.test(name)) delete env[name]; } delete env.PAPERCLIP_E2E_OTLP_RELAY_URL; + delete env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN; + if (isEvalRemoteOrigin(gateway.baseUrl)) env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN = gateway.baseUrl; env[credential] = gateway.token; env.PAPERCLIP_E2E_GATEWAY_BASE_URL = gateway.baseUrl; env.PAPERCLIP_E2E_GATEWAY_TOKEN = gateway.token; @@ -103,13 +107,13 @@ export function resolvePaperclipRemoteRunnerBinaryForHarness( runnerBinary: string | undefined, configuredPath = process.env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH, platform: NodeJS.Platform = process.platform, + architecture: NodeJS.Architecture = process.arch, ): string | undefined { if (configuredPath?.trim()) return configuredPath; if (!runnerBinary) return undefined; - // Daytona runs Linux. A default debug binary built by a macOS developer is - // Mach-O and cannot be staged into that sandbox. Leave the remote override - // unset so the pinned Daytona image's verified runnerd is discovered instead. - if (platform !== "linux") return undefined; + // The pinned Daytona runtime is Linux/x64. Other controller binaries cannot + // run there; discover its verified runnerd unless an explicit cross-build is set. + if (platform !== "linux" || architecture !== "x64") return undefined; return executions.some( (execution) => execution.profile.generation === "native" && @@ -133,6 +137,7 @@ export function buildRunnerE2EProcessEnvironment( const result = { ...source }; delete result.PAPERCLIP_E2E_GATEWAY_PHASE_PATH; delete result.PAPERCLIP_E2E_OTLP_RELAY_URL; + delete result.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN; // Announcements are unrelated to the scenarios and obscure screenshot evidence. result.PAPERCLIP_ANNOUNCEMENTS_ENABLED = "false"; delete result.OPENCODE_ALLOW_ALL_MODELS; diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index b65d4f9fecb..74ec9179866 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -595,6 +595,9 @@ async function runAttempt(input: { }; if (gateway) { applyEvalGatewayEnvironment(childEnv, execution.profile.credential, gateway); + // The fixture/server controller needs Daytona provisioning access. The + // remote provider still receives only the scoped model and OTLP tokens. + if (execution.environment.id === "daytona") childEnv.DAYTONA_API_KEY = process.env.DAYTONA_API_KEY; } // The database URLs are stripped here and again at the Playwright web-server // boundary so a developer's shell can never redirect this paid test. diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index 348db2d08af..517b25b3d90 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -1,3 +1,4 @@ +import { isEvalRemoteOrigin } from "./eval-remote-gateway.js"; import type { AiConnectionBinding } from "../../packages/shared/src/ai-connections.js"; import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js"; import { NATIVE_COMPLETION_BUDGET_CENTS } from "./native-completion-defaults.js"; @@ -280,8 +281,9 @@ export async function setupLiveFixtures(input: { const gatewayUrl = process.env.PAPERCLIP_E2E_GATEWAY_BASE_URL; const gatewayToken = process.env.PAPERCLIP_E2E_GATEWAY_TOKEN; const budgetedGateway = Boolean(gatewayUrl || gatewayToken); - if (budgetedGateway && (!gatewayUrl || !gatewayToken || !/^http:\/\/127\.0\.0\.1:[0-9]+$/.test(gatewayUrl) - || execution.environment.id !== "local" || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id))) throw new Error("Unqualified fixture gateway route"); + if (budgetedGateway && (!gatewayUrl || !gatewayToken || !(execution.environment.id === "local" ? /^http:\/\/127\.0\.0\.1:[0-9]+$/.test(gatewayUrl) + : isEvalRemoteOrigin(gatewayUrl) && gatewayUrl === process.env.PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN) + || !["local", "daytona"].includes(execution.environment.id) || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id))) throw new Error("Unqualified fixture gateway route"); if (managedHiring || savedCopilot || budgetedGateway) { registry.register({ id: "ai-connection", diff --git a/tests/runner-e2e/operations.test.ts b/tests/runner-e2e/operations.test.ts index 452403765fd..8e03148dfdc 100644 --- a/tests/runner-e2e/operations.test.ts +++ b/tests/runner-e2e/operations.test.ts @@ -3,6 +3,7 @@ import { access, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promise import { createHash } from "node:crypto"; import path from "node:path"; import { tmpdir } from "node:os"; +import * as remoteGateway from "./eval-remote-gateway.js"; import { runnerMatrix } from "./catalog.js"; import { validateEvalOperationsManifest, openEvalOperations, type EvalOperationsManifest } from "./operations.js"; import { setupLiveFixtures } from "./live-fixtures.js"; @@ -17,7 +18,7 @@ function manifest(): EvalOperationsManifest { models: [{ provider: "openai", maximumInputTokens: 1000, maximumOutputTokens: 100, timeoutMs: 1000, price: { provider: "openai", model: cell.profile.model, version: "fixture-only", sourceUrl: "https://provider.invalid/fixture", inputUsdPerMillion: "2", cachedReadUsdPerMillion: "1", outputUsdPerMillion: "10", cacheWrite30mUsdPerMillion: "2", cacheWrite12hUsdPerMillion: "2" } }] }; } -afterEach(() => { vi.unstubAllEnvs(); vi.unstubAllGlobals(); }); +afterEach(() => { vi.restoreAllMocks(); vi.unstubAllEnvs(); vi.unstubAllGlobals(); }); it("admits only the exact bounded local selection and explicitly priced models", () => { expect(validateEvalOperationsManifest(manifest(), [cell], 1)).toMatchObject({ route: "direct" }); expect(() => validateEvalOperationsManifest({ ...manifest(), executionIds: [] }, [cell], 1)).toThrow("selection"); @@ -191,3 +192,44 @@ it.each([ expect(agent!.budgetMonthlyCents).toBe(1_000); } }); + + +it("requires explicit remote gateway admission and infrastructure credentials before opening a journal", async () => { + const remote = runnerMatrix.find(e => e.suite.id === cell.suite.id && e.profile.id === cell.profile.id && e.environment.id === "daytona")!; + const value = { ...manifest(), executionIds: [remote.id], remoteGateway: "cloudflare" as const }; + expect(validateEvalOperationsManifest(value, [remote], 0)).toBe(value); + const root = await mkdtemp(path.join(tmpdir(), "eval-remote-preflight-")); + try { + value.ledgerPath = path.join(root, "ledger.jsonl"); + const file = path.join(root, "manifest.json"); await writeFile(file, JSON.stringify(value)); + await expect(openEvalOperations(file, [remote], 0, { OPENAI_API_KEY: "synthetic" })).rejects.toThrow("credential"); + await expect(access(value.ledgerPath)).rejects.toMatchObject({ code: "ENOENT" }); + } finally { await rm(root, { recursive: true, force: true }); } +}); + +it("gives remote workers scoped gateway/relay URLs and closes both local listeners and the tunnel", async () => { + const remote = runnerMatrix.find(e => e.suite.id === cell.suite.id && e.profile.id === cell.profile.id && e.environment.id === "daytona")!; + const root = await mkdtemp(path.join(tmpdir(), "eval-remote-operation-")); + const close = vi.fn(async () => {}), origin = "https://synthetic-owned.trycloudflare.com"; + const start = vi.spyOn(remoteGateway, "startEvalRemoteGateway").mockResolvedValue({ baseUrl: origin, close }); + const value = { ...manifest(), executionIds: [remote.id], remoteGateway: "cloudflare" as const, ledgerPath: path.join(root, "ledger.jsonl") }; + value.limits.deadlineAt = new Date(Date.now() + 24 * 3_600_000).toISOString(); + const file = path.join(root, "manifest.json"), env = { OPENAI_API_KEY: "provider-private", DAYTONA_API_KEY: "infra-private", + BRAINTRUST_API_KEY: "trace-private", PAPERCLIP_E2E_BRAINTRUST_PROJECT_ID: "11111111-1111-4111-8111-111111111111" }; + let operations: Awaited> | undefined; + try { + await writeFile(file, JSON.stringify(value)); operations = await openEvalOperations(file, [remote], 0, env); + const endpoint = await operations.start(remote, "remote-one", 1); + const worker: NodeJS.ProcessEnv = { ...env }; applyEvalGatewayEnvironment(worker, remote.profile.credential, endpoint); + expect(worker).toMatchObject({ PAPERCLIP_E2E_GATEWAY_BASE_URL: origin, PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN: origin, + PAPERCLIP_E2E_OTLP_RELAY_URL: `${origin}/otel`, OPENAI_API_KEY: endpoint.token, BRAINTRUST_API_KEY: endpoint.observability!.token }); + expect(JSON.stringify(worker)).not.toMatch(/provider-private|infra-private|trace-private/); + const local = start.mock.calls[0]![0]; + expect(local.gateway.baseUrl).toMatch(/^http:\/\/127\.0\.0\.1:/); + expect(local.deadlineAt).toBeLessThanOrEqual(Date.now() + 4 * 3_600_000); + expect(local.deadlineAt).toBeLessThan(Date.parse(value.limits.deadlineAt)); + await operations.close(); operations = undefined; expect(close).toHaveBeenCalledOnce(); + await expect(fetch(local.gateway.baseUrl)).rejects.toThrow(); + await expect(fetch(local.observability!.baseUrl)).rejects.toThrow(); + } finally { await operations?.close(); await rm(root, { recursive: true, force: true }); } +}); diff --git a/tests/runner-e2e/operations.ts b/tests/runner-e2e/operations.ts index ed4e82d1cee..831520dd9e2 100644 --- a/tests/runner-e2e/operations.ts +++ b/tests/runner-e2e/operations.ts @@ -4,6 +4,7 @@ import { createHash } from "node:crypto"; import { createEvalAttemptPhase, readEvalAttemptPhase } from "./attempt-phase.js"; import { EvalSpendState, openEvalCostLedger, type EvalSpendLimits } from "./cost-ledger.js"; import { evalRequestReservation, startEvalGateway, type EvalGatewayRoute, type EvalGatewayReceipt } from "./eval-gateway.js"; +import { startEvalRemoteGateway } from "./eval-remote-gateway.js"; import type { MatrixExecution } from "./types.js"; import { evalBraintrustConfig } from "./braintrust.js"; import { startEvalOtlpRelay } from "./eval-otlp-relay.js"; @@ -17,13 +18,15 @@ export interface EvalOperationsManifest { maxAttemptsPerExecution: number; route: "direct" | "braintrust"; projectId?: string; + remoteGateway?: "cloudflare"; models: Array>; } const fields = (value: object, names: string[]) => { if (!value || typeof value !== "object" || Array.isArray(value) || Object.keys(value).some(name => !names.includes(name))) throw new Error("Invalid eval operations manifest fields"); }; export function validateEvalOperationsManifest(value: EvalOperationsManifest, executions: readonly MatrixExecution[], maxRetries: number) { - fields(value, ["version", "limits", "ledgerPath", "executionIds", "maxAttemptsPerExecution", "route", "projectId", "models"]); + fields(value, ["version", "limits", "ledgerPath", "executionIds", "maxAttemptsPerExecution", "route", "projectId", "remoteGateway", "models"]); + if (value.remoteGateway !== undefined && value.remoteGateway !== "cloudflare") throw new Error("Invalid remote eval gateway policy"); const initial = new EvalSpendState(value.limits).snapshot(); const initialModelAllowance = usdToUnits(value.limits.maxUsd) - usdToUnits(initial.committedUsd); if (value.version !== 1 || !["direct", "braintrust"].includes(value.route) || !path.isAbsolute(value.ledgerPath) @@ -46,9 +49,11 @@ export function validateEvalOperationsManifest(value: EvalOperationsManifest, ex models.set(key, usdToUnits(evalRequestReservation({ ...model, mode: value.route, apiKey: "validation" }))); } for (const execution of executions) { - if (execution.environment.id !== "local" || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id) + if (!(execution.environment.id === "local" || execution.environment.id === "daytona" && value.remoteGateway === "cloudflare" + && ["runner-codex", "runner-acpx-claude"].includes(execution.profile.id)) || !["core-compatibility", "feedback-persistence"].includes(execution.suite.id) || !["runner-codex", "runner-acpx-claude", "legacy-codex", "legacy-claude"].includes(execution.profile.id) - || execution.requiredCredentials.length !== 1) throw new Error(`Unqualified budgeted eval route: ${execution.id}`); + || execution.requiredCredentials.length !== (execution.environment.id === "daytona" ? 2 : 1) + || execution.requiredCredentials.filter(name => name !== "DAYTONA_API_KEY").length !== 1) throw new Error(`Unqualified budgeted eval route: ${execution.id}`); const provider = execution.profile.credential === "ANTHROPIC_API_KEY" ? "anthropic" : "openai"; if (!models.has(`${provider}:${execution.profile.model}`)) throw new Error(`Missing exact model pricing: ${execution.id}`); if (models.get(`${provider}:${execution.profile.model}`)! > initialModelAllowance) { @@ -65,7 +70,8 @@ export async function openEvalOperations(file: string, executions: readonly Matr || Date.parse(manifest.limits.deadlineAt) >= Date.parse(manifest.limits.overhead.expiresAt))) { throw new Error("Operating overhead allowance does not cover the campaign window"); } - const required = manifest.route === "braintrust" ? ["BRAINTRUST_API_KEY"] : executions.map(execution => execution.profile.credential); + const required = [...(manifest.route === "braintrust" ? ["BRAINTRUST_API_KEY"] : executions.map(execution => execution.profile.credential)), + ...(executions.some(execution => execution.environment.id === "daytona") ? ["DAYTONA_API_KEY"] : [])]; if (required.some(name => !env[name]?.trim())) throw new Error("Missing eval gateway upstream credential"); const observability = evalBraintrustConfig({ ...env, PAPERCLIP_E2E_OTLP_RELAY_URL: undefined }); if (observability && manifest.route === "braintrust" && observability.projectId !== manifest.projectId) throw new Error("Eval gateway and OTLP projects differ"); @@ -99,14 +105,19 @@ export async function openEvalOperations(file: string, executions: readonly Matr persist: (receipt: EvalGatewayReceipt) => atomicJson(path.join(receiptDirectory, `${receipt.requestId}.json`), receipt), }); let relay: Awaited> | undefined; + let remote: Awaited> | undefined; try { if (observability) relay = await startEvalOtlpRelay({ projectId: observability.projectId, apiKey: observability.apiKey, campaignId: manifest.limits.campaignId, attemptId, deadlineAt: Math.min(Date.parse(manifest.limits.deadlineAt), Date.now() + 4 * 3_600_000), receiptPath: path.join(`${manifest.ledgerPath}.otlp`, `${createHash("sha256").update(attemptId).digest("hex")}.jsonl`) }); - } catch (error) { await gateway.close(); throw error; } + if (execution.environment.id === "daytona") remote = await startEvalRemoteGateway({ gateway, observability: relay, + deadlineAt: Math.min(Date.parse(manifest.limits.deadlineAt), Date.now() + 4 * 3_600_000) }, env); + } catch (error) { await Promise.allSettled([gateway.close(), relay?.close()]); throw error; } let closing: Promise | undefined; - const endpoint = { ...gateway, phasePath, observability: relay, close: () => closing ??= (async () => { - const results = await Promise.allSettled([gateway.close(), relay?.close()]); + const endpoint = { ...gateway, baseUrl: remote?.baseUrl ?? gateway.baseUrl, phasePath, + observability: relay ? { ...relay, baseUrl: remote ? `${remote.baseUrl}/otel` : relay.baseUrl } : undefined, close: () => closing ??= (async () => { + const remoteResults = await Promise.allSettled([remote?.close()]); + const results = [...remoteResults, ...await Promise.allSettled([gateway.close(), relay?.close()])]; active.delete(endpoint); if (results.some(result => result.status === "rejected")) throw new Error("Eval gateway or OTLP relay cleanup failed"); })() }; diff --git a/tests/runner-e2e/support.test.ts b/tests/runner-e2e/support.test.ts index 7de63dd9068..07c22457fe6 100644 --- a/tests/runner-e2e/support.test.ts +++ b/tests/runner-e2e/support.test.ts @@ -175,6 +175,7 @@ describe("runner E2E local binary resolution", () => { runnerBinary, undefined, "linux", + "x64", ), ).toBe(runnerBinary); expect( @@ -183,6 +184,7 @@ describe("runner E2E local binary resolution", () => { runnerBinary, undefined, "linux", + "x64", ), ).toBeUndefined(); expect( @@ -201,7 +203,14 @@ describe("runner E2E local binary resolution", () => { "darwin", ), ).toBe("/cross-compiled/paperclip-runnerd"); + expect(resolvePaperclipRemoteRunnerBinaryForHarness( + [remoteNativeExecution], runnerBinary, undefined, "linux", "arm64", + )).toBeUndefined(); + expect(resolvePaperclipRemoteRunnerBinaryForHarness( + [remoteNativeExecution], runnerBinary, "/cross-compiled/linux-x64-runnerd", "linux", "arm64", + )).toBe("/cross-compiled/linux-x64-runnerd"); }); + }); describe("runner E2E provider environment", () => { From 0b65926822aacde48729f4b82a5c333f4cadf375 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Tue, 27 Oct 2026 14:39:06 -0700 Subject: [PATCH 17/41] fix(heartbeat): preserve inherited policy after workspace binding (#17924) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Follow-up runs restore saved sessions when execution configuration is unchanged. > - A project can inherit workspace policy from its storage plugin. > - The initial run resolves those defaults before it binds a workspace. > - The first bound follow-up skipped that resolution and changed the session fingerprint. > - This pull request resolves the same effective policy before and after binding. > - Unchanged follow-ups keep their saved session and workspace. ## Linked Issues or Issue Description **What happened?** A repository-backed project with no explicit workspace policy inherits an isolated Git worktree policy from its storage plugin. After the first run binds a workspace, the next run parsed only the raw project policy. It dropped the inherited provider and strategy fields. Core treated this representation change as a `workspaceConfig` change and discarded the saved session, although the workspace had not changed. **Expected behavior** Binding a workspace must not change the effective project policy. Real policy changes must still invalidate session compatibility. The workspace must keep its recorded storage owner. **Steps to reproduce** 1. Install a default project storage plugin with isolated Git worktrees. 2. Create a project with a repository and no explicit workspace policy. 3. Run a task, then submit its first follow-up. 4. Observe a session reset for `workspaceConfig` despite an unchanged workspace. **Paperclip version or commit** Reproduced on master `50b932a142216331b8d476042b454bc123739c40`. Related: #17447 fixed the separate source-free project case. A duplicate PR search found no matching repository-backed fix. ## What Changed - Resolve effective project policy for bound repository-backed workspaces through the same company-scoped resolver as initial admission. The source-free project guard remains. - Add a PostgreSQL heartbeat regression covering initial binding, two follow-ups, a real policy change, and retained provider ownership. - Settle the synthetic adapter fixture as Done and reopen each explicit follow-up. This prevents an automatic review-path recovery from hiding the first bound admission. - Add the verification recipe to the session feature map. ## Verification - The new regression fails before the production change: the first explicit follow-up receives `sessionId: null` instead of the saved session. - All six tests in `server/src/__tests__/heartbeat-session-preparation.test.ts` pass after the fix. - A targeted run across that file and `project-workspace-provider.integration.test.ts` passed 17 cases, including inherited policy, model changes, incompatible explicit policy, provider availability, no fallback after disablement, and retained local/plugin ownership. - Commands: from `server/`, `node ../node_modules/vitest/vitest.mjs run src/__tests__/heartbeat-session-preparation.test.ts`; targeted controls used the same runner with `-t 'inherited|effective model|retains both plugin and local workspace ownership|never falls back|unavailable|explicit settings|compatible explicit'`. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Initial fixture attempts hit source-binding, provider-execution, missing working-directory, and branch-inspection guards. The fixture was corrected; those guards remain unchanged. An automatic review recovery also hid the initial transition until fixture settlement was corrected. - Tests use real PostgreSQL, real policy resolution, and heartbeat persistence. Provider execution is synthetic; a temporary local working directory replaces environment realization for the test adapter. - Current head `4d21b5180960659576e5950e906056fa4dabb54c` removes two comments after standard review; executable code is unchanged from the locally verified commit. Standard Greptile scores this head 5/5 with no unresolved threads. GitHub CI is now green on this head: 54 successful checks and two intentional Storybook skips. Standard Greptile and Apex both score this head 5/5. Apex found no actionable issues; no unresolved threads or merge conflicts remain. - CI job [workspaces-b](https://github.com/paperclipai/paperclip/actions/runs/40564712367/job/122406649223) was interrupted by a runner shutdown signal at 19:09:39 UTC. Its available log shows no test assertion failure; completed groups passed. The serialized server shard [1/9](https://github.com/paperclipai/paperclip/actions/runs/40564712367/job/122406648374) also received a runner shutdown at 19:12:03 UTC; its final test summary reports 532 passing tests and no failed assertion. Both interrupted jobs passed in the failed-jobs-only rerun, and the final verify aggregate passed. The first retry request was rejected while the containing run was still active; it was accepted after the other jobs settled. - Local verification is targeted as requested. Full test, build, and typecheck coverage is delegated to GitHub CI. No live deployment or latency claim is made. ## Risks Bound repository runs now read the current storage-plugin defaults through one provider-list query. An ambiguous or incompatible effective policy can fail admission instead of being skipped. Existing plugin preparation still uses the recorded workspace owner, so a changed default does not move existing storage to another provider. Explicit policy changes still rotate the session. No schema or API changes are included. ## Model Used OpenAI GPT-6 through Codex, using repository inspection, code editing, shell tools, and targeted test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- maps/feature-map/runs-adapters.md | 8 +++ .../heartbeat-session-preparation.test.ts | 69 ++++++++++++++++++- server/src/services/heartbeat.ts | 2 +- 3 files changed, 75 insertions(+), 4 deletions(-) diff --git a/maps/feature-map/runs-adapters.md b/maps/feature-map/runs-adapters.md index 28d3c31f6d9..9432c9a792b 100644 --- a/maps/feature-map/runs-adapters.md +++ b/maps/feature-map/runs-adapters.md @@ -74,6 +74,14 @@ mutations. Qualify a useful model task separately from saving the account. ### `session` +For a repository project that inherits storage-plugin defaults, run a task and +send two follow-ups. Binding the workspace must preserve the effective policy, +session fingerprint, and workspace identity. An actual project-policy change +must still reset the session and preserve the stored workspace owner. The +[session admission regression](../../server/src/__tests__/heartbeat-session-preparation.test.ts) +checks this with PostgreSQL and synthetic provider execution; it is not live +warm-process or latency evidence. + For `runs-adapters/session`, complete a task in a disposable warm sandbox and retain its saved workspace file. Let the warm session close normally and verify that the retained lease has a complete, lease-bound harness backup before diff --git a/server/src/__tests__/heartbeat-session-preparation.test.ts b/server/src/__tests__/heartbeat-session-preparation.test.ts index 784b3ff2df4..13d0544016b 100644 --- a/server/src/__tests__/heartbeat-session-preparation.test.ts +++ b/server/src/__tests__/heartbeat-session-preparation.test.ts @@ -1,9 +1,15 @@ import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { eq } from "drizzle-orm"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; -import { agents, agentTaskSessions, authUsers, companies, companyMemberships, createDb, heartbeatRuns, issues, plugins, projects } from "@paperclipai/db"; +import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; +import { agents, agentTaskSessions, authUsers, companies, companyMemberships, createDb, executionWorkspaces, heartbeatRuns, issues, plugins, projects } from "@paperclipai/db"; import { registerServerAdapter, unregisterServerAdapter, type AdapterExecutionContext } from "../adapters/index.js"; import { heartbeatService } from "../services/heartbeat.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import type { PluginWorkerManager } from "../services/plugin-worker-manager.js"; +import * as environmentOrchestrator from "../services/environment-run-orchestrator.js"; import { createPluginWorkspaceFixture } from "./helpers/plugin-workspaces.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; @@ -29,7 +35,7 @@ describePostgres("heartbeat session preparation through adapter dispatch", () => captured.set(input.runId, input); // Settle the fixture's work so no automatic unfinished-task retry // competes with the next explicit session-continuation wake. - await db.update(issues).set({ status: "in_review" }).where(eq(issues.id, String(input.context.issueId))); + await db.update(issues).set({ status: "done" }).where(eq(issues.id, String(input.context.issueId))); return { exitCode: 0, signal: null, timedOut: false, sessionId: "saved-session", sessionParams: { sessionId: "saved-session", adapterState: "preserved" }, resultJson: {} }; }, testEnvironment: async () => ({ adapterType, status: "pass", checks: [], testedAt: new Date().toISOString() }), @@ -54,6 +60,7 @@ describePostgres("heartbeat session preparation through adapter dispatch", () => } async function execute(f: Awaited>, context: Record = {}) { + await db.update(issues).set({ status: "todo" }).where(eq(issues.id, f.issueId)); const run = await heartbeat.wakeup(f.agentId, { source: "on_demand", triggerDetail: "manual", reason: "manual", payload: { issueId: f.issueId }, contextSnapshot: { issueId: f.issueId, ...context }, @@ -105,6 +112,62 @@ describePostgres("heartbeat session preparation through adapter dispatch", () => } finally { await db.delete(plugins).where(eq(plugins.id, storage.pluginId)); } }); + it("keeps inherited repository policy stable through binding and retains the provider on policy changes", async () => { + const f = await seed(); + const storage = await createPluginWorkspaceFixture(db, f.companyId); + const [plugin] = await db.select().from(plugins).where(eq(plugins.id, storage.pluginId)); + await db.update(plugins).set({ manifestJson: { ...plugin.manifestJson, workspaceProvider: { + ...plugin.manifestJson.workspaceProvider!, defaultForProjects: { mode: "isolated_workspace", strategy: "git_worktree" }, + } } }).where(eq(plugins.id, storage.pluginId)); + await db.update(projects).set({ executionWorkspacePolicy: null }).where(eq(projects.id, storage.input.projectId)); + await db.update(issues).set({ projectId: storage.input.projectId, projectWorkspaceId: storage.input.projectWorkspaceId, + executionWorkspaceSettings: { workspaceStrategy: { type: "git_worktree", branchTemplate: "work" } }, + }).where(eq(issues.id, f.issueId)); + await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: true, enableIsolatedWorkspacesByDefault: true }); + const cwd = await mkdtemp(join(tmpdir(), "session-provider-fixture-")); + const createOrchestrator = environmentOrchestrator.environmentRunOrchestrator; + const realization = vi.spyOn(environmentOrchestrator, "environmentRunOrchestrator").mockImplementation((...args) => ({ + ...createOrchestrator(...args), + realizeForRun: async input => { + input.executionWorkspace.cwd = cwd; + return { lease: input.lease, persistedExecutionWorkspace: input.persistedExecutionWorkspace, + workspaceRealization: {}, executionTarget: null, remoteExecution: null }; + }, + })); + const originalHeartbeat = heartbeat; + heartbeat = heartbeatService(db, { pluginWorkerManager: storage.workers as PluginWorkerManager }); + try { + const first = await execute(f); + const [bound] = await db.select().from(issues).where(eq(issues.id, f.issueId)); + expect(bound.executionWorkspaceId).toEqual(expect.any(String)); + const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, bound.executionWorkspaceId!)); + expect([...captured.values()].filter(input => input.context.issueId === f.issueId)).toHaveLength(1); + expect(workspace).toMatchObject({ providerType: "plugin", strategyType: "git_worktree" }); + for (let turn = 0; turn < 2; turn++) { + const next = await execute(f); + expect(next.input.runtime.sessionId).toBe("saved-session"); + expect(next.input.config.workspaceStrategy).toEqual(first.input.config.workspaceStrategy); + expect(next.finished.resultJson).toMatchObject({ configFreshness: { + session: { reset: false, taskSessionReused: true, changedCategories: [] }, + workspace: { action: "reuse", workspaceReused: true, changedCategories: [], activeWorkspaceId: bound.executionWorkspaceId }, + } }); + } + await db.update(projects).set({ executionWorkspacePolicy: { enabled: true, allowIssueOverride: false } }).where(eq(projects.id, storage.input.projectId)); + const changed = await execute(f); + expect(changed.input.runtime.sessionId).toBeNull(); + expect(changed.finished.resultJson).toMatchObject({ configFreshness: { session: { reset: true, changedCategories: ["workspaceConfig"] } } }); + const [retained] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, bound.executionWorkspaceId!)); + expect(retained.metadata?.workspaceProviderOwner).toBe(storage.pluginId); + } finally { + await heartbeat.drainActiveRunExecutions(); + heartbeat = originalHeartbeat; + realization.mockRestore(); + await rm(cwd, { recursive: true, force: true }); + await db.delete(plugins).where(eq(plugins.id, storage.pluginId)); + await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: false, enableIsolatedWorkspacesByDefault: false }); + } + }); + it("starts fresh when effective model configuration changes and persists the reason", async () => { const f = await seed(); await execute(f); diff --git a/server/src/services/heartbeat.ts b/server/src/services/heartbeat.ts index 59d2bd04a61..82f12b71f10 100644 --- a/server/src/services/heartbeat.ts +++ b/server/src/services/heartbeat.ts @@ -2380,7 +2380,7 @@ export function heartbeatService( !boundSourceWorkspace && !taskWorkspaceIntentRow?.intent && !issueContext?.projectWorkspaceId ? taskProjectContext : null ); - const parsedProjectExecutionWorkspacePolicy = workspacePolicyContext?.hasWorkspace && !boundSourceWorkspace + const parsedProjectExecutionWorkspacePolicy = workspacePolicyContext?.hasWorkspace ? await effectiveProjectWorkspacePolicy(db, agent.companyId, workspacePolicyContext.executionWorkspacePolicy, workspacePolicyContext.hasGitSource) : parseProjectExecutionWorkspacePolicy(workspacePolicyContext?.executionWorkspacePolicy); const projectExecutionWorkspacePolicy = From d40f9ac377d9d549d3d59890dd45e42ae361e00c Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:07:50 -0500 Subject: [PATCH 18/41] fix: enforce first-task completion decisions and retain E2E evidence (#17940) ## Thinking Path > - Paperclip lets people delegate work to agents and approve proposed work. > - The onboarding first task asks for scope, proposes a task, and waits for a human decision. > - An answer to its opening question supplies scope. It does not approve a proposal. > - Task creation already checks the saved proposal decision. > - Native completion used the general completion policy and could mark the onboarding parent Done without that decision. > - This PR checks the trusted decision before completion feedback and again when the native status is committed. > - The first task stays in review until a saved human decision permits closure. Ordinary completion and acknowledged refusal keep their current behavior. > - The Product E2E guard also retains closed scalar evidence so future failures show which boundary condition was observed. ## Linked Issues or Issue Description Refs: #17914. Profile compatibility follow-up to #17922. **What happened?** A native onboarding run can submit a valid low-risk Done result after only the opening answer. The native finalizer then marks the first task Done even though no human has accepted or rejected a proposal. **Expected behavior** The opening answer must not permit completion. A trusted saved proposal decision must permit it. A saved rejection permits acknowledgment and closure, but does not authorize new work. A later pending proposal must keep the task in review. **Steps to reproduce** Create a server-owned onboarding first task with an answered opening question, no proposal decision, and a valid low-risk completion contract. Pass a Done result through native completion feedback, the control-plane port, and the finalizer. Before this fix, the persisted status is Done. The regression expects In review. Ordinary completion and acknowledged rejection pass as controls. ## What Changed - Share the trusted latest-proposal and human-response checks with completion admission. Keep task creation acceptance-only. - Refuse premature native completion feedback. Keep an already saved premature result in review during finalization. - Check the decision again under the existing task and run locks before committing Done. - Cover accepted decisions, saved refusals, forged context, pending or changed proposals, and a real database race. - Retain a closed scalar receipt when the existing E2E pre-acceptance guard fails. Validate and display it without task content, IDs, filenames, or raw rows. Preserve the original failure, blocked dependents, and historical reports. - Include the evidence module in the journey definition hash. Bind the receipt to its exact failure code and check its closed keys against the full type. - Repair base-branch profile drift found by the full harness check: include the active Copilot v66 fixture in image hashing and update stale Pi version/hash expectations from #17922. Keep all older image inputs and strict hash checks. - Update the onboarding feature and agent access maps and E2E evidence documentation. ## Verification - Semantic regression before implementation: one failure and two passing controls on PostgreSQL 18. The failed assertion observed Done instead of In review. - The local database uses an owned disposable container with an immutable image digest. Actual migrations, feedback, control-plane port, finalizer, and status commit run against PostgreSQL. A temporary test adapter replaces only the embedded database transport because this host has exhausted its PostgreSQL semaphores. - Native control-plane suite: 55 tests passed, including 17 new controls. Existing task-creation admission: 25 passed. Existing completion feedback: 19 passed. - Credential-free E2E support: 206 Vitest suites passed, 4,407 tests passed, and two skipped. All 137 Node support checks passed. Harness typecheck passed. - The failure-receipt regression had 11 failed assertions before capture was implemented. Two failure-code binding assertions fail without their gate. Eight placement/binding cases first validate the receipt-free control, then reject the misplaced receipt. - The untouched base reproduced the three profile compatibility failures. The narrow repair passed all 83 affected tests before the full harness run. - Private workflow preparation and credential-free onboarding preview: nine compatibility checks passed. Ownership, recovery, evidence, shared-contract and workflow compatibility: 77 passed. The actual provider preview also confirms immediate success deletion, earlier declared failure eligibility, limits, and zero retries without credential access. Map checks and git diff checks passed. - Required repository typecheck passed for the product fix. The combined branch build passed. The local full repository suite was stopped after repeated PostgreSQL startup failures; a separate initdb probe confirmed host SysV semaphore exhaustion. No shared IPC resources were changed. [Hosted CI](https://github.com/paperclipai/paperclip/actions/runs/40591020873) passed the full database suites, typecheck, build, runner checks, serialized suites, and browser shards on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b` (55 success, two expected skips, one advisory neutral). - Independent source review approved the product guard, atomic race regression, evidence contract, and maps with no actionable findings. Both automated reviews are 5/5 on final head `1ac4c23605a03bfd1f1f8a017b14ce29fda85f6b`; prior findings are fixed and resolved. Final CI is green and the PR is conflict-free. - No Cloud workspace, provider call, release publication, or deployment is part of this verification. These tests do not identify a historical Cloud failure or qualify live onboarding. - Local dependency installation uses the existing CI-generated lockfile from #17811. No lockfile change is included. ## Risks - Existing onboarding runs without a trusted saved proposal decision remain in review even when the agent reports success. This is the intended approval boundary. - Decision changes can race completion. The status commit checks the decision under its existing locks and uses the existing status-race recovery path. - A saved refusal permits closure only. Creation admission still rejects it. - Evidence readers reject malformed receipts and unrelated failure codes. The version-1 key registry is checked against every receipt field at compile time. Existing attempts do not gain evidence after the fact. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), ultra reasoning, through Codex. Used repository inspection, code execution, and collaborating coding agents. The runtime did not report a context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes:` / `Closes:` / `Refs:` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub references) - [x] My branch name describes the change and contains no internal ticket or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- maps/agent-access-map/onboarding.md | 10 + maps/feature-map/onboarding.md | 9 + .../native-completion-feedback.ts | 6 + .../native-runtime/native-run-finalizer.ts | 11 +- .../paperclip-control-plane-port.test.ts | 188 ++++++++++++++++++ .../status-decision-committer.ts | 6 + .../onboarding-first-task-admission.ts | 69 +++++-- tests/runner-e2e/PRODUCT-JOURNEYS.md | 3 +- tests/runner-e2e/README.md | 10 + tests/runner-e2e/SECURITY.md | 10 + tests/runner-e2e/case-outcome.ts | 16 ++ tests/runner-e2e/daytona-image-content.ts | 1 + tests/runner-e2e/pi-controls-evidence.test.ts | 6 +- .../product-journeys/campaign.test.ts | 19 +- tests/runner-e2e/product-journeys/catalog.ts | 2 +- .../product-journeys/cloud-onboarding.test.ts | 16 ++ .../product-journeys/cloud-onboarding.ts | 3 +- tests/runner-e2e/product-journeys/launch.ts | 2 +- .../onboarding-first-task.test.ts | 85 ++++++++ .../product-journeys/onboarding-first-task.ts | 22 +- .../onboarding-pre-acceptance.test.ts | 105 ++++++++++ .../onboarding-pre-acceptance.ts | 59 ++++++ tests/runner-e2e/product-journeys/types.ts | 1 + tests/runner-e2e/result-validation.ts | 5 + 24 files changed, 632 insertions(+), 32 deletions(-) create mode 100644 tests/runner-e2e/product-journeys/onboarding-pre-acceptance.test.ts create mode 100644 tests/runner-e2e/product-journeys/onboarding-pre-acceptance.ts diff --git a/maps/agent-access-map/onboarding.md b/maps/agent-access-map/onboarding.md index 9c5dded3eff..5160136841c 100644 --- a/maps/agent-access-map/onboarding.md +++ b/maps/agent-access-map/onboarding.md @@ -189,6 +189,16 @@ conversational resolution with a genuine later same-task user message supplies acceptance; an agent-only resolution does not. A plan approval must name the current same-task revision. Newer pending/rejected proposals stop new creation. Already admitted task replay and ordinary delegation retain their existing rules. + +Native `paperclip_finish` does not grant completion authority from an opening +answer. The onboarding parent requires a trusted saved human decision on its +latest proposal. Acceptance permits completion under the existing policy; +rejection permits acknowledgment and closure without authorizing new work. +The native status commit checks this decision again under its existing task/run +locks. This adds no new agent endpoint or cross-company authority. The +[native completion controls](../../server/src/services/native-runtime/paperclip-control-plane-port.test.ts) +check persisted outcomes on disposable PostgreSQL; they are not live Cloud +credential or onboarding qualification. Source: [creation admission](../../server/src/services/onboarding-first-task-admission.ts). Controls: [native/REST PostgreSQL regressions](../../server/src/__tests__/onboarding-first-task-admission.test.ts). These controls do not qualify a live Cloud provider journey. diff --git a/maps/feature-map/onboarding.md b/maps/feature-map/onboarding.md index 347613e3829..d90025b2a31 100644 --- a/maps/feature-map/onboarding.md +++ b/maps/feature-map/onboarding.md @@ -142,6 +142,15 @@ admit new work. [Admission controls](../../server/src/__tests__/onboarding-first exercise native and REST calls against disposable PostgreSQL; live model and Cloud onboarding remain separate evidence. +Native completion also requires a saved human proposal decision. An opening +answer alone must leave the first task in review. Verify that premature native +completion is refused, including a result already saved before finalization. +Accepted proposals and acknowledged human rejections can close the first task; +rejection still does not authorize new tasks. The +[native completion controls](../../server/src/services/native-runtime/paperclip-control-plane-port.test.ts) +exercise these boundaries against disposable PostgreSQL. These controls do not +identify a historical Cloud failure or establish live onboarding success. + For accepted file-producing work, verify exactly one self-assigned child with a self-contained description. The onboarding run must yield through the existing dependency/block disposition so the child's own execution can publish its file. diff --git a/server/src/services/native-runtime/native-completion-feedback.ts b/server/src/services/native-runtime/native-completion-feedback.ts index 9bb94b39c72..668af156fc3 100644 --- a/server/src/services/native-runtime/native-completion-feedback.ts +++ b/server/src/services/native-runtime/native-completion-feedback.ts @@ -1,4 +1,5 @@ import { pendingTaskRepository } from "../task-workspace-tools.js"; +import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js"; import { eligibleIssueMonitorWait } from "../issue-monitors.js"; import { assertNativeCaseHandoffResult } from "./native-case-handoff.js"; import { activeIssueInteractionCondition, ordinaryQuestionCondition } from "../issue-question-context.js"; @@ -140,6 +141,11 @@ export async function nativeCompletionFeedback( if (issue.executionRunId && issue.executionRunId !== runId) { return "Report accepted; a newer run owns the task. Do not claim this report changed its status."; } + if (result.reportedWorkDisposition === "done" && !await onboardingFirstTaskCompletionAllowed(db, { + companyId: run.companyId, issueId: issue.id, runId, + })) { + throw new Error("First-task proposal requires a persisted human decision before completing onboarding. The opening answer supplies scope, not approval. Present the proposal and keep this task in_review until the human accepts or rejects it; acknowledge a rejection without executing that work."); + } const continuation = run.contextSnapshot?.executionContinuation as { objective?: unknown } | undefined; const objective = typeof continuation?.objective === "string" ? continuation.objective : [issue.title, issue.description].filter(Boolean).join("\n"); diff --git a/server/src/services/native-runtime/native-run-finalizer.ts b/server/src/services/native-runtime/native-run-finalizer.ts index 4ade80add27..88efc195b86 100644 --- a/server/src/services/native-runtime/native-run-finalizer.ts +++ b/server/src/services/native-runtime/native-run-finalizer.ts @@ -1,4 +1,5 @@ import { pendingNativeGovernance } from "./native-governance.js"; +import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js"; import { notifyExecutionRunWork } from "../execution-control-notifications.js"; import { reconcileMuseNativeTerminalRun } from "./muse-terminal-reconciliation.js"; import { assertNativeCaseHandoffResult } from "./native-case-handoff.js"; @@ -1429,7 +1430,7 @@ export async function finalizeNativeRun(input: { authoritativeIssue.executionRunId !== run.id && authoritativeIssue.checkoutRunId !== run.id ? await readNativeMonitorDelivery(input.db, { companyId: run.companyId, issueId: coordinator.issueId, runId: run.id }) : null; - const proposedDecision = resolveNativeFinalizerStatus({ + let proposedDecision = resolveNativeFinalizerStatus({ caseHandoffAuthorized: Boolean(await assertNativeCaseHandoffResult(input.db, run.id, authoritativeIssue.id, result)), monitorWaitDispatched: monitorDelivery !== null, monitorWaitPreserved: preservedMonitorWait !== null, @@ -1471,6 +1472,14 @@ export async function finalizeNativeRun(input: { agentId: run.agentId, priorIssueStatus: authoritativeStatus(authoritativeIssue.status), }); + // Already accepted results still require the server-owned first-task decision. + // Keep the task open without inventing an interaction or rerunning the work. + if (proposedDecision.statusAction === "done" && !await onboardingFirstTaskCompletionAllowed(input.db, { + companyId: run.companyId, issueId: authoritativeIssue.id, runId: run.id, + })) { + proposedDecision = { ...proposedDecision, statusAction: "in_review", toStatus: "in_review", + reasonCode: "onboarding_first_task_decision_required", unblockDescriptor: null, effects: [{ kind: "release_checkout" }] }; + } const decision = conversationNativeDecision({ conversation: isConversation(authoritativeIssue), terminalState, workspaceFinalizeStatus: input.workspaceFinalizeStatus, hasGovernanceGate: !!governanceGate, diff --git a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts index f8b9eb1d022..4676979de87 100644 --- a/server/src/services/native-runtime/paperclip-control-plane-port.test.ts +++ b/server/src/services/native-runtime/paperclip-control-plane-port.test.ts @@ -2,6 +2,7 @@ import { afterAll, beforeAll, describe, expect, it } from "vitest"; import { asc, eq, sql } from "drizzle-orm"; import { generateKeyPairSync, randomUUID } from "node:crypto"; import { renderPaperclipWakePrompt } from "@paperclipai/adapter-utils/server-utils"; +import { ONBOARDING_FIRST_TASK_ORIGIN_KIND } from "@paperclipai/shared"; import { readCompletedAssistantMessageCandidate, resolveHeartbeatRunResponse, selectHeartbeatRunFinalAgentMessage } from "../heartbeat-run-summary.js"; import { activityLog, @@ -46,11 +47,13 @@ import { NativeRunCoordinatorStore } from "./native-run-coordinator-store.js"; import { finalizeNativeRun, repairCommittedNativeChatResponse } from "./native-run-finalizer.js"; import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js"; import { issueThreadInteractionService } from "../issue-thread-interactions.js"; +import { documentService } from "../documents.js"; import { claimNativeReviewExecutionLock, getNativeReviewAssignment } from "./native-review-participant.js"; import { claimQueuedNativeReviewRun } from "./native-review-dispatch.js"; import { stageNativeRunnerWakeAttachments } from "./native-runner-file-handoff.js"; import { PaperclipRunnerToolAuthority } from "./paperclip-runner-tool-authority.js"; import { nativeCompletionFeedback } from "./native-completion-feedback.js"; +import { buildOnboardingFirstTaskOpeningQuestion, ONBOARDING_FIRST_TASK_OPENING_QUESTION_ID } from "../onboarding-first-task-assets.js"; import { assertCurrentWakeCommentsRead } from "./current-wake-comments.js"; import { buildExecutionContinuation } from "../execution-continuation.js"; import { childReviewOutcomes } from "./child-review-outcomes.js"; @@ -1709,6 +1712,191 @@ describe("PaperclipControlPlanePort conformance", () => { }); + const allowedFirstTaskCompletionScenarios = [ + "ordinary task", "accepted proposal", "accepted conversational proposal", "accepted current plan", "acknowledged rejection", "acknowledged plan rejection", + ] as const; + const refusedFirstTaskCompletionScenarios = [ + "opening answer only", "agent-resolved without human comment", "quarantined human comment", "newest proposal pending", "stale plan acceptance", + ] as const; + type FirstTaskCompletionScenario = typeof allowedFirstTaskCompletionScenarios[number] + | typeof refusedFirstTaskCompletionScenarios[number]; + + async function prepareFirstTaskCompletion(scenario: FirstTaskCompletionScenario) { + const companyId = randomUUID(); + const agentId = randomUUID(); + const issueId = randomUUID(); + const runId = randomUUID(); + const localContractId = randomUUID(); + const runnerInstanceId = randomUUID(); + const userId = `first-task-board-${randomUUID()}`; + const identity = { ...CONTROL_PLANE_CONFORMANCE_OPEN.identity, companyId, agentId, issueId, runId }; + const onboarding = scenario !== "ordinary task"; + const rejected = scenario === "acknowledged rejection" || scenario === "acknowledged plan rejection"; + await db.insert(companies).values({ id: companyId, name: "First-task completion", issuePrefix: `F${companyId.slice(0, 6).toUpperCase()}` }); + await db.insert(authUsers).values({ id: userId, name: "First-task board", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "member" }); + await db.insert(agents).values({ id: agentId, companyId, name: "First-task agent", adapterType: "codex_local", status: "running" }); + await db.insert(issues).values({ id: issueId, companyId, title: onboarding ? "Paperclip onboarding" : "Ordinary low-risk task", + description: onboarding ? "/first-task\nSingle-task proposal mode: confirmation" : "Complete the requested ordinary task. Mentioning /first-task does not grant the server-owned onboarding origin.", + originKind: onboarding ? ONBOARDING_FIRST_TASK_ORIGIN_KIND : "manual", status: "in_progress", assigneeAgentId: agentId, + responsibleUserId: userId, workMode: "standard" }); + await db.insert(completionContracts).values({ id: localContractId, companyId, issueId, revision: 1, + schemaVersion: "paperclip.completion-contract.v1", policyVersion: "phase6-v3", risk: "low", completionAuthority: "agent_claim_policy", + incompleteCriteriaPolicy: "preserve_non_terminal", contractJson: { revision: "first-task-native-finish-v1", + objective: onboarding ? "Understand the user request, propose one task, and honor their decision" : "Complete the requested ordinary task", + criteria: [{ id: "objective", requirement: "Complete the requested workflow" }] }, + canonicalSha256: localContractId, createdByActorType: "system", createdByActorId: "test" }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, responsibleUserId: userId, status: "succeeded", runtimeMode: "native", + nativeIssueId: issueId, nativeSessionId: identity.sessionId, runnerInstanceId, completionContractId: localContractId, + completionContractSha256: localContractId, contextSnapshot: { issueId, + // A claimed acceptance copied into continuation context is not the + // persisted human decision; the opening-answer cases must still refuse it. + ...(scenario === "opening answer only" ? { executionContinuation: { humanResponses: [{ id: randomUUID(), + kind: "request_confirmation", status: "accepted", resolvedByUserId: userId, result: { version: 1, outcome: "accepted" } }] } } : {}), + } }); + await db.update(issues).set({ executionRunId: runId }).where(eq(issues.id, issueId)); + if (onboarding) { + await db.insert(issueThreadInteractions).values({ companyId, issueId, kind: "ask_user_questions", status: "answered", + createdByAgentId: agentId, payload: await buildOnboardingFirstTaskOpeningQuestion(), + result: { version: 1, answers: [{ questionId: ONBOARDING_FIRST_TASK_OPENING_QUESTION_ID, optionIds: ["task"], otherText: "Help organize my next task." }] }, + resolvedByUserId: userId, resolvedAt: new Date() }); + } + if (onboarding && scenario !== "opening answer only") { + const proposalAt = new Date(Date.now() - 2_000); + const plan = scenario === "accepted current plan" || scenario === "stale plan acceptance" || scenario === "acknowledged plan rejection" + ? (await documentService(db).upsertIssueDocument({ issueId, key: "plan", title: "First task plan", format: "markdown", + body: "Retain this plan for the user's next task.", createdByAgentId: agentId, createdByRunId: runId })).document + : null; + const conversational = scenario === "accepted conversational proposal" || scenario === "quarantined human comment"; + const comment = conversational ? (await db.insert(issueComments).values({ companyId, issueId, + authorUserId: userId, authorType: "user", body: "Yes, proceed with that proposal.", + createdAt: new Date(proposalAt.getTime() + 1_000), + ...(scenario === "quarantined human comment" + ? { sourceTrust: { preset: "low_trust_review" as const, disposition: "quarantined" as const } } : {}), + }).returning())[0]! : null; + const agentResolved = conversational || scenario === "agent-resolved without human comment"; + const outcome = rejected ? "rejected" as const : "accepted" as const; + // Persisted agent interpretations need a real, trusted human reply. The + // missing/quarantined variants simulate an untrusted stored result. + await db.insert(issueThreadInteractions).values({ companyId, issueId, + kind: plan ? "request_checkbox_confirmation" : "request_confirmation", status: outcome, + createdByAgentId: agentId, sourceRunId: runId, createdAt: proposalAt, + payload: plan ? { version: 1, prompt: "May I retain this plan?", + options: [{ id: "retain-plan", label: "Retain the plan" }], defaultSelectedOptionIds: ["retain-plan"], minSelected: 1, + target: { type: "issue_document", key: "plan", issueId, documentId: plan.id, revisionId: plan.latestRevisionId! }, + } : { version: 1, prompt: "May I create one task to organize your agenda?" }, + result: { version: 1, outcome, ...(plan && outcome === "accepted" ? { selectedOptionIds: ["retain-plan"] } : {}), + ...(comment ? { commentId: comment.id } : {}), + ...(outcome === "rejected" ? { reason: "Do not do this work." } : {}) }, + resolvedByUserId: agentResolved ? null : userId, + resolvedByAgentId: agentResolved ? agentId : null, resolvedByRunId: agentResolved ? runId : null, + resolvedAt: new Date(), + }); + if (scenario === "newest proposal pending") { + await db.insert(issueThreadInteractions).values({ companyId, issueId, + kind: "request_checkbox_confirmation", status: "pending", createdByAgentId: agentId, sourceRunId: runId, + createdAt: new Date(proposalAt.getTime() + 1_000), + payload: { version: 1, prompt: "Accept the revised task scope?", options: [{ id: "revised", label: "Use revised scope" }], + defaultSelectedOptionIds: ["revised"], minSelected: 1 }, + }); + } + if (scenario === "stale plan acceptance" || scenario === "acknowledged plan rejection") { + await documentService(db).upsertIssueDocument({ issueId, key: "plan", format: "markdown", + baseRevisionId: plan!.latestRevisionId, body: "A revised scope that has no persisted human decision.", + createdByAgentId: agentId, createdByRunId: runId }); + } + } + const port = new PaperclipControlPlanePort(db, { companyId, issueId, runId, agentId, sessionId: identity.sessionId, + completionContractId: localContractId, completionContractSha256: localContractId, + sourceInstanceId: runnerInstanceId, controlPlaneSourceInstanceId: "first-task-native-finish-control" }); + await port.openRun({ identity, backendKind: "mock", sourceInstanceId: runnerInstanceId }); + const result: PrpStructuredRunResult = { ...structuredClone(CONTROL_PLANE_CONFORMANCE_RESULT), + summary: rejected + ? "You declined the proposal. I will stop here and have not created or performed the proposed task." + : "The requested workflow is complete.", + completionClaim: { contractRevision: "first-task-native-finish-v1", objectiveSatisfied: true, + criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [] }], remainingWork: [] }, + evidence: [], verification: [{ commandOrCheck: "Check the current task state", status: "passed" }], attentionRequests: [] }; + return { companyId, agentId, issueId, runId, port, result }; + } + + it.each(refusedFirstTaskCompletionScenarios)( + "refuses native completion feedback without a trusted current first-task decision (%s)", async (scenario) => { + const { issueId, runId, result } = await prepareFirstTaskCompletion(scenario); + await expect(nativeCompletionFeedback(db, runId, result)) + .rejects.toThrow("First-task proposal requires a persisted human decision"); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]!.status).toBe("in_progress"); + expect(await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, issueId))).toEqual([]); + expect(await db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, runId))).toEqual([]); + }); + + it.each([...allowedFirstTaskCompletionScenarios, ...refusedFirstTaskCompletionScenarios])( + "honors the first-task human decision during native completion (%s)", async (scenario) => { + const { issueId, runId, port, result } = await prepareFirstTaskCompletion(scenario); + const allowed = allowedFirstTaskCompletionScenarios.some(candidate => candidate === scenario); + if (allowed) await expect(nativeCompletionFeedback(db, runId, result)).resolves.toContain("Completion report accepted"); + // Invalid results bypass only early feedback to exercise the backstop for a + // report already committed by a disconnected or older controller. + await port.completeRun({ result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, callerResultId: "first-task-native-finish-result" }); + await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + + const [completedIssue] = await db.select().from(issues).where(eq(issues.id, issueId)); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, issueId)); + if (!allowed) { + const expectedReason = scenario === "newest proposal pending" ? "governed_gate_pending" : "onboarding_first_task_decision_required"; + expect(completedIssue).toMatchObject({ status: "in_review" }); + expect(decisions).toContainEqual(expect.objectContaining({ toStatus: "in_review", reasonCode: expectedReason })); + expect(decisions).not.toContainEqual(expect.objectContaining({ toStatus: "done" })); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]!.resultJson) + .toMatchObject({ finalizationReasonCode: expectedReason }); + } else { + expect(completedIssue).toMatchObject({ status: "done" }); + expect(decisions).toContainEqual(expect.objectContaining({ toStatus: "done", reasonCode: "completion_claim_policy_accepted" })); + } + }); + + it("rechecks the first-task human decision atomically before native status commit", async () => { + const { companyId, agentId, issueId, runId, port, result } = await prepareFirstTaskCompletion("accepted proposal"); + await expect(nativeCompletionFeedback(db, runId, result)).resolves.toContain("Completion report accepted"); + await port.completeRun({ result, terminal: CONTROL_PLANE_CONFORMANCE_TERMINAL, callerResultId: "first-task-race-result" }); + expect(await db.select().from(workAssessments).where(eq(workAssessments.runId, runId))).toEqual([]); + + const interactionId = randomUUID(); + const trigger = `first_task_race_${randomUUID().replace(/-/g, "")}`; + // Assessment persistence follows the policy read and precedes status commit. + // Insert a newer proposal at that real database boundary, exactly once. + await db.execute(sql.raw(`create function ${trigger}() returns trigger language plpgsql as $$ + begin + if NEW.issue_id = '${issueId}'::uuid and NEW.run_id = '${runId}'::uuid then + insert into issue_thread_interactions + (id, company_id, issue_id, kind, status, created_by_agent_id, source_run_id, payload, created_at) + values + ('${interactionId}'::uuid, '${companyId}'::uuid, '${issueId}'::uuid, + 'request_confirmation', 'pending', '${agentId}'::uuid, '${runId}'::uuid, + '{"version":1,"prompt":"Accept the revised task scope?"}'::jsonb, clock_timestamp()) + on conflict (id) do nothing; + end if; + return NEW; + end $$`)); + try { + await db.execute(sql.raw(`create trigger ${trigger} before insert on work_assessments + for each row execute function ${trigger}()`)); + await finalizeNativeRun({ db, runId, workspaceFinalizeStatus: "succeeded", projectRunStatus: true }); + + expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, interactionId)))[0]) + .toMatchObject({ status: "pending", issueId }); + expect((await db.select().from(issues).where(eq(issues.id, issueId)))[0]!.status).toBe("in_review"); + const decisions = await db.select().from(statusDecisions).where(eq(statusDecisions.issueId, issueId)); + expect(decisions).toEqual([expect.objectContaining({ toStatus: "in_review", reasonCode: "governed_gate_pending" })]); + expect(decisions).not.toContainEqual(expect.objectContaining({ toStatus: "done" })); + expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, runId)))[0]) + .toMatchObject({ nativePhase: "committed", resultJson: { finalizationReasonCode: "governed_gate_pending" } }); + } finally { + await db.execute(sql.raw(`drop trigger if exists ${trigger} on work_assessments`)); + await db.execute(sql.raw(`drop function if exists ${trigger}()`)); + } + }); + it.each(["none", "queued", "claimed", "deferred_issue_execution", "coalesced", "current_run", "current_intent"])( "preserves completion caveats and pending child delivery (%s)", async (delivery) => { const identity = CONTROL_PLANE_CONFORMANCE_OPEN.identity; diff --git a/server/src/services/native-runtime/status-decision-committer.ts b/server/src/services/native-runtime/status-decision-committer.ts index 0e34904c5b3..f19bb8cc244 100644 --- a/server/src/services/native-runtime/status-decision-committer.ts +++ b/server/src/services/native-runtime/status-decision-committer.ts @@ -1,4 +1,5 @@ import { scheduledIssueMonitorWait } from "../issue-monitors.js"; +import { onboardingFirstTaskCompletionAllowed } from "../onboarding-first-task-admission.js"; import { hasPublishedGitHubReview, readInheritedGitHubReviewWait } from "../chat-channels/github/review-wait.js"; import { pendingNativeGovernance } from "./native-governance.js"; import { findSupersededExternalReviewRequests, supersedeExternalReviewRequestsInTransaction } from "./superseded-external-review-requests.js"; @@ -1657,6 +1658,11 @@ export async function commitNativeStatusDecision(input: { ) { throw new NativeStatusRaceError(); } + // Recheck under the issue lock, retaining run/proposal/document proof until + // commit. A newer decision cannot race an earlier first-task Done assessment. + if (input.decision.statusAction === "done" && !await onboardingFirstTaskCompletionAllowed(tx as unknown as Db, { + companyId: input.companyId, issueId: input.issueId, runId: input.runId, + })) throw new NativeStatusRaceError(); if (reasonCode === "scheduled_monitor_waiting") { const expected = input.requireMonitorWait; const inReview = input.decision.statusAction === "in_review" && input.decision.toStatus === "in_review"; diff --git a/server/src/services/onboarding-first-task-admission.ts b/server/src/services/onboarding-first-task-admission.ts index e649120bb6b..00182df9e1e 100644 --- a/server/src/services/onboarding-first-task-admission.ts +++ b/server/src/services/onboarding-first-task-admission.ts @@ -59,72 +59,107 @@ export async function assertOnboardingFirstTaskCreationAllowed(tx: Db, input: { )).for("share"); if (!run || runIssueId(run) !== source.id) throw acceptanceRequired(); + if (await readOnboardingFirstTaskDecision(tx, { companyId: input.companyId, issueId: source.id, outcomes: ["accepted"] }) !== "accepted") throw acceptanceRequired(); +} + +/** Shared persisted-decision proof; creation and completion retain different authority. */ +async function readOnboardingFirstTaskDecision(tx: Db, input: { + companyId: string; issueId: string; outcomes: Array<"accepted" | "rejected">; +}): Promise<"accepted" | "rejected" | null> { // Select the newest proposal before checking its outcome: a later pending or // rejected proposal must never fall back to an older accepted scope. Governed // tool/credential actions and custom completion reviews are not proposals. const [proposal] = await tx.select().from(issueThreadInteractions).where(and( - eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, source.id), + eq(issueThreadInteractions.companyId, input.companyId), eq(issueThreadInteractions.issueId, input.issueId), inArray(issueThreadInteractions.kind, ["request_confirmation", "request_checkbox_confirmation"]), sql`NOT (${issueThreadInteractions.payload} ?| ARRAY['toolAction', 'secretProposal', 'connectionAuthorization'])`, sql`COALESCE(${issueThreadInteractions.payload}->'target'->>'type', '') <> 'custom'`, )).orderBy(desc(issueThreadInteractions.createdAt), desc(issueThreadInteractions.id)).limit(1).for("share"); - if (!proposal || proposal.status !== "accepted" || !proposal.resolvedAt) throw acceptanceRequired(); + if (!proposal || (proposal.status !== "accepted" && proposal.status !== "rejected") || + !input.outcomes.includes(proposal.status) || !proposal.resolvedAt) return null; const checkbox = proposal.kind === "request_checkbox_confirmation"; const payload = checkbox ? requestCheckboxConfirmationPayloadSchema.safeParse(proposal.payload) : requestConfirmationPayloadSchema.safeParse(proposal.payload); const result = checkbox ? requestCheckboxConfirmationResultSchema.safeParse(proposal.result) : requestConfirmationResultSchema.safeParse(proposal.result); - if (!payload.success || !result.success || result.data.outcome !== "accepted") throw acceptanceRequired(); - if (checkbox) { + if (!payload.success || !result.success || result.data.outcome !== proposal.status) return null; + if (checkbox && proposal.status === "accepted") { const selected = object(result.data).selectedOptionIds; - if (!("options" in payload.data) || !Array.isArray(selected) || selected.length === 0) throw acceptanceRequired(); + if (!("options" in payload.data) || !Array.isArray(selected) || selected.length === 0) return null; const allowed = new Set(payload.data.options.map(option => option.id)); if (selected.some(id => typeof id !== "string" || !allowed.has(id)) || selected.length < payload.data.minSelected - || (payload.data.maxSelected != null && selected.length > payload.data.maxSelected)) throw acceptanceRequired(); + || (payload.data.maxSelected != null && selected.length > payload.data.maxSelected)) return null; } if (proposal.sourceRunId) { const [origin] = await tx.select(runOriginFields).from(heartbeatRuns).where(and( eq(heartbeatRuns.id, proposal.sourceRunId), eq(heartbeatRuns.companyId, input.companyId), )).for("share"); - if (!origin || runIssueId(origin) !== source.id || origin.agentId !== proposal.createdByAgentId) throw acceptanceRequired(); + if (!origin || runIssueId(origin) !== input.issueId || origin.agentId !== proposal.createdByAgentId) return null; } else if (proposal.createdByAgentId) { const [creator] = await tx.select({ companyId: agents.companyId }).from(agents).where(eq(agents.id, proposal.createdByAgentId)); - if (creator?.companyId !== input.companyId) throw acceptanceRequired(); - } else if (!proposal.createdByUserId) throw acceptanceRequired(); + if (creator?.companyId !== input.companyId) return null; + } else if (!proposal.createdByUserId) return null; const directHumanDecision = Boolean(proposal.resolvedByUserId && !proposal.resolvedByAgentId && !proposal.resolvedByRunId); if (!directHumanDecision) { const commentId = result.data.commentId; - if (!commentId || !proposal.resolvedByAgentId || !proposal.resolvedByRunId) throw acceptanceRequired(); + if (!commentId || !proposal.resolvedByAgentId || !proposal.resolvedByRunId) return null; const [comment] = await tx.select().from(issueComments).where(and( - eq(issueComments.id, commentId), eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, source.id), + eq(issueComments.id, commentId), eq(issueComments.companyId, input.companyId), eq(issueComments.issueId, input.issueId), )).for("share"); if (!comment || comment.deletedAt || comment.authorType !== "user" || !comment.authorUserId || comment.authorAgentId || comment.createdByRunId || comment.derivedAuthorAgentId || comment.derivedCreatedByRunId || comment.sourceTrust?.disposition === "quarantined" || comment.createdAt < proposal.createdAt - || comment.body.trim() === "/new") throw acceptanceRequired(); + || comment.body.trim() === "/new") return null; const [resolver] = await tx.select(runOriginFields).from(heartbeatRuns).where(and( eq(heartbeatRuns.id, proposal.resolvedByRunId), eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, proposal.resolvedByAgentId), )).for("share"); - if (!resolver || runIssueId(resolver) !== source.id - || (resolver.responsibleUserId && resolver.responsibleUserId !== comment.authorUserId)) throw acceptanceRequired(); + if (!resolver || runIssueId(resolver) !== input.issueId + || (resolver.responsibleUserId && resolver.responsibleUserId !== comment.authorUserId)) return null; } + // Refusal permits acknowledgement and closure, never execution. An agent-only + // plan edit cannot turn that human refusal into permission to execute. + if (proposal.status === "rejected") return "rejected"; + const target = payload.data.target; const [plan] = await tx.select({ documentId: documents.id, revisionId: documents.latestRevisionId }) .from(issueDocuments).innerJoin(documents, eq(documents.id, issueDocuments.documentId)).where(and( eq(issueDocuments.companyId, input.companyId), eq(documents.companyId, input.companyId), - eq(issueDocuments.issueId, source.id), eq(issueDocuments.key, "plan"), + eq(issueDocuments.issueId, input.issueId), eq(issueDocuments.key, "plan"), )).for("share", { of: documents }); // Document writers do not all lock the issue, so hold the document row too. // A saved plan must be accepted by revision; an untargeted card cannot stand // in for approval of a different or subsequently edited document. if (plan || target) { if (!target || target.type !== "issue_document" || target.key !== "plan" - || (target.issueId && target.issueId !== source.id) || !plan || plan.revisionId !== target.revisionId - || (target.documentId && plan.documentId !== target.documentId)) throw acceptanceRequired(); + || (target.issueId && target.issueId !== input.issueId) || !plan || plan.revisionId !== target.revisionId + || (target.documentId && plan.documentId !== target.documentId)) return null; } + return "accepted"; +} + +/** Completion may acknowledge a trusted refusal; it never authorizes rejected work. */ +export async function onboardingFirstTaskCompletionAllowed(tx: Db, input: { + companyId: string; issueId: string; runId: string; +}): Promise { + const [source] = await tx.select({ id: issues.id, originKind: issues.originKind }).from(issues).where(and( + eq(issues.id, input.issueId), eq(issues.companyId, input.companyId), + )); + if (!source) return false; + if (source.originKind !== ONBOARDING_FIRST_TASK_ORIGIN_KIND) return true; + // Only the first-task path takes additional authority locks. At status commit + // these are held until the decision is written, in issue/run/proposal order. + const [lockedSource] = await tx.select({ id: issues.id }).from(issues).where(and( + eq(issues.id, source.id), eq(issues.companyId, input.companyId), eq(issues.originKind, ONBOARDING_FIRST_TASK_ORIGIN_KIND), + )).for("update"); + if (!lockedSource) return false; + const [run] = await tx.select({ ...runOriginFields, runtimeMode: heartbeatRuns.runtimeMode }).from(heartbeatRuns).where(and( + eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId), + )).for("share"); + if (!run || run.runtimeMode !== "native" || run.nativeIssueId !== source.id) return false; + return await readOnboardingFirstTaskDecision(tx, { companyId: input.companyId, issueId: source.id, outcomes: ["accepted", "rejected"] }) !== null; } diff --git a/tests/runner-e2e/PRODUCT-JOURNEYS.md b/tests/runner-e2e/PRODUCT-JOURNEYS.md index 2d23dc214e1..a86949df8e7 100644 --- a/tests/runner-e2e/PRODUCT-JOURNEYS.md +++ b/tests/runner-e2e/PRODUCT-JOURNEYS.md @@ -118,7 +118,8 @@ ownership manifests. An expired lease permits cleanup recovery, not reuse. Managed self-hosted instances are removed after every attempt. Existing Cloud stacks retain the QA account and declared baseline while removing verified run-owned fixtures. Successful disposable stacks are deleted. Failed disposable -stacks are stopped and retained for 24 hours, then swept. Interrupted cleanup +stacks are stopped and retained for up to 24 hours. The deployment provider +declares earlier cleanup eligibility to leave time for its sweeper. Interrupted cleanup remains visible and retains exclusive stack ownership. QA identity and inbox never belong to disposable ownership. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 554b10b32d1..af2ba3758b4 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -261,6 +261,16 @@ status, known adapters, and allowlisted outcome/check/server codes. The case report renders these fields explicitly; older results without the receipt remain valid. This diagnostic does not change the journey grade or qualify a live run. +When the canonical first-task journey fails `onboarding_work_before_acceptance`, +`productJourney.onboardingPreAcceptance` retains the failed checkpoint and every +observed guard violation: unexpected task set, existing attachment, non-planning +document, or first task already Done. The closed version-1 receipt contains only +counts, whether the first task matches the canonical parent, and closed first-task +and canonical-parent statuses. Reports validate and display these observations; +they do not infer a product cause. Only the failing boundary is retained, without +task/document IDs, document keys, text, or bodies. Historical results may omit the +receipt. The original error, failed grade, and dependent blocked outcomes remain. + ## Live provider connection journeys See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index 30e2f2bc1a9..c3dde9aef93 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -28,6 +28,16 @@ Unknown codes become `unrecognized`; raw bodies, messages, URLs, headers, and browser errors are excluded. Reports validate the receipt before rendering its explicit fields, and historical results may omit it. +The optional `onboardingPreAcceptance` receipt captures only a failing first-task +acceptance guard. Its closed version-1 fields are a known checkpoint, up to four +distinct guard-violation codes, nonnegative integer task/attachment/document +counts, a canonical-parent match boolean, and closed first-task/canonical-parent +statuses (`missing` and `unknown` remain explicit). It excludes all identifiers, +task text, document keys and bodies, filenames, URLs, credentials, and raw API +responses. Validation rejects extra or inconsistent fields before report use; +older results may omit the receipt. It preserves the original failure grade and +does not attribute a product cause or authorize acceptance. + ## Attended connection authentication See [PROVIDER-CONNECTIONS.md](PROVIDER-CONNECTIONS.md) for the explicit-only diff --git a/tests/runner-e2e/case-outcome.ts b/tests/runner-e2e/case-outcome.ts index 489f16f8384..ca463553322 100644 --- a/tests/runner-e2e/case-outcome.ts +++ b/tests/runner-e2e/case-outcome.ts @@ -1,6 +1,7 @@ import type { RunnerE2EResult } from "./types.js"; import { isBlockedConnectionResult } from "./connection-evidence.js"; import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js"; +import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js"; const html = (value: unknown) => String(value ?? "").replace( /[&<>"']/g, @@ -31,6 +32,20 @@ function renderOnboardingConnect(result: RunnerE2EResult): string { ${response.truncatedChecks ? "

Only the first 32 validation checks were retained.

" : ""}`; }).join("")}`; } +function renderOnboardingPreAcceptance(result: RunnerE2EResult): string { + const receipt = result.productJourney?.onboardingPreAcceptance; + if (receipt === undefined) return ""; + try { validateOnboardingPreAcceptanceEvidence(receipt); } + catch { return "

Onboarding pre-acceptance evidence unavailable: invalid receipt.

"; } + return `
Onboarding pre-acceptance evidence +

Checkpoint: ${html(receipt.checkpoint)}

+

Observed guard violations:

    ${receipt.violations.map(code => `
  • ${html(code)}
    • `).join("")}
+
${[ + ["Tasks", receipt.taskCount], ["Attachments", receipt.attachmentCount], ["Documents", receipt.documentCount], + ["Non-planning documents", receipt.nonPlanningDocumentCount], ["First task matches canonical parent", receipt.firstTaskMatchesCanonicalParent], + ["First task status", receipt.firstTaskStatus], ["Canonical parent status", receipt.canonicalParentStatus], + ].map(([label, value]) => `
${html(label)}
${html(value)}
`).join("")}
`; +} /** An interrupted first-task journey is not evidence of failed downstream behavior. */ export function isIncompleteFirstTaskResult( result: RunnerE2EResult, @@ -93,6 +108,7 @@ export function renderCaseOutcome( ${result.productJourney?.operations ? `

Eval operations: ${html(result.productJourney.operations.campaignId)} · ${result.productJourney.operations.requestCount} requests · model cost $${html(result.productJourney.operations.modelCostUsd ?? "unknown")} (setup $${html(result.productJourney.operations.setupCostUsd ?? "unknown")}, counted once). Feedback: ${html(result.productJourney.operations.feedback.collection)} · ${result.productJourney.operations.feedback.complaints} complaints · ${result.productJourney.operations.feedback.suggestions} suggestions.

` : ""} ${result.productJourney?.operations?.traceIds.length ? `
Trace correlation IDs
    ${result.productJourney.operations.traceIds.map(id => `
  • ${html(id)}
    • `).join("")}
` : ""} ${renderOnboardingConnect(result)} + ${renderOnboardingPreAcceptance(result)} ${result.providerConnection ? `

Target: ${html(result.providerConnection.target.origin)} · ${html(result.providerConnection.target.commit ?? "revision unavailable")} · ${result.providerConnection.assisted ? "Assisted login" : "Unassisted"} · ${html(result.providerConnection.authFreshness)} browser · ${html(result.providerConnection.entry)} / ${html(result.providerConnection.method)} · stopped at ${html(result.providerConnection.phase)}

` : ""} ${failures.length ? `

Failed checks:

    ${failures.map((c) => `
  • ${html(c.id)} — ${html(c.detail)}
    • `).join("")}
` : checks.length ? `

No behavioral matcher failed.${notReached.length ? " The journey is incomplete; unexercised checks are not passes." : failed ? " The overall failure came from a separate run, cleanup, or evidence check." : ""}

` : ""} ${notReached.length ? `

Not reached:

    ${notReached.map((c) => `
  • ${html(c.id)} — ${html(c.notReached)}
    • `).join("")}
` : ""} diff --git a/tests/runner-e2e/daytona-image-content.ts b/tests/runner-e2e/daytona-image-content.ts index 8ab17a54a2e..917bcd3d592 100644 --- a/tests/runner-e2e/daytona-image-content.ts +++ b/tests/runner-e2e/daytona-image-content.ts @@ -67,6 +67,7 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [ "packages/paperclip-runner/test/fixtures/copilot-profile-v58-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v61-identity.json", "packages/paperclip-runner/test/fixtures/copilot-profile-v65-identity.json", + "packages/paperclip-runner/test/fixtures/copilot-profile-v66-identity.json", "packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs", "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs", "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs", diff --git a/tests/runner-e2e/pi-controls-evidence.test.ts b/tests/runner-e2e/pi-controls-evidence.test.ts index fe8e458c89b..5c49d899045 100644 --- a/tests/runner-e2e/pi-controls-evidence.test.ts +++ b/tests/runner-e2e/pi-controls-evidence.test.ts @@ -188,8 +188,8 @@ describe("Pi controls catalog admission", () => { // Extended v3 states the strict file oracle's task-wide Bash limit. const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile; expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter"); - expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 29, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" }); - expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 29, + expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 23, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 30, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "single-json-write-and-content-bound-native-read-both-runs", agentMemoryReadAuthority: "local-withheld-or-exact-remote-agent-run-file", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" }); + expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 10, remoteProcExit: "separately-confirmed-absence-after-read-failure", taskCreation: "explicit-title-and-creation-response-id", profileVersion: 30, remoteProcessIdentity: "observer-pid-startTicks-bootId", controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" }); expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 6, piFileArtifactTitle: "exact-filename", piFileCommandTransport: "fenced-bash-markdown-paste" }); @@ -199,7 +199,7 @@ describe("Pi controls catalog admission", () => { } const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)])); expect(hashes).toEqual({ - "pi-native": "d7d691088af76d460e353a3982f7732e6a65b2e5a23a66d2ef0d534f9d8dae48", + "pi-native": "04d49c6e098c24091d2939108a97722d1b8353e39192b87e98ee33377f183001", "native-active-stop": "4654a2169bff96cf8377fdd62441f5cabea5c613f447e518830f6b1efd82c051", "rich-acp-warm-continuity": "7e8a0012f2f67fba03cd94403b096a07cfc95cdb7aa568b4c3fd18b852d735f3", "extended-harnesses": "d5bcf5de0ade3a54e4e1560e76feb5d8a787039d56d2379cd501626d85ed7418", diff --git a/tests/runner-e2e/product-journeys/campaign.test.ts b/tests/runner-e2e/product-journeys/campaign.test.ts index 27d42b934f9..e039157c8f1 100644 --- a/tests/runner-e2e/product-journeys/campaign.test.ts +++ b/tests/runner-e2e/product-journeys/campaign.test.ts @@ -5,6 +5,10 @@ import os from "node:os"; import { pathToFileURL } from "node:url"; import { createConnectionProof } from "../connection-evidence.js"; import { retainOnboardingArtifact, validateOnboardingArtifact } from "./artifact-evidence.js"; +import { validateOnboardingPreAcceptanceEvidence, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js"; +const preAcceptanceEvidence: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"], + taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0, + firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" }; const persistenceFault = vi.hoisted(() => ({ kind: null as "write" | "rename" | null, active: false, latestReport: null as any })); vi.mock("node:fs/promises", async importOriginal => { const actual = await importOriginal(); @@ -33,7 +37,7 @@ import { regenerateRunnerDashboard } from "../dashboard-regenerate.js"; const roots: string[]=[]; afterEach(async()=>{vi.useRealTimers(); vi.restoreAllMocks(); persistenceFault.kind=null; persistenceFault.active=false; persistenceFault.latestReport=null; process.exitCode=0; await Promise.all(roots.splice(0).map(root=>rm(root,{recursive:true,force:true})));}); type Hook = "collect" | "release" | "stop" | "finalizeEvidence" | "checkpointEvidence"; -async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) { +async function run(options:{authFailure?:boolean;onboardingFailure?:boolean;preAcceptanceFailure?:boolean;missingArtifact?:boolean;drift?:boolean;cleanupFailure?:boolean;collectionFailure?:boolean;mini?:boolean;missingReceipt?:boolean;zeroRequests?:boolean;missingCheck?:boolean;finalizationFailure?:boolean;persistenceFailure?:"write"|"rename";persistenceAfter?:"collection"|"release";stallHook?:Hook;onHook?:(hook:Hook,signal?:AbortSignal)=>void}={}) { const root=await mkdtemp(path.join(os.tmpdir(),"journey-campaign-test-")); roots.push(root); const module=path.join(root,"provider.mjs"); const proof = createConnectionProof(); @@ -44,7 +48,7 @@ let onboarded=false; const context={identity,company:{id:'company-1',issuePrefix:'QA'},api:{post:async()=>({id:'issue-fixture-1',identifier:'QA-1'})},startingState:'onboarding',capabilities:['company','board-user','task-write'],page:{async screenshot(){} }}; export const provider={version:1, async acquire(input){return { async authenticate(){${options.authFailure ? "throw {outcome:'failed',code:'login_code_expired'};" : ""}}, -async onboard(evidence){${options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}}, +async onboard(evidence){${options.preAcceptanceFailure ? `evidence.onboardingPreAcceptance=${JSON.stringify(preAcceptanceEvidence)};throw {outcome:'failed',code:'onboarding_work_before_acceptance'};` : options.onboardingFailure ? "throw {outcome:'failed',code:'onboarding_failed'};" : `${options.missingArtifact ? "" : `evidence.artifact=${JSON.stringify(artifact)};`}evidence.checks.push(...['fresh-workspace','ui-connection-and-native-agent','first-task-complete','downloadable-artifact-verified'${options.missingCheck ? "" : ",'native-trace-verified'"}].map(id=>({id,passed:true})));onboarded=true;return context;`}}, async prepare(){throw Error('onboarding receipt was not reused')}, async identity(){return {...identity,commit:'${(options.drift?"b":"a").repeat(40)}'}}, async collect(){${options.collectionFailure ? "throw Error('private collection error');" : options.missingReceipt ? "return [];" : `return onboarded?[{journeyId:'cloud-onboarding',runIds:['run-1'],usage:{runs:[{runId:'run-1',usage:{costUsd:0.01}}]},operations:{campaignId:input.campaignId,attemptId:input.campaignId+':cloud-onboarding:1',modelCostUsd:0.01,setupCostUsd:0.005,requestCount:${options.zeroRequests ? 0 : 2},feedback:{collection:'complete',complaints:0,suggestions:0},traceIds:['a'.repeat(32)]}}]:[];`}}, async own(){}, async release(){${options.cleanupFailure ? "throw Error('private raw error');" : "return 'released';"}}, async stop(){} @@ -109,6 +113,17 @@ describe("product journey campaign boundary outcomes",()=>{ expect(campaign.results.map((r:any)=>r.productJourney.outcome)).toEqual(["failed","blocked"]); expect(campaign.results[0].productJourney.phases.find((p:any)=>p.phase==="journey").outcome).toBe("failed"); }); + it("preserves failed pre-acceptance evidence in normalized results and blocks dependent reuse", async () => { + const campaign = await run({ preAcceptanceFailure: true }); + const [onboarding, dependent] = campaign.results; + expect(campaign.results.map((row:any) => row.productJourney.outcome)).toEqual(["failed", "blocked"]); + expect(onboarding).toMatchObject({ status: "failed", failureClass: "candidate_failure", error: "onboarding_work_before_acceptance" }); + expect(onboarding.productJourney.onboardingPreAcceptance).toEqual(preAcceptanceEvidence); + expect(() => validateOnboardingPreAcceptanceEvidence(onboarding.productJourney.onboardingPreAcceptance)).not.toThrow(); + expect(onboarding.productJourney.phases).toContainEqual({ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" }); + expect(dependent.error).toBe("prerequisite_journey_failed"); + expect(dependent.productJourney.onboardingPreAcceptance).toBeUndefined(); + }); it("reuses a verified onboarding context for the same shared story",async()=>{ const campaign=await run(); expect(campaign.results.map((r:any)=>r.status)).toEqual(["passed","passed"]); diff --git a/tests/runner-e2e/product-journeys/catalog.ts b/tests/runner-e2e/product-journeys/catalog.ts index c018e769aab..6077ff48c36 100644 --- a/tests/runner-e2e/product-journeys/catalog.ts +++ b/tests/runner-e2e/product-journeys/catalog.ts @@ -9,7 +9,7 @@ export function preparationPreview(config: TargetConfig) { target: { id: config.id, mode: config.mode, executionEnvironment: config.executionEnvironment }, journeys: config.journeys.map(id => productJourneys.find(journey => journey.id === id)!), configuration: config.baseline, - lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures for 24 hours" : "managed instance; remove after every attempt", + lifetime: config.mode === "existing-cloud" ? "persistent; clean run-owned resources only" : config.mode === "new-cloud" ? "disposable; delete on pass, retain failures up to 24 hours" : "managed instance; remove after every attempt", limits: config.limits, authentication: config.mode === "managed-self-hosted" ? "local-trusted" : "normal email-code sign-in", modelProfileId: config.modelProfileId ?? null, maxDisposableStacks: config.mode === "new-cloud" ? 1 : 0, diff --git a/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts b/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts index 0878ade8065..68adbdd2d53 100644 --- a/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts +++ b/tests/runner-e2e/product-journeys/cloud-onboarding.test.ts @@ -2,6 +2,8 @@ import { describe, expect, it, vi } from "vitest"; import { cloudOnboarding, type CloudOnboardingInput } from "./cloud-onboarding.js"; import { onboardingFirstTask } from "./onboarding-first-task.js"; import { retainOnboardingArtifact } from "./artifact-evidence.js"; +import { JourneyError } from "./types.js"; +import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js"; vi.mock("@playwright/test", () => ({ expect: (locator: { isVisible(): Promise }) => ({ toBeVisible: async () => { expect(await locator.isVisible()).toBe(true); }, @@ -118,6 +120,20 @@ function fixture(entry: Entry, options: { intentFailure?: boolean; qualification } describe("Cloud onboarding visible controls", () => { + it("retains pre-acceptance evidence while preserving the failed error and withholding completion markers", async () => { + const f = fixture("portfolio"); + const receipt: OnboardingPreAcceptanceEvidence = { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"], + taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0, + firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" }; + vi.mocked(onboardingFirstTask).mockImplementationOnce(async input => { + input.onPreAcceptanceFailure?.(receipt); + throw new JourneyError("failed", "onboarding_work_before_acceptance"); + }); + await expect(cloudOnboarding(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" }); + expect(JSON.parse(JSON.stringify(f.input.evidence)).onboardingPreAcceptance).toEqual(receipt); + expect(f.input.evidence.checks.map(check => check.id)).toEqual(["fresh-workspace", "ui-connection-and-native-agent"]); + expect(f.events).not.toContain("native:verified"); + }); it.each(["first-run", "portfolio", "create-form"] as const)("creates from %s with intent before dispatch and qualification before paid Connect", async entry => { const f = fixture(entry); const result = await cloudOnboarding(f.input); diff --git a/tests/runner-e2e/product-journeys/cloud-onboarding.ts b/tests/runner-e2e/product-journeys/cloud-onboarding.ts index 6f9a75634a7..215871b411d 100644 --- a/tests/runner-e2e/product-journeys/cloud-onboarding.ts +++ b/tests/runner-e2e/product-journeys/cloud-onboarding.ts @@ -83,7 +83,8 @@ export async function cloudOnboarding(input: CloudOnboardingInput): Promise { evidence.artifact = artifact; } }); + own: input.own, observeRun: input.observeRun, onArtifact: artifact => { evidence.artifact = artifact; }, + onPreAcceptanceFailure: receipt => { evidence.onboardingPreAcceptance = receipt; } }); await input.verifyExecution(context, [completed.artifact.originatingRunId]); evidence.checks.push({ id: "first-task-complete", passed: true }, { id: "downloadable-artifact-verified", passed: true }, { id: "native-trace-verified", passed: true }); diff --git a/tests/runner-e2e/product-journeys/launch.ts b/tests/runner-e2e/product-journeys/launch.ts index 410bb7725bf..980822013fd 100644 --- a/tests/runner-e2e/product-journeys/launch.ts +++ b/tests/runner-e2e/product-journeys/launch.ts @@ -99,7 +99,7 @@ export async function launchProductJourneys(args: string[], repositoryRoot: stri try { sha = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repositoryRoot, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); } catch { /* Packaged harnesses may not include Git metadata. */ } const source = resolveRunnerE2ESource({ sha, ref: null, workflowRunUrl: null }); const definitionHash = createHash("sha256").update(JSON.stringify(productJourneys)); - for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); + for (const name of ["catalog.ts", "task-properties.ts", "cloud-onboarding.ts", "onboarding-connect.ts", "onboarding-first-task.ts", "onboarding-pre-acceptance.ts", "artifact-evidence.ts", "fixtures.ts", "launch.ts", "types.ts", "collection.ts", "local-target.ts"]) definitionHash.update(await readFile(path.join(import.meta.dirname, name))); const suiteDefinitionHash = definitionHash.digest("hex"); const expected = config.journeys.map(id => `product-journeys.${config.id}.${id === "cloud-onboarding" ? config.modelProfileId : "browser"}.${config.executionEnvironment}.${id}`); let reportPersisted = false; diff --git a/tests/runner-e2e/product-journeys/onboarding-first-task.test.ts b/tests/runner-e2e/product-journeys/onboarding-first-task.test.ts index ab452f1901c..2e559aa8584 100644 --- a/tests/runner-e2e/product-journeys/onboarding-first-task.test.ts +++ b/tests/runner-e2e/product-journeys/onboarding-first-task.test.ts @@ -4,6 +4,7 @@ import { createConnectionProof } from "../connection-evidence.js"; import { connectionDelay } from "../connection-target.js"; import { onboardingFirstTask, type OnboardingFirstTaskInput } from "./onboarding-first-task.js"; import { validateOnboardingArtifact } from "./artifact-evidence.js"; +import { validateOnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js"; vi.mock("../connection-target.js", () => ({ connectionDelay: vi.fn(async () => {}) })); type Row = { id: string; [key: string]: any }; @@ -92,6 +93,90 @@ function fixture(options: Options = {}) { afterEach(() => { vi.restoreAllMocks(); vi.mocked(connectionDelay).mockReset().mockResolvedValue(undefined); }); describe("Cloud canonical onboarding first task", () => { + it.each([ + ["unexpected task set", "unexpected_task_set", (state: Parameters>[1]) => { state.tasks.push({ id: "private-early-task", title: "private task text", status: "todo" }); }, { taskCount: 2 }], + ["attachment", "attachment_present", (state: Parameters>[1]) => { state.attachments.push({ id: "private-attachment", issueId: "parent", originalFilename: "private-file.txt" }); }, { attachmentCount: 1 }], + ["non-planning document", "non_planning_document_present", (state: Parameters>[1]) => { state.documents.push({ id: "private-document", issueId: "parent", key: "private-result-key", body: "private document body" }); }, { documentCount: 1, nonPlanningDocumentCount: 1 }], + ["first task done", "first_task_done", (state: Parameters>[1]) => { state.tasks[0]!.status = "done"; }, { firstTaskStatus: "done", canonicalParentStatus: "done" }], + ] as const)("retains the observed %s violation at the failing acceptance boundary", async (_label, violation, mutate, observed) => { + const f = fixture({ mutate(phase, state) { if (phase === "proposal") mutate(state); } }); + const capture = vi.fn(); + f.input.onPreAcceptanceFailure = capture; + await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ outcome: "failed", code: "onboarding_work_before_acceptance" }); + expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal", violations: [violation], + taskCount: 1, attachmentCount: 0, documentCount: 0, nonPlanningDocumentCount: 0, + firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review", ...observed }); + expect(() => validateOnboardingPreAcceptanceEvidence(JSON.parse(JSON.stringify(capture.mock.calls[0]![0])))).not.toThrow(); + const retained = JSON.stringify(capture.mock.calls[0]![0]); + for (const text of ["private-", "parent", "proposal-run", "qa.example.test", "connection", "campaign-1"]) expect(retained).not.toContain(text); + expect(f.events).not.toContain("click:Approve"); + }); + it("retains every simultaneous guard violation from the same failed snapshot", async () => { + const f = fixture({ plan: true, mutate(phase, state) { + if (phase !== "proposal") return; + state.tasks[0]!.status = "done"; + state.tasks.push({ id: "private-early-task" }); + state.attachments.push({ id: "private-attachment", issueId: "parent" }); + state.documents.push({ id: "private-document", issueId: "parent", key: "private-result", body: "private body" }); + } }); + const capture = vi.fn(); + f.input.onPreAcceptanceFailure = capture; + await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" }); + expect(capture).toHaveBeenCalledExactlyOnceWith({ version: 1, checkpoint: "awaiting-proposal", + violations: ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"], + taskCount: 2, attachmentCount: 1, documentCount: 2, nonPlanningDocumentCount: 1, + firstTaskMatchesCanonicalParent: true, firstTaskStatus: "done", canonicalParentStatus: "done" }); + expect(f.events).not.toContain("click:Confirm selection"); + }); + it.each(["missing", "replaced", "reordered", "unknown-status"] as const)("records closed task observations when the snapshot is %s", async mode => { + const f = fixture({ mutate(phase, state) { + if (phase !== "proposal") return; + if (mode === "missing") state.tasks = []; + if (mode === "replaced") state.tasks = [{ id: "private-replacement", status: "done" }]; + if (mode === "reordered") state.tasks.unshift({ id: "private-first", status: "done" }); + if (mode === "unknown-status") { state.tasks[0]!.status = "private future status"; state.attachments.push({ id: "private-attachment", issueId: "parent" }); } + } }); + const capture = vi.fn(); + f.input.onPreAcceptanceFailure = capture; + await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" }); + expect(capture).toHaveBeenCalledOnce(); + expect(capture.mock.calls[0]![0]).toMatchObject({ + violations: mode === "unknown-status" ? ["attachment_present"] : mode === "missing" ? ["unexpected_task_set"] : ["unexpected_task_set", "first_task_done"], + taskCount: mode === "missing" ? 0 : mode === "reordered" ? 2 : 1, + firstTaskMatchesCanonicalParent: mode === "unknown-status", + firstTaskStatus: mode === "missing" ? "missing" : mode === "unknown-status" ? "unknown" : "done", + canonicalParentStatus: mode === "reordered" ? "in_review" : mode === "unknown-status" ? "unknown" : "missing", + }); + expect(JSON.stringify(capture.mock.calls[0]![0])).not.toContain("private"); + expect(f.events).not.toContain("click:Approve"); + }); + it("retains an opening-boundary failure before the opening answer dispatches work", async () => { + const f = fixture({ mutate(phase, state) { if (phase === "opening") state.tasks[0]!.status = "done"; } }); + const capture = vi.fn(); + f.input.onPreAcceptanceFailure = capture; + await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" }); + expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ checkpoint: "before-opening-answer", violations: ["first_task_done"] })); + expect(f.events).not.toContain("click:Continue"); + }); + it("does not claim the first-slot Done guard when only a later canonical parent is Done", async () => { + const f = fixture({ mutate(phase, state) { + if (phase !== "proposal") return; + state.tasks[0]!.status = "done"; + state.tasks.unshift({ id: "private-first", status: "todo" }); + } }); + const capture = vi.fn(); + f.input.onPreAcceptanceFailure = capture; + await expect(onboardingFirstTask(f.input)).rejects.toMatchObject({ code: "onboarding_work_before_acceptance" }); + expect(capture).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ violations: ["unexpected_task_set"], + firstTaskMatchesCanonicalParent: false, firstTaskStatus: "todo", canonicalParentStatus: "done" })); + }); + it.each([false, true])("does not retain failure snapshots during clean canonical success, plan mode %s", async plan => { + const f = fixture({ plan }); + const capture = vi.fn(); + f.input.onPreAcceptanceFailure = capture; + await expect(onboardingFirstTask(f.input)).resolves.toMatchObject({ onboardingIssueId: "parent", childIssueId: "child" }); + expect(capture).not.toHaveBeenCalled(); + }); it.each(["reload", "download-link"] as const)("retains the independently verified download when later %s fails", async readbackFailure => { const f = fixture({ readbackFailure }); const capture = vi.fn(); diff --git a/tests/runner-e2e/product-journeys/onboarding-first-task.ts b/tests/runner-e2e/product-journeys/onboarding-first-task.ts index ee50b6466ad..474bd615a84 100644 --- a/tests/runner-e2e/product-journeys/onboarding-first-task.ts +++ b/tests/runner-e2e/product-journeys/onboarding-first-task.ts @@ -6,6 +6,7 @@ import { activeRuns, firstTaskCompletionSettled, type Row } from "../first-task- import { isBlockedUnstartedWake, isTerminalUnstartedWake } from "../non-execution-wake.js"; import { JourneyError, type JourneyContext, type OnboardingArtifactEvidence, type TargetSession } from "./types.js"; import { retainOnboardingArtifact } from "./artifact-evidence.js"; +import { preAcceptanceTaskStatus, preAcceptanceViolations, type OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js"; export interface OnboardingFirstTaskInput { context: JourneyContext; @@ -18,6 +19,7 @@ export interface OnboardingFirstTaskInput { own: TargetSession["own"]; observeRun(run: Row): Promise; onArtifact?(artifact: OnboardingArtifactEvidence): void; + onPreAcceptanceFailure?(evidence: OnboardingPreAcceptanceEvidence): void; } export interface OnboardingFirstTaskResult { runIds: string[]; @@ -91,9 +93,19 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom fail("onboarding_first_task_failed"); } }; - const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string) => { - if (snapshot.tasks.length !== 1 || snapshot.tasks[0]!.id !== parentId || snapshot.attachments.length || - snapshot.documents.some(document => !isPlanningDocument(document)) || snapshot.tasks[0]!.status === "done") fail("onboarding_work_before_acceptance"); + const assertBeforeAcceptance = (snapshot: Snapshot, parentId: string, checkpoint: OnboardingPreAcceptanceEvidence["checkpoint"]) => { + const observed = { + taskCount: snapshot.tasks.length, attachmentCount: snapshot.attachments.length, + documentCount: snapshot.documents.length, nonPlanningDocumentCount: snapshot.documents.filter(document => !isPlanningDocument(document)).length, + firstTaskMatchesCanonicalParent: snapshot.tasks[0]?.id === parentId, + firstTaskStatus: preAcceptanceTaskStatus(snapshot.tasks[0]), + canonicalParentStatus: preAcceptanceTaskStatus(snapshot.tasks.find(task => task.id === parentId)), + }; + const violations = preAcceptanceViolations(observed); + if (violations.length) { + input.onPreAcceptanceFailure?.({ version: 1, checkpoint, violations, ...observed }); + fail("onboarding_work_before_acceptance"); + } assertRuns(snapshot, [parentId]); }; @@ -105,7 +117,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom await input.own({ kind: "issue", name: parent.title, id: parent.id }); snapshot = await load(parent.id); if (seen.size) fail("onboarding_opening_started_model_work"); - assertBeforeAcceptance(snapshot, parent.id); + assertBeforeAcceptance(snapshot, parent.id, "before-opening-answer"); const opening = snapshot.interactions.filter(row => row.kind === "ask_user_questions" && row.status === "pending"); const questions = opening[0]?.payload?.questionSet?.questions ?? opening[0]?.payload?.questions ?? []; const question = questions.find((row: Row) => row.id === "first-task-opening"); @@ -129,7 +141,7 @@ export async function onboardingFirstTask(input: OnboardingFirstTaskInput): Prom let proposal: Row | undefined; while (!proposal) { snapshot = await load(parent.id); - assertBeforeAcceptance(snapshot, parent.id); + assertBeforeAcceptance(snapshot, parent.id, "awaiting-proposal"); const pending = snapshot.interactions.filter(row => row.status === "pending"); const proposals = pending.filter(isProposal); if (proposals.length > 1 || pending.some(row => row.kind === "ask_user_questions" && row.id !== opening[0]!.id)) fail("onboarding_proposal_scope_unclear"); diff --git a/tests/runner-e2e/product-journeys/onboarding-pre-acceptance.test.ts b/tests/runner-e2e/product-journeys/onboarding-pre-acceptance.test.ts new file mode 100644 index 00000000000..84e8ca87aa1 --- /dev/null +++ b/tests/runner-e2e/product-journeys/onboarding-pre-acceptance.test.ts @@ -0,0 +1,105 @@ +import { describe, expect, it } from "vitest"; +import { validateRetainedRunnerResult } from "../result-validation.js"; +import { renderCaseOutcome } from "../case-outcome.js"; +import type { RunnerE2EResult } from "../types.js"; +import type { OnboardingPreAcceptanceEvidence } from "./onboarding-pre-acceptance.js"; + +function result(): RunnerE2EResult { + return { schema: "paperclip.runner-e2e.result/v2", suiteId: "product-journeys", suiteDefinitionHash: "test", + executionId: "product-journeys.qa.runner-codex.local.cloud-onboarding", profileId: "runner-codex", environmentId: "local", caseId: "cloud-onboarding", + provider: "codex", model: "synthetic-model", runtimeMode: "native", status: "failed", error: "onboarding_work_before_acceptance", cleanup: "passed", attempt: 1, + startedAt: "2026-01-01T00:00:00Z", finishedAt: "2026-01-01T00:00:01Z", durationMs: 1000, + productJourney: { outcome: "failed", startingState: "onboarding", phases: [{ phase: "journey", outcome: "failed", code: "onboarding_work_before_acceptance" }], + checks: [], privacy: "private", resourceDisposition: "retained" } }; +} +function preAcceptanceReceipt(): OnboardingPreAcceptanceEvidence { + return { version: 1, checkpoint: "awaiting-proposal", violations: ["attachment_present"], + taskCount: 1, attachmentCount: 1, documentCount: 0, nonPlanningDocumentCount: 0, + firstTaskMatchesCanonicalParent: true, firstTaskStatus: "in_review", canonicalParentStatus: "in_review" }; +} +function passedResult(): RunnerE2EResult { + const retained = result(); + retained.status = "passed"; + delete retained.error; + retained.runIds = ["run-1"]; + Object.assign(retained.productJourney!, { outcome: "passed", campaignId: "campaign", attemptId: "attempt", + phases: [{ phase: "journey", outcome: "passed" }], resourceDisposition: "released", + checks: ["fresh-workspace", "ui-connection-and-native-agent", "first-task-complete", "downloadable-artifact-verified", "native-trace-verified"].map(id => ({ id, passed: true })), + deployment: { kind: "cloud", targetId: "qa", origin: "https://qa.example.test", commit: "a".repeat(40), cloudRelease: "release", runtimeImage: "image", configurationHash: "config" }, + operations: { campaignId: "campaign", attemptId: "attempt", modelCostUsd: null, setupCostUsd: null, requestCount: 1, + feedback: { collection: "complete", complaints: 0, suggestions: 0 }, traceIds: ["trace-1"] } }); + return retained; +} +describe("onboarding pre-acceptance retained evidence", () => { + it("accepts historical results without pre-acceptance evidence and preserves optional receipts in reports", () => { + const retained = result(); + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + expect(renderCaseOutcome(retained, true, [])).not.toContain("Onboarding pre-acceptance evidence"); + Object.assign(retained.productJourney!, { onboardingPreAcceptance: preAcceptanceReceipt() }); + const roundTrip = JSON.parse(JSON.stringify(retained)); + expect(() => validateRetainedRunnerResult(roundTrip)).not.toThrow(); + const output = renderCaseOutcome(roundTrip, true, []); + for (const text of ["Onboarding pre-acceptance evidence", "awaiting-proposal", "attachment_present", "First task status", "Canonical parent status", "in_review", "Attachments", "Non-planning documents", "onboarding_work_before_acceptance"]) expect(output).toContain(text); + expect(roundTrip.status).toBe("failed"); + expect(roundTrip.productJourney.outcome).toBe("failed"); + }); + it.each([ + ["a passed result", passedResult], + ["another case", (): RunnerE2EResult => ({ ...result(), caseId: "task-properties", + executionId: "product-journeys.qa.browser.local.task-properties", profileId: "browser", provider: null, model: null, runtimeMode: "none" })], + ["another suite", (): RunnerE2EResult => ({ ...result(), suiteId: "first-task", executionId: "first-task.runner-codex.local.cloud-onboarding" })], + ] as const)("rejects a valid receipt on %s after accepting the same result without it", (_label, makeResult) => { + const retained = makeResult(); + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt(); + expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance"); + }); + it.each(["passed", "blocked", "unsupported"] as const)("rejects a valid receipt on a %s journey outcome after accepting the same result without it", outcome => { + const retained = result(); + retained.productJourney!.outcome = outcome; + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt(); + expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance"); + }); + it.each(["onboarding_artifact_evidence_missing", undefined])("rejects a valid receipt for an unrelated or absent failure code (%s)", error => { + const retained = result(); + retained.error = error; + expect(() => validateRetainedRunnerResult(retained)).not.toThrow(); + retained.productJourney!.onboardingPreAcceptance = preAcceptanceReceipt(); + expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance"); + }); + it.each([ + ["raw bodies", (): unknown => ({ ...preAcceptanceReceipt(), body: "private-body-must-not-render" })], + ["task identifiers", (): unknown => ({ ...preAcceptanceReceipt(), taskIds: ["private-task"] })], + ["document keys", (): unknown => ({ ...preAcceptanceReceipt(), documentKeys: ["private-key"] })], + ["unknown version", (): unknown => ({ ...preAcceptanceReceipt(), version: 2 })], + ["unknown checkpoint", (): unknown => ({ ...preAcceptanceReceipt(), checkpoint: "private text" })], + ["unknown violation", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["private text"] })], + ["duplicate violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["attachment_present", "attachment_present"] })], + ["empty violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: [] })], + ["inconsistent violations", (): unknown => ({ ...preAcceptanceReceipt(), violations: ["first_task_done"] })], + ["negative counts", (): unknown => ({ ...preAcceptanceReceipt(), attachmentCount: -1 })], + ["fractional counts", (): unknown => ({ ...preAcceptanceReceipt(), taskCount: 1.5 })], + ["unsafe counts", (): unknown => ({ ...preAcceptanceReceipt(), documentCount: Number.MAX_SAFE_INTEGER + 1 })], + ["invalid document counts", (): unknown => ({ ...preAcceptanceReceipt(), nonPlanningDocumentCount: 1 })], + ["raw task status", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskStatus: "private status" })], + ["raw parent status", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "private status" })], + ["nonboolean match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: "true" })], + ["inconsistent task match", (): unknown => ({ ...preAcceptanceReceipt(), firstTaskMatchesCanonicalParent: false })], + ["inconsistent matched statuses", (): unknown => ({ ...preAcceptanceReceipt(), canonicalParentStatus: "done" })], + ["missing fields", (): unknown => ({ version: 1 })], + ["null", (): unknown => null], + ] as const)("rejects pre-acceptance evidence containing %s", (_label, makeReceipt) => { + const retained = result(); + Object.assign(retained.productJourney!, { onboardingPreAcceptance: makeReceipt() }); + expect(() => validateRetainedRunnerResult(retained)).toThrow("result.productJourney.onboardingPreAcceptance"); + }); + it("renders a fixed fallback for malformed pre-acceptance evidence without exposing its fields", () => { + const retained = result(); + Object.assign(retained.productJourney!, { onboardingPreAcceptance: { ...preAcceptanceReceipt(), body: "private-body-must-not-render" } }); + const output = renderCaseOutcome(retained, false, []); + expect(output).toContain("Onboarding pre-acceptance evidence unavailable: invalid receipt"); + expect(output).not.toContain("private-body-must-not-render"); + expect(output).not.toContain("attachment_present"); + }); +}); diff --git a/tests/runner-e2e/product-journeys/onboarding-pre-acceptance.ts b/tests/runner-e2e/product-journeys/onboarding-pre-acceptance.ts new file mode 100644 index 00000000000..0b0fe7e650e --- /dev/null +++ b/tests/runner-e2e/product-journeys/onboarding-pre-acceptance.ts @@ -0,0 +1,59 @@ +import { ISSUE_STATUSES } from "../../../packages/shared/src/constants.js"; + +const checkpoints = ["before-opening-answer", "awaiting-proposal"] as const; +const violations = ["unexpected_task_set", "attachment_present", "non_planning_document_present", "first_task_done"] as const; +const statuses = [...ISSUE_STATUSES, "missing", "unknown"] as const; + +/** Scalar observations from the first failing pre-acceptance guard; never raw rows. */ +export interface OnboardingPreAcceptanceEvidence { + version: 1; + checkpoint: typeof checkpoints[number]; + violations: Array; + taskCount: number; + attachmentCount: number; + documentCount: number; + nonPlanningDocumentCount: number; + firstTaskMatchesCanonicalParent: boolean; + firstTaskStatus: typeof statuses[number]; + canonicalParentStatus: typeof statuses[number]; +} + +const receiptKeys = { + version: true, checkpoint: true, violations: true, taskCount: true, attachmentCount: true, + documentCount: true, nonPlanningDocumentCount: true, firstTaskMatchesCanonicalParent: true, + firstTaskStatus: true, canonicalParentStatus: true, +} satisfies Record; + +export function preAcceptanceTaskStatus(task: { status?: unknown; [key: string]: unknown } | undefined): OnboardingPreAcceptanceEvidence["firstTaskStatus"] { + if (!task) return "missing"; + return ISSUE_STATUSES.find(status => status === task.status) ?? "unknown"; +} + +export function preAcceptanceViolations(observed: Pick): OnboardingPreAcceptanceEvidence["violations"] { + return [ + ...(observed.taskCount !== 1 || !observed.firstTaskMatchesCanonicalParent ? ["unexpected_task_set" as const] : []), + ...(observed.attachmentCount ? ["attachment_present" as const] : []), + ...(observed.nonPlanningDocumentCount ? ["non_planning_document_present" as const] : []), + ...(observed.firstTaskStatus === "done" ? ["first_task_done" as const] : []), + ]; +} + +/** Closed validation is shared by normalized-result readers and explicit report rendering. */ +export function validateOnboardingPreAcceptanceEvidence(value: unknown): asserts value is OnboardingPreAcceptanceEvidence { + const invalid = (): never => { throw new Error("invalid_onboarding_pre_acceptance_evidence"); }; + const keys = Object.keys(receiptKeys); + if (!value || typeof value !== "object" || Array.isArray(value) || Object.keys(value).length !== keys.length || keys.some(key => !Object.hasOwn(value, key))) invalid(); + const receipt = value as OnboardingPreAcceptanceEvidence; + if (receipt.version !== 1 || !checkpoints.includes(receipt.checkpoint) || + !Array.isArray(receipt.violations) || !receipt.violations.length || receipt.violations.length > violations.length || + receipt.violations.some(code => !violations.includes(code)) || new Set(receipt.violations).size !== receipt.violations.length || + [receipt.taskCount, receipt.attachmentCount, receipt.documentCount, receipt.nonPlanningDocumentCount].some(count => !Number.isSafeInteger(count) || count < 0) || + receipt.nonPlanningDocumentCount > receipt.documentCount || typeof receipt.firstTaskMatchesCanonicalParent !== "boolean" || + !statuses.includes(receipt.firstTaskStatus) || !statuses.includes(receipt.canonicalParentStatus)) invalid(); + if (receipt.taskCount === 0 && (receipt.firstTaskStatus !== "missing" || receipt.canonicalParentStatus !== "missing" || receipt.firstTaskMatchesCanonicalParent) || + receipt.taskCount > 0 && receipt.firstTaskStatus === "missing" || + receipt.firstTaskMatchesCanonicalParent && receipt.firstTaskStatus !== receipt.canonicalParentStatus || + receipt.taskCount === 1 && !receipt.firstTaskMatchesCanonicalParent && receipt.canonicalParentStatus !== "missing" || + preAcceptanceViolations(receipt).join(",") !== receipt.violations.join(",")) invalid(); +} diff --git a/tests/runner-e2e/product-journeys/types.ts b/tests/runner-e2e/product-journeys/types.ts index da0b054a606..00bdeb5c164 100644 --- a/tests/runner-e2e/product-journeys/types.ts +++ b/tests/runner-e2e/product-journeys/types.ts @@ -29,6 +29,7 @@ export interface JourneyEvidence { operations?: JourneyOperations; artifact?: OnboardingArtifactEvidence; onboardingConnect?: import("./onboarding-connect.js").OnboardingConnectEvidence; + onboardingPreAcceptance?: import("./onboarding-pre-acceptance.js").OnboardingPreAcceptanceEvidence; } export interface OnboardingArtifactEvidence { schema: "paperclip.product-journey.artifact/v1"; diff --git a/tests/runner-e2e/result-validation.ts b/tests/runner-e2e/result-validation.ts index 56f38ed0bec..3d6beece13e 100644 --- a/tests/runner-e2e/result-validation.ts +++ b/tests/runner-e2e/result-validation.ts @@ -6,6 +6,7 @@ import type { RunnerE2ECampaignAccounting, RunnerE2EResult } from "./types.js"; import { DEPENDABLE_CELLS } from "./dependable-qualification.js"; import { validateOnboardingConnectEvidence } from "./product-journeys/onboarding-connect.js"; import { validateOnboardingArtifact } from "./product-journeys/artifact-evidence.js"; +import { validateOnboardingPreAcceptanceEvidence } from "./product-journeys/onboarding-pre-acceptance.js"; type Rule = (value: unknown, at: string) => void; const invalid = (at: string): never => { @@ -346,6 +347,7 @@ const fields = { targetId: optional(string), campaignId: optional(string), attemptId: optional(string), onboardingConnect: optional((value, at) => { try { validateOnboardingConnectEvidence(value); } catch { invalid(at); } }), + onboardingPreAcceptance: optional((value, at) => { try { validateOnboardingPreAcceptanceEvidence(value); } catch { invalid(at); } }), artifact: optional((value, at) => { try { validateOnboardingArtifact(value); } catch { invalid(at); } }), operations: optional(closedShape({ campaignId: string, attemptId: string, modelCostUsd: nullable(number), setupCostUsd: nullable(number), requestCount: integer, feedback: closedShape({ collection: oneOf("complete", "partial", "unavailable"), complaints: integer, suggestions: integer }), traceIds: array(string) })), @@ -404,6 +406,9 @@ export function validateRetainedRunnerResult( result(value, "result"); const checked = value as RunnerE2EResult; const artifact = checked.productJourney?.artifact; + if (checked.productJourney?.onboardingPreAcceptance && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" || + checked.status !== "failed" || checked.productJourney.outcome !== "failed" || + checked.error !== "onboarding_work_before_acceptance")) invalid("result.productJourney.onboardingPreAcceptance"); if (artifact && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" || checked.runIds && !checked.runIds.includes(artifact.originatingRunId))) invalid("result.productJourney.artifact"); if (checked.workerTiming !== undefined) { From 8161f2a813ac6da6661cb67bfdf26bac035cdba2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:09:31 -0500 Subject: [PATCH 19/41] fix(server): wake tool-review card repair from committed changes (#17468) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tool requests and invocations store the execution outcome. Review cards show that outcome. > - The maintenance tick repeatedly joins these tables even when every card matches. > - Committed changes can wake repair. Existing rows can restore work after restart. > - A large company backlog also needs a bounded pass so other companies and shutdown can proceed. > - This PR uses the shared coordinator for scoped repair, continuation and failure deadlines. Settled history becomes quiet. ## Linked Issues or Issue Description **What happened?** The aggregate maintenance tick scans tool-review projections on every tick. Matching terminal history still causes database work. A company with many stale cards can also hold one worker in an unlimited drain. **What did you expect?** Relevant writes and startup should wake repair. Failed or unresolved cards should retain recovery. Matching history should cause no recurring SQL. Each normal worker pass should honor drain and shutdown. **Steps to reproduce** Start an instance with completed tool requests and matching cards. Leave it idle. The old maintenance tick keeps querying for mismatches. A large mismatch backlog also drains without a row limit or cancellation check. This is slice 07 of the sleep migration. It now targets master after #17458 landed. Related owners are pending expiry #17453 and approved recovery #17454. Their execution duties remain separate from card repair. The prior open-PR search found no duplicate projection migration. ## What Changed - Register company-scoped projection repair with the existing delivery coordinator. Remove its recurring aggregate-tick scan. - Signal through the existing review-write transaction owner and linked interaction mutations. Ordinary unlinked confirmations do not wake repair. - Keep interpretation, locks, current-state checks, activity and continuation callbacks in the gateway. A repair never repeats a provider tool call. - Limit each normal worker pass to 500 rows. Check abort and admission between rows. Save company progress through truncated and failed passes so blocked early pages cannot hide later cards. - Schedule truncated work after one second. Back off unresolved cards from thirty seconds to five minutes; fresh committed hints reset the delay. Clear obsolete cursors after empty reconstruction. - Skip repeated company discovery for scoped hints and the extra candidate query on an empty sweep with no saved cursor. Attempted or resumed passes still verify remaining mismatches because reflection returns no settlement result. - Add generated migration `0386_steady_epoch.sql` for the existing `(company_id, interaction_id)` lookup. Use the existing concurrent-index executor. Add no table. ## Verification Current candidate: `d4706fbf2a04a00ade30181eb1bd9322e8455818`. All 222 cases in the full gateway/projection/interruption, startup and scoped-deadline suites pass without skips after the query/retry efficiency repair; server types and build pass. A single committed company hint with matching retained history and a separate empty direct sweep each perform one candidate query. The unresolved-card regression failed before repair and now verifies backoff through the five-minute cap, prompt producer activation and quiet after settlement. An initial assertion incorrectly assumed an entire approval emitted one wake; its separate real writes require isolating the single-hint measurement. The earlier master integration `fe6e10629b` additionally established the following evidence before this efficiency follow-up: - **320 cases pass** across full gateway/projection/interruption, startup, scoped-deadline, narrowed identity-read and authenticated gateway HTTP suites. The HTTP selection includes all 81 cases. - **Six real PostgreSQL regressions failed before their fixes:** pass bounds, abort, drain, progress past unprojectable pages, progress past failed pages, and first repair after empty reconstruction. The complete owning selection passes all 220 cases. - Server production typechecking and build pass. Generated index migration, snapshot drift and numbering/safety checks pass. The five newest snapshots remain; master journal entries are unchanged. - A real missing-link query on 30,000 history rows changed from a sequential scan that filtered all 30,000 rows and touched 429 buffers to the new index with two buffers. Both treatments returned no rows. - Normal master integration preserves 35 independent performance files exactly. The gateway also retains master's narrowed identity read. Commands include `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, and server Vitest selections for `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`, `run-identity-projections`, and `connection-management.integration`, with separate PostgreSQL test fixtures. [The behavior and failure audit](doc/idle-tool-review-projection.md) records the producer contract, earlier fixture failures, index measurement, red/green regressions and integration evidence. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Signals use the existing single-active-writer database owner. Another replica or independent SQL writer needs a wake contract. - An already issued database query or reflection finishes before cancellation takes effect. Checks between rows bound further dispatch. - Legacy manual/global and disabled-scheduler one-shot repairs keep their complete-sweep behavior. The normal coordinator worker always supplies its row budget and admission controls. - Unprojectable cards and thrown failures retain recovery with a five-minute backoff cap. The company mismatch query still grows with terminal history. Empty history does not retain a recurring scan. - The index migration uses concurrent creation and the existing invalid-index recovery path. Prior SQL and journal entries remain unchanged. - Cloud sleep eligibility and external-wake coverage do not change. Local worker quiet is not proof of host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-tool-review-projection.md | 229 +++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 57 ++- packages/db/src/client.ts | 1 + .../db/src/migrations/0386_steady_epoch.sql | 1 + ...{0381_snapshot.json => 0386_snapshot.json} | 107 +++++- packages/db/src/migrations/meta/_journal.json | 9 +- packages/db/src/schema/tool_access.ts | 1 + .../server-startup-feedback-export.test.ts | 2 + .../__tests__/tool-gateway-service.test.ts | 356 ++++++++++++++++++ server/src/index.ts | 8 +- .../services/delivery-work-notifications.ts | 1 + .../issue-thread-interactions.test.ts | 15 +- .../src/services/issue-thread-interactions.ts | 50 ++- server/src/services/tool-gateway.ts | 48 +-- .../services/tool-review-projection-work.ts | 33 ++ server/src/services/tool-review-projection.ts | 77 ++++ .../tool-review-work-notifications.ts | 1 + 17 files changed, 935 insertions(+), 61 deletions(-) create mode 100644 doc/idle-tool-review-projection.md create mode 100644 packages/db/src/migrations/0386_steady_epoch.sql rename packages/db/src/migrations/meta/{0381_snapshot.json => 0386_snapshot.json} (99%) create mode 100644 server/src/services/tool-review-projection-work.ts create mode 100644 server/src/services/tool-review-projection.ts diff --git a/doc/idle-tool-review-projection.md b/doc/idle-tool-review-projection.md new file mode 100644 index 00000000000..f1d431ea920 --- /dev/null +++ b/doc/idle-tool-review-projection.md @@ -0,0 +1,229 @@ +# Tool-review interaction repair + +The tool-action request and invocation are authoritative. The interaction card +is a projection of their outcome. Previously the aggregate maintenance tick +joined these tables repeatedly, even when every card already matched. + +The projection worker reconstructs affected companies at startup and reconciles +after committed changes. Producer hints carry the company ID. Subsequent +candidate discovery and paginated repair queries filter by that company; review +writes cannot drive a scan of another company's terminal history. The shared +scoped worker dispatches one company per pass with one-second minimum spacing. +It retains a 30-second retry only for a mismatch or failed reconciliation. +Configured idle tools, pending reviews without terminal outcomes, and matching +terminal history have no recurring projection SQL or provider calls. This is +worker quiet, not whole-instance suspension. + +## Ownership and trigger contract + +`tool-review-projection.ts` owns candidate discovery and pagination. The gateway +retains card interpretation, current-state checks, locks, activity publication, +and the existing continuation callback. The scheduler consumes company discovery, +`sweep(companyId)`, and the returned retry obligation. No provider payload enters shared scheduling. +No table is added. A concurrent company/interaction lookup index supports the +existing link checks and linked-request updates. + +The existing review notification helper now also signals projection repair. +Notifications are awaited before writes and inherit the outer transaction's +commit, rollback, and uncertainty tracking. Unrelated review changes can cause +one empty company-scoped reconciliation; they do not create a recurring deadline. +Interaction-only writes first check for an actual linked tool request using the +company and affected card IDs. Generic confirmation kinds alone do not signal +any tool-review queue. This also preserves recovery for legacy linked cards +without current tool payload flags. These reads happen in the mutation owner, +not a background poll. Sibling selection shares the update predicate and runs +under the existing issue lock before publishing the notification. + +| Producer | Notification ownership | +| --- | --- | +| Gateway signing, approval inference, execution claims, provider outcomes, and invalidation | Existing `writeToolReviewChange`; the closed-issue expiry path now uses it too | +| Human approval/rejection | `commitToolActionReview` signals in its existing transaction | +| Pending expiry, approved recovery, interrupted claims, Test-tab settlement | Existing review-write boundary | +| Review-queue invalidation | Existing review-write boundary | +| Interaction withdrawal, skip, terminal-issue expiry | Linked-request resolution signals before the mutation | +| Linked tool-card accept/reject, deleted-addressee cancellation | Projection notification before replacing card state in the owning transaction; unlinked confirmations do not notify | +| Sibling supersession, startup duplicate cleanup, new/historical comment supersession | Projection notification before card replacement, including cases where the request is already terminal | +| Comment-based confirmation resolution | Calls the interaction owner inside the same outer transaction; its later comment-ID update shares that signal | + +The audited other interaction writers do not create this mismatch: native bridge +and status-committer summary/status-only updates preserve the projection; native +review retirement owns separate completion-review cards; connection and secret +flows own their separate card types. Document-staleness and plan-resume paths +use document targets, while tool cards use `tool-action:` custom targets. +Reflection consent uses its instruction/profile/skill target keys, not tool-action +keys. A future writer that replaces tool-card result state must use the same +transaction notification boundary. A future writer that changes authoritative +tool outcomes must signal the review boundary. + +## Repair, concurrency, and lifecycle + +The mismatch predicate preserves the previous policy: terminal executed, failed, +expired, or cancelled requests with a linked interaction whose displayed tool +status differs. Cancelled requests display `expired`. Invocation error and result +summaries come from saved state. Repair never repeats a provider action. + +Candidate queries page in request-ID order, 100 rows at a time. A card that cannot +yet be projected does not starve later pages. Per-row failures do not prevent +attempting the remaining candidates; failures keep the retry. A final mismatch +check retains retries for no-op repairs rather than silently declaring them done. +Failed companies retain their own backoff across notifications while other +companies progress. An unresolved no-op projection retains a 30-second deadline; +a new relevant company change can make it actionable sooner. A company pass +still drains all its pages, preserving progress past an unprojectable card. +The joins inspect that company's history on activation. This change removes repeated +idle scans; it does not introduce an indexed projection-debt table or claim that +an activated reconciliation has constant cost independent of history size. + +The gateway locks the interaction and request, then checks the current request +status. If another repair already wrote the same status and outcome fields, +it avoids a second write and duplicate activity. Different saved details still +use the existing update path. Missing projection state remains repairable. + +The shared deadline worker retains first writes during drain, failed retries, +and changes arriving during reconciliation. Shutdown joins coordinator work. +Startup reconstructs debt from existing rows. With heartbeat scheduling disabled, +startup still performs one pass, preserving the existing disabled-mode contract. +The public/manual combined review sweep remains available; server maintenance +no longer calls it periodically. + +## Continuation and sleep limits + +Card state is not proof that continuation delivery finished. Existing +`tool_action_deliveries` receipts and their coordinator remain responsible for +that work. Activity publication still occurs after the card transaction; this +slice does not add exactly-once activity delivery or a new outbox. + +Signals remain process-local to the supported database owner. Independent SQL +writers and another active server require a separate notification/reconciliation +contract. Cloud eligibility still conservatively blocks retained tool-action +requests. No external wake or sleep-safety guard is relaxed here. + +## Verification + +Twelve new real-PostgreSQL cases exercise normal approval and single execution, +quiet configured and terminal-history states, four persisted outcome states, +outer commit/rollback, first write during drain, failure retry through drain, +foreground/worker repair contention, multi-page progress past an unprojectable +card, and an administrative card overwrite that reactivates repair. Quiet checks +observe absence of database reads across an hour of controlled time. + +The first focused run caught a missing-projection null access in the new +idempotency check; optional access fixes it. The pagination fixture also attempted +to change a referenced request ID; it now orders the additional candidates without +mutating that key. The focused rerun passed all eleven then-existing tests. +After adding the concurrent repair case, all 148 service/startup tests and 92 +gateway acceptance tests passed. Workspace typechecking and build passed. All 469 tool-access and 106 interaction +service tests also pass (815 tests across the final selections). The first +interaction run passed 105 cases but timed out in one cleanup hook during a +53-minute local run; the isolated full rerun passed in nine seconds. No production +change or timeout increase was made to force that result. Full repository tests +will run in CI; the local selections do not claim a complete repository test run. +Fresh PR CI/review remains required. + +Company-scoping follow-up reproduced two regressions: captured recovery SQL lacked +a company predicate, and ordinary confirmation creation/replacement/cancellation +published three unnecessary tool-projection wakes. Both regressions now pass +through real producers. Additional coverage keeps failed-company retry timing +independent, proves ordinary mutations still complete without recovery scans, +and exercises administrative repair of legacy cards identified by their saved +link rather than payload flags. + +Company-scoping follow-up verification: all 180 gateway and startup cases and all 152 interaction and secret-proposal route cases passed against dedicated PostgreSQL test databases. Server typechecking passed. These are local checks; stack CI and deployed sleep qualification remain separate. + + +## Indexed interaction links + +Review found that the new link check had no supporting interaction index. An +ordinary unlinked confirmation could scan all tool-action history for its company. +The generated `0386_steady_epoch.sql` migration adds the non-unique +`(company_id, interaction_id)` index concurrently, and the existing top-level +migration executor handles its creation and interrupted-build recovery. The +schema and generated snapshot describe the same index; older SQL and journal +entries are preserved and snapshot pruning retains the five newest snapshots. + +A real PostgreSQL check applied the migration to a fresh dedicated test database +and seeded 30,000 tool-action rows. The ordinary missing-link query returned no +rows in both treatments: without the index it performed a sequential scan, +filtered 30,000 rows and touched 429 shared buffers; with the index it used the +new index and touched two buffers. These are local planner measurements, not a +fleet latency or cost claim. The baseline index removal was rolled back in that +dedicated database, and the index remained valid afterwards. An initial probe +failed while binding a JSON fixture parameter before any comparison; correcting +the parameter encoding made the complete probe pass. That failed fixture is not +counted as a product regression. + +Migration numbering/safety checks, generated snapshot drift, server types and +server build pass. All 214 cases in the full gateway/projection/startup/scoped- +deadline selection pass against a separate freshly migrated database, preserving +notification quiet, company isolation, transaction settlement, and interruption +behavior. Latest-head CI and automated reviews remain required merge gates. + +## Bounded projection and lifecycle interruption + +A review found that one company could drain an unlimited backlog in a single +worker pass. Five real PostgreSQL regressions reproduced 501 rows being consumed +instead of the 500-row budget, missing abort/drain handling, and the absence of +continuation past a full page of unprojectable or failed cards. + +The worker now gives its existing service a 500-row budget, the coordinator's +abort signal, and its admission predicate. The service checks admission between +rows and reads at most 100 candidates per page. It retains the last attempted +request ID for each unfinished company, including after a failed pass, so a +poisoned early page cannot hide later healthy cards. A truncated pass schedules +a one-second continuation; the existing shared per-company fairness and failure +backoff remain in force. A completed pass that still has an unprojectable card +backs off from thirty seconds to five minutes. Fresh committed company changes +reset that delay and wake the affected scope promptly. These are real unresolved projections, +not an unconditional empty-history scan. + +Reconstruction clears continuation state for companies that no longer have +mismatches. A sixth real regression reproduced an abandoned cursor skipping the +first new earlier card after an empty reconstruction; clearing that cursor makes +the first new pass repair it immediately. The maps contain no durable truth: +restart reconstructs from the existing rows, and completed rows stop matching. +There is no cursor table, independent timer or per-producer recovery wrapper. + +Legacy manual/global and scheduler-disabled one-shot repair calls retain their +complete-sweep behavior unless they request a budget. The normal coordinator +worker always supplies the bounded options. Cancellation is observed between +rows and pages; an already issued database query or reflection finishes before +the service stops. A sweep never replays a provider tool action. + +All 220 cases across the full gateway/projection/interruption, startup and scoped +deadline suites pass, with no skipped cases. The five initial regressions and +sixth cursor regression failed before their fixes; server types and build pass. +The earlier 219-case acceptance remains evidence for the preceding revision. +Fresh integrated-head CI and both automated reviews remain merge gates. + + +After normal reconciliation with master `f6bc653f7f`, all 320 integrated cases +pass, including the 81 authenticated gateway HTTP cases and the upstream narrowed +identity-read tests. Server types and build pass. Thirty-five independent master +performance files are preserved exactly; the gateway's narrowed identity read +is preserved alongside the projection extraction. The pre-integration 220-case +selection above remains evidence for its stated revision. + + +## Review efficiency follow-up + +A company-scoped hint now enters the bounded owner directly rather than first +repeating company discovery. Startup/global reconstruction still discovers +companies from existing rows. An empty sweep with no saved cursor returns after +one candidate query; a resumed cursor or attempted reflection still needs the +final query to detect unresolved earlier cards. The reflection callback returns +no settlement result, so an attempted reflection cannot establish that a card +was repaired. + +Returned unresolved work now backs off through 30, 60, 120, 240 and 300 seconds, +matching the shared error-backoff cap. Truncated backlog continuation remains +one second. Settlement removes the retry; a new committed hint resets backoff. +No pending row is discarded, and no independent timer or table is added. + +The fixed-cadence regression failed before repair and now verifies the full +backoff cap, producer reactivation and quiet after settlement. An initial scan +assertion incorrectly assumed that a complete approval emitted only one wake; +its separate legitimate writes caused two scans even after optimization. The +corrected case isolates one committed hint and verifies one candidate scan with +matching history, followed by an empty direct sweep with one query. All 222 +cases in the full gateway, startup and scoped-deadline suites pass without skips; +server typecheck and build pass. Fresh-head CI and both reviews are still required. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index b8976fb34fb..25d3ebf70bc 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -18,7 +18,7 @@ Do not mark an item done from a timer removal or a green check alone. | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | -| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), merged after current-head CI and both 5/5 reviews; interrupted claims: [#17458](https://github.com/paperclipai/paperclip/pull/17458), integration verified and awaiting fresh CI/review; projection: [#17468](https://github.com/paperclipai/paperclip/pull/17468), interaction-link index verified and awaiting fresh CI/review | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), merged after current-head CI and both 5/5 reviews; interrupted claims: [#17458](https://github.com/paperclipai/paperclip/pull/17458), merged after current-head CI and both 5/5 reviews; projection: [#17468](https://github.com/paperclipai/paperclip/pull/17468), interaction-link index verified and awaiting fresh CI/review | | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | @@ -280,3 +280,58 @@ initial concurrent-run timeout is retained above as evidence, not hidden. Workspace typecheck/build and migration safety/numbering pass. The first four PRs are mergeable and have all checks passed at their recorded heads. #17454 is mergeable with fresh checks and review in progress at `7a1cf03e62`. + +Slice 06 is [#17458](https://github.com/paperclipai/paperclip/pull/17458), head +`85003a7ac4`, based on #17454. Greptile review was requested and the PR is +attached to this chat. Slice 05 has fresh Greptile 5/5 at `7a1cf03e62`; CI is +running with no failure in the latest snapshot. The next branch is +`codex/sleep-07-tool-review-projection`, based on #17458. No seventh implementation +or readiness is claimed yet. The full remaining inventory stays in scope. + + +## Slice 07: tool-review interaction projection + +The implementation removes the last combined review sweep from the main tick. +It pages terminal mismatches and retains retries only for actual debt or failures. +Gateway reflection keeps locks, current-state checks, activity, and continuation +ownership; concurrent identical repairs do not write twice. The producer audit +also covers administrative card result replacements, before the mutation in the +existing transaction. See [projection repair](../idle-tool-review-projection.md). + +Twelve new lifecycle cases are included in 148 passing service/startup tests. +Gateway acceptance passes all 92 tests and workspace typechecking passes. The +initial missing-state bug and referenced-key fixture failure are recorded in the +behavior document. Build and additional consumer regressions are running; no +seventh PR is claimed ready yet. + +Historical parent evidence at the initial migration: #17454 at `7a1cf03e62` +and #17458 at `85003a7ac4` passed the recorded CI matrices and Greptile review. +Those snapshots preceded their current-head review and landing. The first six +stack PRs have now merged. The full remaining inventory is still in scope. + +Initial slice 07 local verification: 815 selected tests passed, with workspace +typecheck and build passing. One initial interaction cleanup-hook timeout is recorded in +the behavior document; the complete isolated suite then passed. Both parent +PRs #17454 and #17458 now have every check passed and are mergeable, including +the additional Paperclip Review checks. Full CI/review for slice 07 is next. + + +Slice 07 current integration: master `f6bc653f7f` contains the first six landed +slices and the independent identity/resume/sidebar performance work. This slice +adds generated concurrent interaction-link index migration `0386_steady_epoch` +and retains five schema snapshots. Projection worker passes stop at 500 rows, +honor abort/drain between rows, and resume past failed or unprojectable pages; +empty reconstruction clears obsolete cursors. Six real regressions failed before +the fixes. All 320 integrated gateway/projection/startup/deadline/identity/HTTP +cases, server types and build pass; 35 independent master files are preserved +exactly. Current-head CI and Greptile/Ponytail review remain required. Earlier +case counts above describe the broader selections at their original revisions. + + +Slice 07 follow-up: scoped hints avoid redundant discovery, empty sweeps avoid a +second candidate read, and unresolved-card retries back off to five minutes. +Fresh writes reactivate promptly and settlement is quiet. The full affected +gateway/startup/deadline selection passes 222 cases; server types/build pass. +The initial invalid one-wake-per-approval assertion and corrected isolated-hint +measurement are retained in the behavior document. Current-head CI/reviews remain +required before this slice can merge. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 3e4cce15e2e..342256ac462 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -504,6 +504,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0381_mean_scream.sql", "0382_narrow_the_executioner.sql", "0385_open_madame_web.sql", + "0386_steady_epoch.sql", "0384_funny_adam_warlock.sql", "0383_tool_review_company_expiry.sql", ]); diff --git a/packages/db/src/migrations/0386_steady_epoch.sql b/packages/db/src/migrations/0386_steady_epoch.sql new file mode 100644 index 00000000000..06356e30ced --- /dev/null +++ b/packages/db/src/migrations/0386_steady_epoch.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "tool_action_requests_company_interaction_idx" ON "tool_action_requests" USING btree ("company_id","interaction_id"); diff --git a/packages/db/src/migrations/meta/0381_snapshot.json b/packages/db/src/migrations/meta/0386_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0381_snapshot.json rename to packages/db/src/migrations/meta/0386_snapshot.json index 23802aaeece..8dd26eef4fb 100644 --- a/packages/db/src/migrations/meta/0381_snapshot.json +++ b/packages/db/src/migrations/meta/0386_snapshot.json @@ -1,6 +1,6 @@ { - "id": "2e93deb7-f24c-42d1-8c8e-30b21cff90bc", - "prevId": "bfda87fa-662b-47bb-87c0-dabc814aebd7", + "id": "da3229b5-4eab-4d8b-ba6b-d756e6c42430", + "prevId": "9965d300-e355-4f47-ba0a-951ed3b4b8a2", "version": "7", "dialect": "postgresql", "tables": { @@ -52812,6 +52812,88 @@ "method": "btree", "with": {} }, + "tool_action_requests_pending_expiry_idx": { + "name": "tool_action_requests_pending_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_company_pending_expiry_idx": { + "name": "tool_action_requests_company_pending_expiry_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_executing_deadline_idx": { + "name": "tool_action_requests_executing_deadline_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'executing'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_approved_idx": { + "name": "tool_action_requests_approved_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'approved'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_invocation_idx": { "name": "tool_action_requests_invocation_idx", "columns": [ @@ -52827,6 +52909,27 @@ "method": "btree", "with": {} }, + "tool_action_requests_company_interaction_idx": { + "name": "tool_action_requests_company_interaction_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "interaction_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_issue_idx": { "name": "tool_action_requests_issue_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index f4cfed69f11..578d8027ed9 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2689,6 +2689,13 @@ "when": 1793060791729, "tag": "0385_open_madame_web", "breakpoints": true + }, + { + "idx": 386, + "version": "7", + "when": 1793135146893, + "tag": "0386_steady_epoch", + "breakpoints": true } ] -} +} \ No newline at end of file diff --git a/packages/db/src/schema/tool_access.ts b/packages/db/src/schema/tool_access.ts index 9f79743d6c3..b8b317ddb91 100644 --- a/packages/db/src/schema/tool_access.ts +++ b/packages/db/src/schema/tool_access.ts @@ -839,6 +839,7 @@ export const toolActionRequests = pgTable( index("tool_action_requests_executing_deadline_idx").on(table.updatedAt, table.id).where(sql`${table.status} = 'executing'`), index("tool_action_requests_approved_idx").on(table.id).where(sql`${table.status} = 'approved'`), index("tool_action_requests_invocation_idx").on(table.invocationId), + index("tool_action_requests_company_interaction_idx").on(table.companyId, table.interactionId), index("tool_action_requests_issue_idx").on(table.companyId, table.issueId), ], ); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 1a9fcf51576..cb7da55e61a 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -72,6 +72,8 @@ const { settleInterruptedActionReviews: vi.fn(async () => ({ scanned: 0 })), nextActionReviewInterruptionAt: vi.fn(async () => null), recoverApprovedActionReviews: vi.fn(async () => ({ scanned: 0, retry: false })), + repairActionReviewProjections: vi.fn(async () => ({ scanned: 0, retry: false })), + actionReviewProjectionCompanies: vi.fn(async () => []), approvedActionReviewCompanies: vi.fn(async () => []), sweepActionReviews: vi.fn(async () => ({ scanned: 0 })), sweepExpiredActionReviews: vi.fn(async () => ({ scanned: 0 })), diff --git a/server/src/__tests__/tool-gateway-service.test.ts b/server/src/__tests__/tool-gateway-service.test.ts index 35dabccbe22..0d277ee4dbf 100644 --- a/server/src/__tests__/tool-gateway-service.test.ts +++ b/server/src/__tests__/tool-gateway-service.test.ts @@ -1,3 +1,6 @@ +import { toolReviewProjectionService } from "../services/tool-review-projection.js"; +import { registerToolReviewProjectionWork } from "../services/tool-review-projection-work.js"; +import { issueThreadInteractionService } from "../services/issue-thread-interactions.js"; import { registerToolReviewInterruptionWork } from "../services/tool-review-interruption-work.js"; import { writeToolReviewChange } from "../services/tool-review-work-notifications.js"; import { deleteCompany } from "../services/company-deletion.js"; @@ -281,6 +284,8 @@ describeEmbeddedPostgres("tool gateway service", () => { await coordinator?.stop(); coordinator = undefined; vi.useRealTimers(); vi.restoreAllMocks(); await db.execute(sql`drop trigger if exists sleep_review_fail on tool_invocations`); + await db.execute(sql`drop trigger if exists sleep_projection_fail on issue_thread_interactions`); + await db.execute(sql`drop function if exists sleep_projection_fail()`); await db.execute(sql`drop function if exists sleep_review_fail()`); await db.execute(sql`drop sequence if exists sleep_review_failure_attempt`); expect(errors).toEqual([]); @@ -352,6 +357,357 @@ describeEmbeddedPostgres("tool gateway service", () => { return { provider, release, started }; } + describe("tool review projection worker", () => { + async function startProjection() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerToolReviewProjectionWork(coordinator, { sweep: gateway.repairActionReviewProjections, companies: gateway.actionReviewProjectionCompanies }, () => !draining).ready; + } + async function savedOutcome(pending: Awaited>, source = db, + status: "executed" | "failed" | "expired" | "cancelled" = "executed") { + await writeToolReviewChange(source, pending.companyId, async tx => { + await tx.update(toolInvocations).set({ status: status === "executed" ? "succeeded" : "failed", + errorCode: status === "executed" ? null : "saved_failure", errorMessage: null, + resultSummary: { summary: "Durable result" }, completedAt: new Date() }) + .where(eq(toolInvocations.id, pending.invocationId)); + await tx.update(toolActionRequests).set({ status, updatedAt: new Date() }) + .where(eq(toolActionRequests.id, pending.id)); + }); + } + async function acceptedCard(pending: Awaited>) { + expect(pending.interactionId).not.toBeNull(); + await db.update(issueThreadInteractions).set({ status: "accepted", result: { version: 1, outcome: "accepted" } }) + .where(eq(issueThreadInteractions.id, pending.interactionId!)); + } + async function projection(pending: Awaited>) { + const [row] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, pending.interactionId!)); + return (row.result as { toolAction?: { status: string; resultSummary: string; errorCode: string | null } } | null)?.toolAction; + } + + it("becomes quiet with configuration and repairs a real foreground approval without replay", async () => { + await startProjection(); await quiet(); + const pending = await request(); + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + await quiet(); expect(await projection(pending)).toMatchObject({ status: "executed" }); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toHaveLength(1); + const before = (await db.select().from(activityLog).where(eq(activityLog.action, "issue.thread_interaction_updated"))).length; + await gateway.repairActionReviewProjections(); + expect((await db.select().from(activityLog).where(eq(activityLog.action, "issue.thread_interaction_updated"))).length).toBe(before); + }); + + it("keeps projection scans scoped to the company whose review changed", async () => { + const other = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company)); + const foreign = await request(gateway, db, other); + await gateway.approveActionRequest({ companyId: other.company.id, actionRequestId: foreign.id, actor: { userId: "board" } }); + await startProjection(); await quiet(); + const before = await projection(foreign); + const reads: Array<{ query: string; params: unknown[] }> = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query, params) => { + if (/^select/i.test(query) && query.includes('"tool_action_requests"') && query.includes("coalesce(")) + reads.push({ query, params: [...params] }); + }; + try { + for (let index = 0; index < 3; index++) { + const pending = await request(); + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + await settle(1500); expect(await projection(pending)).toMatchObject({ status: "executed" }); + } + } finally { db.$client.options.debug = debug; } + expect(reads.length).toBeGreaterThan(0); + for (const read of reads) { + expect(read.query).toContain('"tool_action_requests"."company_id" ='); + expect(read.params).toContain(fixture.company.id); + expect(read.params).not.toContain(other.company.id); + } + expect(await projection(foreign)).toEqual(before); await quiet(); + }); + + it("uses one candidate scan for a committed company hint with matching retained history", async () => { + const pending = await request(); + await gateway.approveActionRequest({ companyId: fixture.company.id, actionRequestId: pending.id, actor: { userId: "board" } }); + await startProjection(); await quiet(); + const reads: string[] = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query) => { + if (/^select/i.test(query) && query.includes('"tool_action_requests"') && query.includes("coalesce(")) reads.push(query); + }; + try { + await writeToolReviewChange(db, pending.companyId, async tx => { + await tx.update(toolActionRequests).set({ updatedAt: new Date() }).where(eq(toolActionRequests.id, pending.id)); + }); + await settle(); + expect(reads).toHaveLength(1); + await quiet(); + expect(reads).toHaveLength(1); + reads.length = 0; + expect(await gateway.repairActionReviewProjections(fixture.company.id)).toMatchObject({ scanned: 0, retry: false }); + expect(reads).toHaveLength(1); + } finally { db.$client.options.debug = debug; } + }); + + it("backs off unresolved cards to five minutes and clears the retry after settlement", async () => { + const pending = await request(); + await savedOutcome(pending, db, "failed"); + await startProjection(); + for (const delay of [30_000, 60_000, 120_000, 240_000, 300_000, 300_000]) { + const next = coordinator!.nextWakeAt(); + expect(next).not.toBeNull(); + expect(next! - Date.now()).toBeGreaterThan(delay - 1500); + expect(next! - Date.now()).toBeLessThanOrEqual(delay); + await settle(next! - Date.now() + 100); + } + expect(await projection(pending)).toBeUndefined(); + await acceptedCard(pending); + await writeToolReviewChange(db, pending.companyId, async tx => { + await tx.update(toolActionRequests).set({ updatedAt: new Date() }).where(eq(toolActionRequests.id, pending.id)); + }); + await settle(); + expect(await projection(pending)).toMatchObject({ status: "failed" }); + await quiet(); + }); + + it("does not wake projection repair for ordinary confirmation creation, replacement, or cancellation", async () => { + await startProjection(); await quiet(); + const interactions = issueThreadInteractionService(db); + const scope = { id: fixture.issue.id, companyId: fixture.company.id }; + const [recipient] = await db.insert(agents).values({ companyId: fixture.company.id, + name: "Plan recipient", role: "engineer", adapterType: "process" }).returning(); + const input = { kind: "request_confirmation" as const, addresseeAgentId: recipient.id, + payload: { version: 1 as const, prompt: "Approve this plan?" } }; + const signal = vi.fn(); + const unsubscribe = subscribeDeliveryWork(db, DELIVERY_QUEUES.toolReviewProjection, signal); + const reads: string[] = []; + const debug = db.$client.options.debug; + db.$client.options.debug = (_id, query) => { + if (/^select/i.test(query) && query.includes('"tool_action_requests"') && query.includes("coalesce(")) reads.push(query); + }; + try { + const first = await interactions.create(scope, input, { agentId: fixture.agent.id }); + const second = await interactions.create(scope, input, { agentId: fixture.agent.id }); + const cancelled = await interactions.cancelPendingForDeletedAddressee(fixture.company.id, recipient.id); + expect(cancelled.map(row => row.id)).toEqual([second.id]); + expect((await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, first.id)))[0].status).toBe("expired"); + await settle(); + expect(signal).not.toHaveBeenCalled(); + expect(reads).toEqual([]); + await quiet(); + } finally { unsubscribe(); db.$client.options.debug = debug; } + }); + + it.each(["executed", "failed", "expired", "cancelled"] as const)("reconstructs a saved %s result after restart and leaves terminal history quiet", async status => { + const pending = await request(); await acceptedCard(pending); await savedOutcome(pending, db, status); + await startProjection(); await quiet(); + expect(await projection(pending)).toMatchObject({ status: status === "cancelled" ? "expired" : status, + resultSummary: "Durable result", errorCode: status === "executed" ? null : "saved_failure" }); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); + }); + + it("retains a first terminal write during drain without passive scans", async () => { + const pending = await request(); await acceptedCard(pending); + await startProjection(); await quiet(); draining = true; + await savedOutcome(pending); await settle(); + const reads = vi.spyOn(db, "select"); + await settle(60 * 60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + expect(await projection(pending)).toBeUndefined(); + draining = false; await settle(); await quiet(); + expect(await projection(pending)).toMatchObject({ status: "executed" }); + }); + + it.each([false, true])("waits for the outer outcome transaction (rollback %s)", async rollback => { + const pending = await request(); await acceptedCard(pending); await startProjection(); await quiet(); + let entered!: () => void, release!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const aborted = new Error("fixture rollback"); + const running = db.transaction(async tx => { + await savedOutcome(pending, tx as unknown as typeof db); + entered(); await gate; if (rollback) throw aborted; + }).catch(error => { if (error !== aborted) throw error; }); + await ready; + try { await vi.advanceTimersByTimeAsync(60_000); expect(await projection(pending)).toBeUndefined(); } + finally { release(); await running; } + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(idleWorkSnapshot().active).toBe(0); + if (rollback) expect(await projection(pending)).toBeUndefined(); + else expect(await projection(pending)).toMatchObject({ status: "executed" }); + }, { timeout: 5000 }); + await quiet(); + }); + + it("retries a failed projection through drain without executing the saved action", async () => { + const pending = await request(); await acceptedCard(pending); await savedOutcome(pending); + await db.execute(sql`create function sleep_projection_fail() returns trigger language plpgsql as $$ + begin raise exception 'projection fixture failure'; end $$`); + await db.execute(sql`create trigger sleep_projection_fail before update on issue_thread_interactions + for each row when (new.result->'toolAction'->>'status' = 'executed') execute function sleep_projection_fail()`); + await startProjection(); expect(errors.length).toBeGreaterThan(0); errors.length = 0; + expect(await projection(pending)).toBeUndefined(); + draining = true; + await db.execute(sql`drop trigger sleep_projection_fail on issue_thread_interactions`); + await settle(60 * 60_000); draining = false; + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(await projection(pending)).toMatchObject({ status: "executed" }); + }, { timeout: 5000 }); + await quiet(); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); + }); + + it("backs off a failing company while another company's projection completes", async () => { + const other = await createRunFixture(db); + await db.insert(toolPolicies).values(reviewNotePolicyRow(other.company)); + const foreign = await request(gateway, db, other); await acceptedCard(foreign); await savedOutcome(foreign, db, "failed"); + await db.execute(sql`create function sleep_projection_fail() returns trigger language plpgsql as $$ + begin raise exception 'company projection fixture failure'; end $$`); + await db.execute(sql`create trigger sleep_projection_fail before update on issue_thread_interactions + for each row when (new.result->'toolAction'->>'status' = 'failed') execute function sleep_projection_fail()`); + await startProjection(); expect(errors).toHaveLength(1); errors.length = 0; + const retryAt = coordinator!.nextWakeAt(); expect(retryAt).not.toBeNull(); + const pending = await request(); await acceptedCard(pending); await savedOutcome(pending); + await request(gateway, db, other); + await settle(5000); + expect(await projection(pending)).toMatchObject({ status: "executed" }); + expect(await projection(foreign)).toBeUndefined(); + expect(errors).toEqual([]); expect(coordinator!.nextWakeAt()).toBe(retryAt); + await db.execute(sql`drop trigger sleep_projection_fail on issue_thread_interactions`); + await settle(); await quiet(); + expect(await projection(foreign)).toMatchObject({ status: "failed" }); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); + }); + + it("rechecks the projection under lock when foreground repair wins the race", async () => { + const pending = await request(); await acceptedCard(pending); await savedOutcome(pending); + const transaction = db.transaction.bind(db); + let entered!: () => void, release!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const intercepted = vi.spyOn(db, "transaction").mockImplementationOnce(async callback => { + entered(); await gate; return transaction(callback); + }); + const repair = gateway.repairActionReviewProjections(); + await ready; + try { await gateway.repairActionReviewProjections(); } finally { release(); } + await repair; intercepted.mockRestore(); + const updates = await db.select().from(activityLog).where(and(eq(activityLog.companyId, fixture.company.id), + eq(activityLog.action, "issue.thread_interaction_updated"))); + expect(updates).toHaveLength(1); + expect(await projection(pending)).toMatchObject({ status: "executed" }); + await startProjection(); await quiet(); + }); + + async function projectionBacklog(count: number) { + const pending = await request(); await acceptedCard(pending); await savedOutcome(pending, db, "failed"); + const [template] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, pending.interactionId!)); + const cards = Array.from({ length: count - 1 }, () => ({ ...template, id: randomUUID(), idempotencyKey: randomUUID() })); + if (cards.length) await db.insert(issueThreadInteractions).values(cards); + const requests = cards.map((card, index) => ({ ...pending, + id: `ffffffff-ffff-4fff-afff-${String(index).padStart(12, "0")}`, + interactionId: card.id, status: "failed" as const })); + if (requests.length) await db.insert(toolActionRequests).values(requests); + const cardByRequest = new Map([pending, ...requests].map(row => [row.id, row.interactionId!])); + const reflect = async (id: string) => { + await db.update(issueThreadInteractions).set({ result: sql`${issueThreadInteractions.result} + || ${JSON.stringify({ toolAction: { status: "failed" } })}::jsonb` }) + .where(eq(issueThreadInteractions.id, cardByRequest.get(id)!)); + }; + return { cardByRequest, reflect }; + } + + it("bounds a projection worker pass and continues its saved backlog", async () => { + await projectionBacklog(501); + await startProjection(); + const projected = async () => (await db.select().from(issueThreadInteractions)) + .filter(card => (card.result as { toolAction?: { status: string } } | null)?.toolAction?.status === "failed").length; + expect(await projected()).toBe(500); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + await settle(2000); await quiet(); + expect(await projected()).toBe(501); + expect(await db.select().from(toolCallEvents).where(eq(toolCallEvents.reasonCode, "approved_action_executed"))).toEqual([]); + }); + + it("clears an abandoned projection cursor after empty reconstruction before the next write", async () => { + const backlog = await projectionBacklog(3); + const service = toolReviewProjectionService(db, input => backlog.reflect(input.actionRequestId)); + expect(await service.sweep(fixture.company.id, { maxRows: 1 })).toMatchObject({ scanned: 1, truncated: true }); + for (const id of backlog.cardByRequest.keys()) await backlog.reflect(id); + expect(await service.companies(fixture.company.id)).toEqual([]); + const firstId = [...backlog.cardByRequest.keys()].sort()[0]!; + await db.update(issueThreadInteractions).set({ result: sql`${issueThreadInteractions.result} - 'toolAction'` }) + .where(eq(issueThreadInteractions.id, backlog.cardByRequest.get(firstId)!)); + expect(await service.sweep(fixture.company.id, { maxRows: 500 })).toMatchObject({ scanned: 1, retry: false }); + }); + + it.each(["abort", "drain"] as const)("stops projection between rows on %s and resumes without losing cards", async mode => { + const backlog = await projectionBacklog(5), controller = new AbortController(); + const seen: string[] = []; let admitted = true; + const service = toolReviewProjectionService(db, async input => { + seen.push(input.actionRequestId); await backlog.reflect(input.actionRequestId); + if (seen.length === 1) { if (mode === "abort") controller.abort(); else admitted = false; } + }); + expect(await service.sweep(fixture.company.id, { maxRows: 500, signal: controller.signal, shouldContinue: () => admitted })) + .toMatchObject({ scanned: 1, retry: true, truncated: true }); + expect(seen).toHaveLength(1); + admitted = true; + expect(await service.sweep(fixture.company.id, { maxRows: 500 })).toMatchObject({ scanned: 4, retry: false }); + expect(new Set(seen).size).toBe(5); + }); + + it.each(["unprojectable", "failed"] as const)("continues beyond a full %s page before revisiting it", async mode => { + const backlog = await projectionBacklog(501); + const blocked = new Set([...backlog.cardByRequest.keys()].sort().slice(0, 500)), seen: string[] = []; + const service = toolReviewProjectionService(db, async input => { + seen.push(input.actionRequestId); + if (blocked.has(input.actionRequestId)) { if (mode === "failed") throw new Error("saved card unavailable"); return; } + await backlog.reflect(input.actionRequestId); + }); + const first = service.sweep(fixture.company.id, { maxRows: 500 }); + if (mode === "failed") await expect(first).rejects.toBeInstanceOf(AggregateError); + else expect(await first).toMatchObject({ scanned: 500, retry: true, truncated: true }); + expect(seen).toHaveLength(500); seen.length = 0; + expect(await service.sweep(fixture.company.id, { maxRows: 500 })).toMatchObject({ scanned: 1, retry: true, truncated: false }); + expect(seen).toHaveLength(1); expect(blocked.has(seen[0]!)).toBe(false); + }); + + it("does not starve later pages behind a card that cannot yet reflect its outcome", async () => { + const pending = await request(); await savedOutcome(pending, db, "failed"); + const [template] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, pending.interactionId!)); + const copies = Array.from({ length: 101 }, () => ({ ...template, id: randomUUID(), idempotencyKey: randomUUID(), + status: "accepted" as const, result: { version: 1 as const, outcome: "accepted" as const } })); + await db.insert(issueThreadInteractions).values(copies); + await db.insert(toolActionRequests).values(copies.map((card, index) => ({ ...pending, + id: `ffffffff-ffff-4fff-afff-${String(index).padStart(12, "0")}`, + interactionId: card.id, status: "failed" as const }))); + await startProjection(); + expect(coordinator!.nextWakeAt()).not.toBeNull(); + const cards = await db.select().from(issueThreadInteractions); + expect(cards.filter(card => (card.result as { toolAction?: { status: string } } | null)?.toolAction?.status === "failed")).toHaveLength(101); + await db.update(issueThreadInteractions).set({ addresseeAgentId: fixture.agent.id }).where(eq(issueThreadInteractions.id, pending.interactionId!)); + await issueThreadInteractionService(db).cancelPendingForDeletedAddressee(fixture.company.id, fixture.agent.id); + await settle(); await quiet(); + expect(await projection(pending)).toMatchObject({ status: "failed" }); + }); + + it.each([false, true])("reactivates when an administrative card mutation overwrites a settled projection (legacy %s)", async legacy => { + const pending = await request(); + if (legacy) { + const [card] = await db.select().from(issueThreadInteractions).where(eq(issueThreadInteractions.id, pending.interactionId!)); + const payload = { ...card.payload } as Record; + delete payload.toolAction; delete payload.target; + await db.update(issueThreadInteractions).set({ payload }).where(eq(issueThreadInteractions.id, pending.interactionId!)); + } + await db.update(issueThreadInteractions).set({ addresseeAgentId: fixture.agent.id, + result: { version: 1, outcome: "accepted" } }).where(eq(issueThreadInteractions.id, pending.interactionId!)); + await savedOutcome(pending, db, "failed"); await startProjection(); await quiet(); + expect(await projection(pending)).toMatchObject({ status: "failed" }); + await issueThreadInteractionService(db).cancelPendingForDeletedAddressee(fixture.company.id, fixture.agent.id); + await settle(); await quiet(); + expect(await projection(pending)).toMatchObject({ status: "failed", errorCode: "saved_failure" }); + }); + }); + describe("interrupted review recovery worker", () => { async function startInterruption() { coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, diff --git a/server/src/index.ts b/server/src/index.ts index 50627307a54..bef68b441e7 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerToolReviewProjectionWork } from "./services/tool-review-projection-work.js"; import { registerToolReviewInterruptionWork } from "./services/tool-review-interruption-work.js"; import { registerToolReviewRecoveryWork } from "./services/tool-review-recovery-work.js"; import { registerToolReviewExpiryWork } from "./services/tool-review-expiry-work.js"; @@ -1486,7 +1487,10 @@ async function startServerWithDatabaseTeardown( } else { await app.locals.toolGateway.settleInterruptedActionReviews(); } - return app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false, includeInterrupted: false }); + return config.heartbeatSchedulerEnabled + ? registerToolReviewProjectionWork(deliveryWork, { sweep: app.locals.toolGateway.repairActionReviewProjections, companies: app.locals.toolGateway.actionReviewProjectionCompanies }, + () => !isIdleTaskDrainActive() && !heartbeatSchedulerStopped && !isWarmStandby()).ready + : app.locals.toolGateway.repairActionReviewProjections(); }).catch((err: unknown) => logger.error({ err }, "startup tool review recovery failed")); await timeStartupRecovery("gateway_sessions", () => config.heartbeatSchedulerEnabled ? registerGatewayCredentialExpiryWork(deliveryWork, app.locals.toolGateway, @@ -1917,8 +1921,6 @@ async function startServerWithDatabaseTeardown( logger.error({ err }, "periodic tool connection health sweep failed"); })); - trackHeartbeatSchedulerWork(app.locals.toolGateway.sweepActionReviews({ includeExpiry: false, includeApproved: false, includeInterrupted: false }).catch((err: unknown) => logger.error({ err }, "tool review recovery failed"))); - if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; if (!(await heartbeat.resolveSchedulingSuppression()).suppressed) { // Periodically reap orphaned runs (5-min staleness threshold) and make sure diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 9c943677dac..8af5f57babe 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + toolReviewProjection: "tool-review-projection", toolReviewInterruption: "tool-review-interruption", toolReviewRecovery: "tool-review-recovery", toolReviewExpiry: "tool-review-expiry", diff --git a/server/src/services/issue-thread-interactions.test.ts b/server/src/services/issue-thread-interactions.test.ts index e2c9ac62044..24e1489006f 100644 --- a/server/src/services/issue-thread-interactions.test.ts +++ b/server/src/services/issue-thread-interactions.test.ts @@ -14,7 +14,7 @@ type SelectRow = Record; function createSelectChain(rows: SelectRow[] | ((table: unknown) => SelectRow[])) { return { from(table: unknown) { - const selectedRows = typeof rows === "function" ? rows(table) : rows; + let selectedRows = typeof rows === "function" ? rows(table) : rows; const query = { innerJoin() { return query; @@ -25,6 +25,10 @@ function createSelectChain(rows: SelectRow[] | ((table: unknown) => SelectRow[]) for() { return query; }, + limit(count: number) { + selectedRows = selectedRows.slice(0, count); + return query; + }, then(callback: (rows: SelectRow[]) => unknown) { return Promise.resolve(callback(selectedRows)); }, @@ -292,7 +296,7 @@ describe("issueThreadInteractionService", () => { ]); }); - it("withdraws a pending interaction with attribution and rejects repeats", async () => { + it("withdraws an unlinked pending interaction with attribution and rejects repeats", async () => { const { issueThreadInteractionService } = await import("./issue-thread-interactions.js"); const interactionRow = { id: "interaction-withdraw", companyId: "company-1", issueId: "11111111-1111-4111-8111-111111111111", @@ -308,8 +312,7 @@ describe("issueThreadInteractionService", () => { expect(withdrawn.status).toBe("cancelled"); expect(withdrawn.result).toEqual({ version: 1, outcome: "withdrawn", reason: "Replanning" }); expect(withdrawn.resolvedByAgentId).toBe("agent-1"); - expect(state.toolActionRequestUpdates).toHaveLength(1); - expect(state.toolActionRequestUpdates[0]).toMatchObject({ status: "cancelled", resolvedByAgentId: "agent-1" }); + expect(state.toolActionRequestUpdates).toHaveLength(0); const resolvedState = createFakeDb({ interactionRow: { ...interactionRow, status: "accepted" } }); const resolvedSvc = issueThreadInteractionService(resolvedState.db as never); await expect(resolvedSvc.withdrawInteraction( @@ -388,7 +391,9 @@ describe("issueThreadInteractionService", () => { }, result: null, resolvedAt: null, createdAt: new Date("2026-07-25T10:00:00.000Z"), updatedAt: new Date("2026-07-25T10:00:00.000Z"), }; - const state = createFakeDb({ interactionRow }); + const state = createFakeDb({ interactionRow, parentRows: [{ + id: "33333333-3333-4333-8333-333333333333", companyId: "company-1", interactionId: interactionRow.id, + }] }); const svc = issueThreadInteractionService(state.db as never); const expired = await svc.expirePendingInteractionsForTerminalIssue( { id: interactionRow.issueId, companyId: "company-1", status: "cancelled" }, diff --git a/server/src/services/issue-thread-interactions.ts b/server/src/services/issue-thread-interactions.ts index 48ecf1d9dbe..44eb4391055 100644 --- a/server/src/services/issue-thread-interactions.ts +++ b/server/src/services/issue-thread-interactions.ts @@ -1073,8 +1073,22 @@ class InteractionResolvedConcurrentlyError extends Error { // tool_action_requests row. Administrative resolutions (withdraw, terminal-issue // expiry) must settle that row too, or the parked call stays approvable under // its own one-hour lifecycle after its card is gone. +async function hasLinkedToolActionRequests(db: Pick, companyId: string, interactionIds: string[]) { + if (interactionIds.length === 0) return false; + const [request] = await db.select({ id: toolActionRequests.id }).from(toolActionRequests) + .where(and(eq(toolActionRequests.companyId, companyId), inArray(toolActionRequests.interactionId, interactionIds))) + .limit(1); + return request !== undefined; +} + +async function notifyToolProjectionForInteractions(db: Pick, companyId: string, interactionIds: string[]) { + // The persisted link also covers legacy cards without current payload flags. + if (await hasLinkedToolActionRequests(db, companyId, interactionIds)) + await notifyDeliveryWork(db, DELIVERY_QUEUES.toolReviewProjection, companyId); +} + async function resolveLinkedToolActionRequests( - db: Pick, + db: Pick, interaction: Pick, outcome: { status: "expired" | "cancelled"; @@ -1083,7 +1097,8 @@ async function resolveLinkedToolActionRequests( now: Date; }, ) { - if (interaction.kind !== "request_confirmation") return; + if (interaction.kind !== "request_confirmation" || + !(await hasLinkedToolActionRequests(db, interaction.companyId, [interaction.id]))) return; await notifyToolReviewChange(db, interaction.companyId); await db .update(toolActionRequests) @@ -2240,6 +2255,7 @@ export function issueThreadInteractionService( }) : undefined; + if (lockedCurrent.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, lockedCurrent.companyId, [lockedCurrent.id]); const [updated] = await tx .update(issueThreadInteractions) .set({ @@ -2517,6 +2533,7 @@ export function issueThreadInteractionService( now, }); + if (lockedCurrent.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, lockedCurrent.companyId, [lockedCurrent.id]); const [resolved] = await tx .update(issueThreadInteractions) .set({ @@ -3273,6 +3290,7 @@ export function issueThreadInteractionService( const cancelled: IssueThreadInteraction[] = []; for (const row of rows) { const updated = await db.transaction(async (tx) => { + if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]); const [resolved] = await tx .update(issueThreadInteractions) .set({ @@ -3357,6 +3375,7 @@ export function issueThreadInteractionService( const expired: IssueThreadInteraction[] = []; for (const { row, replacementInteractionId } of supersededRows) { const updated = await db.transaction(async (tx) => { + if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]); const [updatedRow] = await tx .update(issueThreadInteractions) .set({ @@ -3776,6 +3795,19 @@ export function issueThreadInteractionService( data.kind === "ask_user_questions" ? buildSupersededByNewerInteractionResult(row.id) : buildSupersededByNewerRequestResult(row.id); + const siblingCondition = and( + eq(issueThreadInteractions.companyId, issue.companyId), + eq(issueThreadInteractions.issueId, issue.id), + eq(issueThreadInteractions.kind, data.kind), + eq(issueThreadInteractions.createdByAgentId, actor.agentId), + eq(issueThreadInteractions.status, "pending"), + ne(issueThreadInteractions.id, row.id), + ); + if (data.kind === "request_confirmation") { + // The issue lock above excludes concurrent interaction creation. + const siblings = await tx.select({ id: issueThreadInteractions.id }).from(issueThreadInteractions).where(siblingCondition); + await notifyToolProjectionForInteractions(tx, issue.companyId, siblings.map(sibling => sibling.id)); + } const supersededRows = await tx .update(issueThreadInteractions) .set({ @@ -3786,17 +3818,7 @@ export function issueThreadInteractionService( resolvedAt: now, updatedAt: now, }) - .where( - and( - eq(issueThreadInteractions.companyId, issue.companyId), - eq(issueThreadInteractions.issueId, issue.id), - eq(issueThreadInteractions.kind, data.kind), - eq(issueThreadInteractions.createdByAgentId, actor.agentId), - eq(issueThreadInteractions.status, "pending"), - ne(issueThreadInteractions.id, row.id), - - ), - ) + .where(siblingCondition) .returning(); for (const supersededRow of supersededRows) { await resolveLinkedToolActionRequests(tx, supersededRow, { @@ -4442,6 +4464,7 @@ export function issueThreadInteractionService( .delete(toolOauthStates) .where(eq(toolOauthStates.interactionId, row.id)); } + if (row.kind === "request_confirmation") await notifyToolProjectionForInteractions(tx, row.companyId, [row.id]); const [resolved] = await tx .update(issueThreadInteractions) .set({ @@ -4610,6 +4633,7 @@ export function issueThreadInteractionService( ); if (!sampleConfirmationRow) continue; const updatedInteractions = await db.transaction(async (tx) => { + await notifyToolProjectionForInteractions(tx, issue.companyId, confirmationRowIds); const updatedRows = await tx .update(issueThreadInteractions) .set({ diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 2d94a361de2..4cacb30e47e 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1,3 +1,4 @@ +import { toolReviewProjectionService } from "./tool-review-projection.js"; import { toolReviewInterruptionService } from "./tool-review-interruption.js"; import { bindVerifiedToolApprovalArguments, claimToolTestApproval, releaseToolApproval, persistToolTestOutcome, startToolInvocation, type ToolApprovalClaim } from "./tool-approval-execution.js"; import { approvedToolReviewCompanies, prepareApprovedToolReview, recoverApprovedToolReviews } from "./tool-review-recovery.js"; @@ -2398,6 +2399,11 @@ export function createToolGatewayService( ? "stale_target" : null; if (!outcome) return; + const projection = asRecord(currentResult?.toolAction); + if (projection?.status === (input.status === "cancelled" ? "expired" : input.status) + && projection.errorCode === (input.errorCode ?? null) + && projection.errorMessage === (input.errorMessage ?? null) + && projection.resultSummary === (input.resultSummary ?? null)) return false; const now = new Date(); await tx @@ -8125,10 +8131,10 @@ export function createToolGatewayService( return { projectId: issue.projectId }; } const expiredAt = new Date(); - await db + await writeToolReviewChange(db, claimed.companyId, tx => tx .update(toolActionRequests) .set({ status: "expired", resolvedAt: expiredAt, updatedAt: expiredAt }) - .where(eq(toolActionRequests.id, claimed.id)); + .where(eq(toolActionRequests.id, claimed.id))); await reflectToolActionInteractionLifecycle({ actionRequestId: claimed.id, status: "expired", @@ -8820,6 +8826,7 @@ export function createToolGatewayService( }; } + const reviewProjection = toolReviewProjectionService(db, reflectToolActionInteractionLifecycle); const reviewInterruption = toolReviewInterruptionService(db, async settled => { void emitConnectionInvoked(db, settled.invocationId); await reflectToolActionInteractionLifecycle(settled); @@ -9917,6 +9924,8 @@ export function createToolGatewayService( return buildTestCallStatus(actionRequest, invocation); }, + repairActionReviewProjections: reviewProjection.sweep, + actionReviewProjectionCompanies: reviewProjection.companies, settleInterruptedActionReviews: reviewInterruption.sweep, nextActionReviewInterruptionAt: reviewInterruption.next, sweepExpiredActionReviews: reviewExpiry.sweep, @@ -9951,40 +9960,7 @@ export function createToolGatewayService( }, async sweepActionReviews(input: { includeExpiry?: boolean; includeApproved?: boolean; includeInterrupted?: boolean } = {}) { - // A process may stop between persisting a provider outcome and updating the - // feed projection. Reconcile from authoritative rows before delivering it. - const unreflected = await db - .select({ request: toolActionRequests, invocation: toolInvocations }) - .from(toolActionRequests) - .innerJoin( - issueThreadInteractions, - eq(issueThreadInteractions.id, toolActionRequests.interactionId), - ) - .innerJoin( - toolInvocations, - eq(toolInvocations.id, toolActionRequests.invocationId), - ) - .where( - and( - inArray(toolActionRequests.status, [ - "executed", - "failed", - "expired", - "cancelled", - ]), - sql`coalesce(${issueThreadInteractions.result}->'toolAction'->>'status', '') <> case when ${toolActionRequests.status} = 'cancelled' then 'expired' else ${toolActionRequests.status} end`, - ), - ) - .limit(100); - for (const { request, invocation } of unreflected) - await reflectToolActionInteractionLifecycle({ - actionRequestId: request.id, - status: request.status as - "executed" | "failed" | "expired" | "cancelled", - errorCode: invocation.errorCode, - errorMessage: invocation.errorMessage, - resultSummary: invocation.resultSummary?.summary, - }); + await reviewProjection.sweep(); let scanned = input.includeExpiry === false ? 0 : (await reviewExpiry.sweep()).scanned; if (input.includeApproved !== false) scanned += (await this.recoverApprovedActionReviews()).scanned; if (input.includeInterrupted !== false) scanned += (await reviewInterruption.sweep()).scanned; diff --git a/server/src/services/tool-review-projection-work.ts b/server/src/services/tool-review-projection-work.ts new file mode 100644 index 00000000000..b400f19f615 --- /dev/null +++ b/server/src/services/tool-review-projection-work.ts @@ -0,0 +1,33 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerScopedDeadlineWork } from "./scoped-deadline-work.js"; +import type { toolReviewProjectionService } from "./tool-review-projection.js"; + +export function registerToolReviewProjectionWork( + coordinator: ReturnType, + service: ReturnType, + canRun: () => boolean, +) { + const unresolvedPasses = new Map(); + return registerScopedDeadlineWork(coordinator, { + queue: DELIVERY_QUEUES.toolReviewProjection, + batchSize: 1, + decodeScope: hint => hint === "*" ? undefined : hint, + matches: (entry, companyId?: string) => companyId === undefined || entry.companyId === companyId, + load: async (companyId?: string) => { + const rows = companyId ? [{ companyId }] : await service.companies(); + for (const scope of unresolvedPasses.keys()) { + if (!companyId || scope === companyId) unresolvedPasses.delete(scope); + } + return rows.map(row => ({ ...row, id: row.companyId, dueAt: Date.now() })); + }, + check: async (entry, signal) => { + const result = await service.sweep(entry.companyId, { maxRows: 500, signal, shouldContinue: canRun }); + if (!result.retry) { unresolvedPasses.delete(entry.id); return null; } + if (result.truncated) return { ...entry, dueAt: Date.now() + 1000 }; + const count = Math.min((unresolvedPasses.get(entry.id) ?? 0) + 1, 5); + unresolvedPasses.set(entry.id, count); + return { ...entry, dueAt: Date.now() + Math.min(300_000, 30_000 * 2 ** (count - 1)) }; + }, + }, canRun); +} diff --git a/server/src/services/tool-review-projection.ts b/server/src/services/tool-review-projection.ts new file mode 100644 index 00000000000..32644929cac --- /dev/null +++ b/server/src/services/tool-review-projection.ts @@ -0,0 +1,77 @@ +import { and, asc, eq, gt, inArray, sql } from "drizzle-orm"; +import { issueThreadInteractions, toolActionRequests, toolInvocations, type Db } from "@paperclipai/db"; + +export function toolReviewProjectionService(db: Db, reflect: (input: { + actionRequestId: string; + status: "executed" | "failed" | "expired" | "cancelled"; + errorCode: string | null; + errorMessage: string | null; + resultSummary: string | null; +}) => Promise) { + const mismatch = and( + inArray(toolActionRequests.status, ["executed", "failed", "expired", "cancelled"]), + sql`coalesce(${issueThreadInteractions.result}->'toolAction'->>'status', '') <> case when ${toolActionRequests.status} = 'cancelled' then 'expired' else ${toolActionRequests.status} end`, + ); + const candidates = (companyId?: string, cursor?: string) => db + .select({ request: toolActionRequests, invocation: toolInvocations }) + .from(toolActionRequests) + .innerJoin(issueThreadInteractions, eq(issueThreadInteractions.id, toolActionRequests.interactionId)) + .innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and(mismatch, companyId ? eq(toolActionRequests.companyId, companyId) : undefined, + cursor ? gt(toolActionRequests.id, cursor) : undefined)) + .orderBy(asc(toolActionRequests.id)); + + const continuations = new Map(); + return { + async companies(companyId?: string) { + const rows = await db.select({ companyId: toolActionRequests.companyId }).from(toolActionRequests) + .innerJoin(issueThreadInteractions, eq(issueThreadInteractions.id, toolActionRequests.interactionId)) + .innerJoin(toolInvocations, eq(toolInvocations.id, toolActionRequests.invocationId)) + .where(and(mismatch, companyId ? eq(toolActionRequests.companyId, companyId) : undefined)) + .groupBy(toolActionRequests.companyId); + const pending = new Set(rows.map(row => row.companyId)); + for (const scope of continuations.keys()) { + if (companyId && scope !== companyId) continue; + if (scope === "*" ? rows.length === 0 : !pending.has(scope)) continuations.delete(scope); + } + return rows; + }, + async sweep(companyId?: string, options: { + maxRows?: number; + signal?: AbortSignal; + shouldContinue?: () => boolean; + } = {}) { + const maxRows = options.maxRows ?? Infinity; + const scope = companyId ?? "*"; + let cursor = maxRows === Infinity ? undefined : continuations.get(scope); + const resumed = cursor !== undefined; + let scanned = 0, completed = false; + const canContinue = () => !options.signal?.aborted && (options.shouldContinue?.() ?? true); + const failures: unknown[] = []; + while (scanned < maxRows && canContinue()) { + const pageSize = Math.min(100, maxRows - scanned); + const rows = await candidates(companyId, cursor).limit(pageSize); + for (const { request, invocation } of rows) { + if (!canContinue()) break; + try { + await reflect({ actionRequestId: request.id, + status: request.status as "executed" | "failed" | "expired" | "cancelled", + errorCode: invocation.errorCode, errorMessage: invocation.errorMessage, + resultSummary: invocation.resultSummary?.summary ?? null }); + } catch (error) { failures.push(error); } + scanned++; + cursor = request.id; + } + if (!canContinue()) break; + if (rows.length < pageSize) { completed = true; break; } + } + if (maxRows !== Infinity) { + if (completed) continuations.delete(scope); + else if (cursor) continuations.set(scope, cursor); + } + if (failures.length) throw new AggregateError(failures, "Tool review projection repair failed"); + return { scanned, truncated: !completed, + retry: !completed || ((scanned > 0 || resumed) && (await candidates(companyId).limit(1)).length > 0) }; + }, + }; +} diff --git a/server/src/services/tool-review-work-notifications.ts b/server/src/services/tool-review-work-notifications.ts index 44957fefc1b..1ed9dfe7520 100644 --- a/server/src/services/tool-review-work-notifications.ts +++ b/server/src/services/tool-review-work-notifications.ts @@ -5,6 +5,7 @@ export async function notifyToolReviewChange(transaction: object, companyId: str await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewExpiry, companyId); await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewRecovery, companyId); await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewInterruption, companyId); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.toolReviewProjection, companyId); } export function writeToolReviewChange(db: Db, companyId: string, write: (transaction: Db) => Promise): Promise { From 0a08d37aa0dfd5c60d403c2215a220f9b0a70221 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:10:32 -0500 Subject: [PATCH 20/41] feat(runner): recover controller authority without interrupting execution (#17097) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages agents and owns their business authority. > - Runner owns execution transport, acknowledgements and replay. > - A replacement controller must recover that transport without starting another provider turn. > - An external store can retain the required control records without copying streamed text. > - A duplicate local journal adds synchronous disk work without providing the replacement server's authority. > - This PR adds the optional recovery contract and its standalone proof. Server adoption and browser delivery have separate owners. ## Linked Issues or Issue Description **What existing behavior does this improve?** A surviving runner can reconnect to a replacement controller with its admitted process, session and active turn. **Current behavior** Local controller files do not provide recovery on empty replacement storage. Rewriting those files after an external checkpoint is installed also blocks normal controller work. **Proposed behavior** An optional callback stores opaque checkpoints. The replacement restores that checkpoint before it constructs the controller. Installing the callback makes the external store authoritative. Local storage holds a marker that requires restoration. Consumers without the callback retain their file journal. PR 1 of four: #17097 → #17439 → #17455 → #17456. Related work: #17390 and #17361. The pre-split reference remains `codex/hot-upgrades-reference-2026-10-24`. ## What Changed - Add opaque checkpoint encoding, restoration, incremental persistence and authenticated sparse recovery cursors. - Keep authentication, commands, semantic inputs and lifecycle events behind their commit barriers. Share semantic-call ownership between live dispatch and recovery. - Keep streamed text out of recovery checkpoints. Preserve bounded retention, original deadlines and admitted process identity. - Stop normal local journal writes after external persistence is installed. Fail closed on a local marker until external restoration occurs. Preserve current master's staged file writes for standalone consumers. - Keep provider-native admission and authorization behind adapters. Preserve historical Copilot attestations and bind the merged launch and transport sources in v68; preserve published v66 and prior development identities. - Add the one-process scripted executor fixture and deterministic outage, replay, failure and backpressure tests. Add no production test endpoint or auxiliary process. ## Verification The completion-recovery correction passes three restart regressions that fail before its respective transport/driver fixes. All 230 controller/attestation cases, all 302 Codex transport cases in the full repeat, and all three one-process fixture cases pass. The broader Codex/Dot/Muse run passes 326 cases and times out one existing mid-goal case; its isolated retry and complete Codex repeat pass on unchanged source and binaries. The initial timeout remains recorded. Conformance and replay parity pass. Historical v65/v66 attestation bytes remain frozen; v67 binds the changed transport. The local runner-only API-authority lane passes 2,027 cases but its 47 HTTP cases cannot initialize embedded Postgres on this Mac. The integrated owned-database HTTP path is verified in PR 2. At preceding head `0d8d1b8250`, the Actions workflow passes after one retry of runner-service shutdowns. Greptile is 5/5 with no unresolved threads. The interrupted attempt is retained. All checks on that preceding head are terminal and green/neutral/skipped. Run `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/control-plane/durable-prp-control-plane.test.ts src/live/runnerd-codex-transport.test.ts --maxWorkers=1` and `node packages/paperclip-runner/scripts/copilot-profile-identity.mjs --check`. The [qualification record](https://github.com/paperclipai/paperclip/blob/codex/hot-upgrades-qualification/doc/plans/2026-10-22-hot-upgrade-qualification.md) retains exact revisions, failed attempts and qualification limits. The local cost campaign observes zero hot-path local journal fsync calls. It does not qualify a native harness or hosted deployment. Master reconciliation `d5fc443626` preserves the published v66 and prior development identity bytes and binds the combined transport in v68. All 290 focused runner/profile cases and runner typecheck pass. The first sandboxed invocation cannot open loopback listeners; the permitted repeat passes. Fresh CI and Greptile are pending for this updated head. Current merge candidate `4fb3d1835b` also fixes public-reader publication before external commit. New held/rejected-write cases fail before correction and pass afterward. The full controller repeat passes 168 cases. The combined lane passes 390 other recovery/transport/profile/fixture cases; its first two overlap fixture failures are retained and corrected to observe callback entry instead of provisional state. Runner TypeScript build and typecheck pass. Current-head GitHub Actions is green and Greptile is 5/5 with all threads resolved. The superseded master-reconciliation CI shutdown remains recorded. ## Risks The persistence callback is an acknowledgement barrier. Failure must stop delivery. A consumer that installs external persistence must restore its committed checkpoint before restarting; the marker is not a recovery snapshot. Sparse display recovery does not reconstruct acknowledged text or repeat execution. This slice has no Postgres schema, application authorization or browser behavior. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- packages/paperclip-runner/acpx-profiles.json | 4 +- .../paperclip-runner/docs/durable-recovery.md | 144 ++++ .../runner-core/examples/recovery-fixture.rs | 233 +++++++ .../runner-core/src/durable/transport.rs | 26 + .../src/generated_acpx_profiles.rs | 2 +- .../scripts/copilot-profile-identity.mjs | 4 +- .../durable-prp-control-plane.test.ts | 652 +++++++++++++++++- .../durable-prp-control-plane.ts | 547 +++++++++++++-- .../control-plane/runner-execution-driver.ts | 10 +- .../runner-recovery-fixture.test.ts | 142 ++++ .../drivers/acpx/copilot-installation.test.ts | 59 +- .../src/drivers/acpx/generated-profiles.ts | 4 +- .../src/drivers/acpx/profile-compatibility.ts | 4 +- .../drivers/dot/runnerd-dot-driver.test.ts | 43 +- .../external/runnerd-external-driver.ts | 68 +- packages/paperclip-runner/src/index.ts | 2 + .../src/live/runner-provider-launch.ts | 2 +- ...runnerd-acpx-permission-forwarding.test.ts | 5 + .../runnerd-acpx-permission-ordering.test.ts | 5 + .../src/live/runnerd-codex-transport.test.ts | 270 +++++++- .../src/live/runnerd-codex-transport.ts | 85 ++- ...upgrade-candidate-historical-identity.json | 85 +++ ...upgrade-candidate-historical-identity.json | 85 +++ .../copilot-profile-v67-identity.json | 85 +++ .../copilot-profile-v68-identity.json | 85 +++ 25 files changed, 2490 insertions(+), 161 deletions(-) create mode 100644 packages/paperclip-runner/runner/crates/runner-core/examples/recovery-fixture.rs create mode 100644 packages/paperclip-runner/src/control-plane/runner-recovery-fixture.test.ts create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v59-hot-upgrade-candidate-historical-identity.json create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v66-hot-upgrade-candidate-historical-identity.json create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v67-identity.json create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v68-identity.json diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index eda0ebf68c0..0e5443132cc 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 66, + "agentProfileVersion": 68, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8", + "commandDigest": "sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/docs/durable-recovery.md b/packages/paperclip-runner/docs/durable-recovery.md index 3bfcfe49db3..43898274399 100644 --- a/packages/paperclip-runner/docs/durable-recovery.md +++ b/packages/paperclip-runner/docs/durable-recovery.md @@ -57,6 +57,110 @@ PRP v2 command vocabulary. Incoming wire commands still undergo the negotiated protocol validation before execution; ordinary persisted v1 summaries remain compatible. +## Adoption and retired launch artifacts + +A live runner is adopted with its admitted artifact binding and authenticated +process identity. ACPX and OpenCode launch profiles are constructed only when a +new runner will actually be spawned. Adoption does not read the retired +controller's provider-pack executables, sidecar or proxy files. Fresh launches +still require the existing artifact digests and provider-pack authority. + +The component regression starts the production runner and OpenCode proxy against +a scripted harness, removes the controller's launch files, then recovers the +same process and provider session and completes another turn. This exercises the +real transport; it does not qualify a hosted OpenCode or ACPX deployment. + +## Source-bound profile history + +Copilot profile v68 binds the merged hot-upgrade and current master transport. +Published v66 and the earlier hot-upgrade v67 identity remain byte-for-byte +historical records. The distinct development v66 identity is retained under +`copilot-profile-v66-hot-upgrade-candidate-historical-identity.json`; it does not +replace the published v66 fixture. Decoding historical versions does not grant +fresh-launch permission. Installation still requires the exact current profile, +package and digest. These records do not establish hosted harness qualification. + +## External controller checkpoints + +The optional controller persistence callback stores recovery control state, not +an output transcript. Text deltas do not trigger callbacks; usage updates are +coalesced into the next control checkpoint. Authentication, command intents and +outcomes, semantic inputs, lifecycle and result events retain commit barriers. +Priority-zero and unknown event kinds are never treated as disposable output. + +Installing the callback makes it the controller's durable authority. The local +state file is replaced once with a fail-closed marker; ordinary control changes +no longer rewrite or fsync a second local journal. The controller can continue +if that directory disappears. A new controller must restore the committed +external checkpoint before opening the directory; the marker cannot recover +an execution or authorize a new one. This does not change the sandbox runner's +journal or controllers without an external persistence callback. + +Fresh semantic inputs reserve their original live callback before awaiting the +checkpoint barrier. Recovery cannot steal first dispatch when no saved outcome +exists. The reservation is released if persistence or admission fails; business +execution still starts only after durable persistence succeeds. + +Recovery and live semantic-tool dispatch share per-call ownership. Overlapping +reconciliation calls cannot invoke the same callback twice while its outcome is +pending. An already queued authoritative result is replayed without resolving +its business outcome again. Failed or unknown outcomes retain the existing +reconciliation and authority rules. + +Pending completion inputs (`paperclip_finish` and `paperclip_block`) use the +driver's existing completion handler even when its caller supplies no recovery +lookup for ordinary tools. Codex transport waits for both authenticated startup +and handler installation; the external driver waits for session opening. +An ordinary tool with no outcome lookup remains held. Reconciliation does not +turn a missing lookup into permission to repeat that tool. + +Transport readers use a committed view when external persistence is enabled. +Lifecycle and result events, command outcomes and warm-handoff authority become +visible only after the checkpoint callback succeeds. A failed write leaves the +previous view in place until a successful retry. Internal admission still sees +staged intents so overlapping recovery cannot duplicate them. Command results +flush their checkpoint even when no next command is waiting. + +Streamed output continues without checkpoint writes. The view stops before a +pending control event to preserve event order, reuses immutable event records +and unchanged decoded commands, and caches the visible projection between +changes. It neither copies a second transcript nor adds an idle scan. + +A rejected persistence callback closes the current connection and keeps the last +successful revision. A later flush or reconnect retries from a complete checkpoint, +including when the failed response hid a successful commit. Queued writes also +rebuild their delta against the last successful callback. Commit barriers remain +in force, and no retry timer runs when the controller is idle. + +Supporting runners advertise `recoveryCursorVersion: 1` in `auth_hello`. The core +includes that capability and `recoveryAckedSourceSeq` in both authentication +proofs. The runner verifies the cursor equals its persisted acknowledgement. +After authentication and owner admission, a replacement core can reconcile that +cursor with its sparse checkpoint. It only skips already-acknowledged history; +new delivery remains contiguous. Unacknowledged events use the existing outbox. +A peer without this capability cannot attach to a sparse external checkpoint. +New runners can still authenticate to older controllers that use full journals. + +There is no per-output external checkpoint timer or remote-controller cache +fsync. Local-only controllers retain their full file journal. Existing application +callbacks, the bounded runner outbox, and backpressure remain. Losing the +controller before acknowledged display output reaches its application log can +leave a display tail gap; this must not trigger provider or tool reexecution. + +Prepared environments retain at most two opaque configuration IDs, each bound to +its exact execution identity across a warm handoff. They are captured with control +state, never output, and do not duplicate instructions or preparation requirements. +An execution driver can provide `authorizeRunnerAdoption(configurationId, identity)` +to recheck current authority before registering a surviving runner, without +preparing its resources again. The driver owns policy; Runner knows no project or +account rules. Missing bindings and rejected checks fail closed. Without this +optional hook, adoption retains the existing `prepareLaunch` recovery callback +and requires its controller-local preparation cache. Older checkpoints without +a configuration binding cannot use the cache-independent adoption path. +Supplying the hook for adoption without `prepareLaunch` is rejected at construction; +the transport must not silently skip a requested authority check. Drivers that do +not use prepared environments keep their existing adoption contract. + ## Connection and authentication The connection starts in this order: @@ -424,11 +528,49 @@ A `revoke` envelope persists the `revoked` lifecycle. The runner flushes any existing durable events and exits. It does not accept new work or delete unacknowledged facts. +## Process identity + +An external process owner can publish `startedAt` after launch dispatch returns. +`spawnRunner` preserves that live value through its restart wrapper. Once the +owner has a verified process identity, consumers use its birth time rather than +controller construction or attachment time. Replacing the controller must not +change the admitted process identity. + ## Verification The production durable control plane and live-session suites cover reconnect, ACK replay, restart recovery, backpressure, lease expiry, drain, and revoke. +`src/control-plane/runner-recovery-fixture.test.ts` builds the test-only Cargo +example `recovery-fixture`. It links the unchanged production durable runner with +a scripted `CommandExecutor`, whose worker thread performs bounded hashing steps +and retains each resulting output until the normal executor ACK transfers it to +the runner outbox. There is one process, no HTTP counter service, and no production +provider-admission switch. Private fixture receipts report completed steps, queue +depth and process identity; they do not acknowledge or recover protocol events. + +The short experiment starts one turn, removes the controller, then releases real +work. Eight retained events fill the fixture queue and pause work explicitly. +A replacement authenticates and drains through PRP. At step 32 the executor waits +for a controller command, survives another replacement, then completes 64 steps +without repeating the turn. The test runs with ordinary filesystem persistence +and with opaque external checkpoints restored onto empty controller storage. +An independent short-deadline case checks termination with recovery state intact. +These prove the runner mechanism, not Codex, Claude, Daytona or application +authorization. Server adoption and installed-artifact qualification have separate +owners and evidence. + +Run the short experiment from this package with: + +```sh +pnpm exec vitest run src/control-plane/runner-recovery-fixture.test.ts --maxWorkers=1 +``` + +The fixture builds through Cargo during test setup. Production distribution builds +do not include the example. Existing control-plane tests separately inject failed +checkpoint persistence, stale/unauthenticated cursors and output floods, including +the zero-per-text-checkpoint-write gate. + Use `--json` for the complete trace or `--output ` to write it. The CLI and browser show connection counts, safe lease ID and expiry, stable identities, source and ACK cursors, outbox current/peak bytes, command delivery/effect @@ -441,3 +583,5 @@ Regenerate the checked fault matrix and every exact per-fault trace with: ```sh # Recorded evidence generation is deferred from this release. ``` + +Checkpoint decoding also exposes read-only recovery evidence: the admitted runner/session identity, authenticated connection count, and highest recorded source sequence. This projection grants no authority and adds no checkpoint writes. Qualification compares it across controller replacement; a replayed canonical event by itself does not prove that a replacement controller has authenticated a runner. diff --git a/packages/paperclip-runner/runner/crates/runner-core/examples/recovery-fixture.rs b/packages/paperclip-runner/runner/crates/runner-core/examples/recovery-fixture.rs new file mode 100644 index 00000000000..0398a6b2f3f --- /dev/null +++ b/packages/paperclip-runner/runner/crates/runner-core/examples/recovery-fixture.rs @@ -0,0 +1,233 @@ +use std::collections::VecDeque; +use std::fs; +use std::path::PathBuf; +use std::sync::{Arc, Condvar, Mutex}; +use std::thread; +use std::time::Duration; + +use paperclip_runner_core::durable::{ + capture_bootstrap_ticket, run_durable_runner, Command, CommandExecution, CommandExecutor, + DurableRunnerConfig, DurableRunnerError, EventPriority, PolledEvent, +}; +use serde::Deserialize; +use serde_json::json; +use sha2::{Digest, Sha256}; + +#[derive(Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +struct Script { + steps: usize, + wait_after: usize, + retention_limit: usize, + release_file: PathBuf, + receipt_file: PathBuf, +} + +#[derive(Default)] +struct Work { + started: bool, + stopped: bool, + resumed: bool, + completed: usize, + acknowledged: usize, + peak_retained: usize, + events: VecDeque, +} + +type Shared = Arc<(Mutex, Condvar)>; + +fn receipt(script: &Script, work: &Work, phase: &str) { + let bytes = serde_json::to_vec(&json!({ + "pid": std::process::id(), "phase": phase, "completedSteps": work.completed, + "retainedEvents": work.events.len(), "acknowledgedEvents": work.acknowledged, + "peakRetainedEvents": work.peak_retained, + })) + .unwrap(); + let temporary = script.receipt_file.with_extension("pending"); + fs::write(&temporary, bytes).unwrap(); + fs::rename(temporary, &script.receipt_file).unwrap(); +} + +fn workload(script: Script, shared: Shared) { + let (lock, changed) = &*shared; + let mut work = lock.lock().unwrap(); + receipt(&script, &work, "admitted"); + loop { + if work.stopped { + receipt(&script, &work, "stopped"); + return; + } + if !work.started || !script.release_file.exists() { + work = changed + .wait_timeout(work, Duration::from_millis(10)) + .unwrap() + .0; + continue; + } + if work.completed == script.steps { + receipt(&script, &work, "completed"); + work = changed.wait(work).unwrap(); + continue; + } + if work.completed == script.wait_after && !work.resumed { + receipt(&script, &work, "awaiting_controller"); + work = changed.wait(work).unwrap(); + continue; + } + if work.events.len() == script.retention_limit { + receipt(&script, &work, "backpressured"); + work = changed.wait(work).unwrap(); + continue; + } + let step = work.completed + 1; + drop(work); + let mut digest = Sha256::digest(step.to_le_bytes()); + for _ in 0..4096 { + digest = Sha256::digest(digest); + } + work = lock.lock().unwrap(); + work.events.push_back(PolledEvent { + executor_event_id: format!("fixture-step-{step}"), + event_type: "item.delta".to_owned(), + priority: EventPriority::P2, + payload: json!({"text": format!("step={step} sha256={digest:x}\n")}), + }); + work.completed = step; + work.peak_retained = work.peak_retained.max(work.events.len()); + receipt(&script, &work, "computing"); + } +} + +struct ScriptedExecutor { + shared: Shared, + worker: Option>, +} + +impl ScriptedExecutor { + fn new(script: Script) -> Self { + let shared = Arc::new((Mutex::new(Work::default()), Condvar::new())); + let worker_shared = shared.clone(); + Self { + shared, + worker: Some(thread::spawn(move || workload(script, worker_shared))), + } + } +} + +impl CommandExecutor for ScriptedExecutor { + fn execute(&mut self, command: &Command) -> Result { + let mut work = self.shared.0.lock().unwrap(); + match command.command_type.as_str() { + "turn.start" => { + if work.started { + return Err(DurableRunnerError::invalid("fixture turn started twice")); + } + work.started = true; + } + "turn.steer" => work.resumed = true, + "session.snapshot" => {} + "turn.stop" | "session.close" | "runner.shutdown" | "runner.suspend" => { + work.stopped = true + } + _ => return Err(DurableRunnerError::invalid("unsupported fixture command")), + } + self.shared.1.notify_all(); + Ok(CommandExecution::result(json!({ + "completedSteps": work.completed, "retainedEvents": work.events.len(), + "acknowledgedEvents": work.acknowledged, "pid": std::process::id(), + }))) + } + + fn poll_events(&mut self) -> Result, DurableRunnerError> { + self.retained_events() + } + + fn retained_events(&mut self) -> Result, DurableRunnerError> { + Ok(self + .shared + .0 + .lock() + .unwrap() + .events + .iter() + .cloned() + .collect()) + } + + fn acknowledge_events(&mut self, count: usize) -> Result<(), DurableRunnerError> { + let mut work = self.shared.0.lock().unwrap(); + if count > work.events.len() { + return Err(DurableRunnerError::invalid( + "fixture acknowledged an unowned event", + )); + } + work.events.drain(..count); + work.acknowledged += count; + self.shared.1.notify_all(); + Ok(()) + } + + fn shutdown(&mut self) -> Result<(), DurableRunnerError> { + self.shared.0.lock().unwrap().stopped = true; + self.shared.1.notify_all(); + if let Some(worker) = self.worker.take() { + worker + .join() + .map_err(|_| DurableRunnerError::invalid("fixture worker failed"))?; + } + Ok(()) + } +} + +impl Drop for ScriptedExecutor { + fn drop(&mut self) { + let _ = self.shutdown(); + } +} + +fn main() -> Result<(), Box> { + let args: Vec = std::env::args().skip(1).collect(); + let value = |key: &str| -> Result { + args.windows(2) + .find(|pair| pair[0] == key) + .map(|pair| pair[1].clone()) + .ok_or_else(|| DurableRunnerError::invalid(format!("missing {key}"))) + }; + let script: Script = serde_json::from_slice(&fs::read(value("--fixture-script")?)?)?; + if script.steps == 0 + || script.steps > 4096 + || script.wait_after > script.steps + || !(1..=256).contains(&script.retention_limit) + { + return Err("fixture workload exceeds its bounds".into()); + } + let config = DurableRunnerConfig { + connect_url: value("--connect-url")?, + ca_bundle_path: None, + state_dir: value("--state-dir")?.into(), + runner_instance_id: value("--runner-id")?, + environment_lease_id: value("--environment-lease-id")?, + run_id: value("--run-id")?, + normalized_session_id: value("--session-id")?, + turn_id: value("--turn-id")?, + item_id: value("--item-id")?, + runner_version: value("--runner-version")?, + runner_digest: value("--runner-digest")?, + acpx_launch_profile: None, + opencode_launch_profile: None, + max_outbox_bytes: value("--max-outbox-bytes")?.parse()?, + p0_reserve_bytes: value("--p0-reserve-bytes")?.parse()?, + max_frame_bytes: 1024 * 1024, + reconnect_delay: Duration::from_millis(20), + reconnect_grace: Some(Duration::from_millis( + value("--reconnect-grace-ms")?.parse()?, + )), + max_runtime: Duration::ZERO, + }; + run_durable_runner( + config, + capture_bootstrap_ticket()?.ok_or("missing ticket")?, + ScriptedExecutor::new(script), + )?; + Ok(()) +} diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs b/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs index ae2be51ddfb..3631e7fc815 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/durable/transport.rs @@ -1014,6 +1014,7 @@ impl AuthenticatedTransport { "protocolMin": PROTOCOL_MIN_VERSION, "protocolMax": PROTOCOL_VERSION, "warmTransitionVersion": 1, + "recoveryCursorVersion": 1, "runnerInstanceId": state.runner_instance_id, "environmentLeaseId": state.environment_lease_id, "runId": state.run_id, @@ -1200,6 +1201,10 @@ struct AuthChallenge { warm_transition_version: Option, #[serde(default)] warm_transition_id: Option, + #[serde(default)] + recovery_cursor_version: Option, + #[serde(default)] + recovery_acked_source_seq: Option, } fn validate_challenge( @@ -1275,6 +1280,15 @@ fn validate_challenge( "warm transition capability or receipt was not authenticated", )); } + if (challenge.recovery_cursor_version.is_some() + || challenge.recovery_acked_source_seq.is_some()) + && (challenge.recovery_cursor_version != Some(1) + || challenge.recovery_acked_source_seq != Some(state.acked_source_seq)) + { + return Err(DurableRunnerError::invalid( + "recovery cursor does not match the runner's durable acknowledgement", + )); + } match expected_lease { Some(lease) if challenge.credential_lease_id.as_deref() == Some(lease.lease_id.as_str()) @@ -1319,6 +1333,12 @@ fn challenge_signing_bytes(challenge: &AuthChallenge) -> Vec { if let Some(id) = &challenge.warm_transition_id { body["warmTransitionId"] = json!(id); } + if let Some(version) = challenge.recovery_cursor_version { + body["recoveryCursorVersion"] = json!(version); + } + if let Some(cursor) = challenge.recovery_acked_source_seq { + body["recoveryAckedSourceSeq"] = json!(cursor); + } canonical_json(&body).into_bytes() } @@ -1806,6 +1826,8 @@ mod tests { server_proof: String::new(), warm_transition_version: None, warm_transition_id: None, + recovery_cursor_version: Some(1), + recovery_acked_source_seq: payload["resume"]["ackedSourceSeq"].as_u64(), }; let signing = challenge_signing_bytes(&challenge); challenge.server_proof = hex_encode(&hmac_domain( @@ -1838,6 +1860,8 @@ mod tests { "credentialLeaseId": &challenge.credential_lease_id, "revocationEpoch": challenge.revocation_epoch, "serverProof": &challenge.server_proof, + "recoveryCursorVersion": challenge.recovery_cursor_version, + "recoveryAckedSourceSeq": challenge.recovery_acked_source_seq, }, }), config.max_frame_bytes, @@ -2505,6 +2529,8 @@ mod tests { server_proof: String::new(), warm_transition_version: None, warm_transition_id: None, + recovery_cursor_version: None, + recovery_acked_source_seq: None, }; let signing = challenge_signing_bytes(&challenge); challenge.server_proof = hex_encode(&hmac_domain( diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index 4bae7fe1489..5e9a354c50c 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8", + "sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index bc511760487..c7d9b1143b7 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v66-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v68-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 66 }; +const declaration = { ...historical.declaration, agentProfileVersion: 68 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts index b2a88f8d86b..f3f8473ddb3 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts @@ -6,6 +6,7 @@ import { } from "node:crypto"; import { chmodSync, + existsSync, mkdtempSync, readFileSync, rmSync, @@ -29,6 +30,8 @@ import { validatePrpEvent } from "../protocol/replay-contract.js"; import { digestPaperclipSemanticContent } from "../semantic-tools/receipts.js"; import { DurablePrpControlPlane, + decodeDurablePrpCheckpoint, + type DurablePrpCheckpoint, durableRecoveryInternals, SemanticToolNotDispatchedError, inspectWarmRunTransition, @@ -49,6 +52,605 @@ const identity: DurableRecoveryIdentity = { const expectedRunnerVersion = "0.3.0"; const expectedRunnerDigest = `sha256:${"a".repeat(64)}`; +it("continues externally persisted commands after losing the local controller directory", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-external-store-")); + const options = { stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }; + const core = new DurablePrpControlPlane(options); + let checkpoint!: DurablePrpCheckpoint; + core.setCheckpointPersistence(async saved => { checkpoint = saved; }); + let restored: DurablePrpControlPlane | undefined; + try { + await core.start(); + const client = (await authenticate(core, core.issueBootstrapTicket()))!; + rmSync(options.stateDirectory, { recursive: true }); + core.queueCommand("session.snapshot", {}, "external-only-command", true); + expect(await receiveSecure(client)).toMatchObject({ kind: "command", payload: { commandId: "external-only-command" } }); + await core.flushCheckpoint(); + expect(existsSync(options.stateDirectory)).toBe(false); + client.socket.destroy(); + await core.stop(); + checkpoint.restoreSessionDirectory(root); + restored = new DurablePrpControlPlane(options); + expect(restored.getCommand("external-only-command")).toMatchObject({ type: "session.snapshot" }); + } finally { + await restored?.stop(); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("requires the external checkpoint instead of accepting a stale local controller file", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-external-store-restart-")); + const options = { stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }; + const core = new DurablePrpControlPlane(options); + core.setCheckpointPersistence(async () => {}); + try { + core.queueCommand("session.snapshot", {}, "external-command"); + await core.flushCheckpoint(); + await core.stop(); + expect(() => new DurablePrpControlPlane(options)).toThrow(/external checkpoint/i); + } finally { + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("restores authentication and commands from opaque checkpoint records", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-codec-")); + const options = { stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }; + const first = new DurablePrpControlPlane(options); + let checkpoint!: DurablePrpCheckpoint; + first.setCheckpointPersistence(async saved => { checkpoint = saved; }); + let restored: DurablePrpControlPlane | undefined; + try { + const ticket = first.issueBootstrapTicket(); + first.queueCommand("session.snapshot", {}, "maintenance"); + first.queueCommand("turn.start", {}, "execution"); + first.persistPreparedLaunchConfiguration(identity, "retained-configuration"); + await first.flushCheckpoint(); + expect(checkpoint.workCommandIds).toEqual(["execution"]); + await first.stop(); + rmSync(options.stateDirectory, { recursive: true }); + decodeDurablePrpCheckpoint(new Map(checkpoint.records)).restoreSessionDirectory(root); + restored = new DurablePrpControlPlane(options); + expect(restored.preparedLaunchConfigurationId(identity)).toBe("retained-configuration"); + expect(restored.preparedLaunchConfigurationId({ ...identity, runId: "unrelated" })).toBeUndefined(); + expect(() => restored!.persistPreparedLaunchConfiguration(identity, "substituted-configuration")).toThrow("configuration_changed"); + expect(restored.getCommand("execution")).toMatchObject({ type: "turn.start" }); + await restored.start(); + const client = (await authenticate(restored, ticket))!; + expect(client.welcome.kind).toBe("welcome"); + client.socket.destroy(); + } finally { + await restored?.stop(); + await first.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it.each(["missing record", "invalid state", "wrong sequence", "invalid preparation identity", "duplicate preparation", "malformed header", "malformed cursor"])("rejects %s at checkpoint decoding", async fault => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-invalid-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + let checkpoint!: DurablePrpCheckpoint; + core.setCheckpointPersistence(async saved => { checkpoint = saved; }); + try { + core.queueCommand("session.snapshot", {}, "saved-command"); + await core.flushCheckpoint(); + const records = new Map(checkpoint.records); + if (fault === "missing record") records.delete("command:1"); + if (fault === "malformed header") records.set("header", "{"); + if (fault === "malformed cursor") records.set("cursor", "{"); + if (fault === "invalid state") { + const cursor = JSON.parse(records.get("cursor")!); + cursor.ackedSourceSeq = -1; + records.set("cursor", JSON.stringify(cursor)); + } + if (fault === "wrong sequence") { + const command = JSON.parse(records.get("command:1")!); + command.controllerSeq = 2; + records.set("command:1", JSON.stringify(command)); + } + if (fault === "invalid preparation identity" || fault === "duplicate preparation") { + const header = JSON.parse(records.get("header")!); + const binding = { identity, configurationId: "retained-configuration" }; + header.state.preparedLaunchConfigurations = fault === "duplicate preparation" ? [binding, binding] + : [{ ...binding, identity: { runId: identity.runId } }]; + records.set("header", JSON.stringify(header)); + } + expect(() => decodeDurablePrpCheckpoint(records)).toThrow(/native_runner_checkpoint_/); + } finally { + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("restores the previous checkpoint format and upgrades it on the next capture", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-v1-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + let checkpoint!: DurablePrpCheckpoint; + core.setCheckpointPersistence(async saved => { checkpoint = saved; }); + try { + core.queueCommand("session.snapshot", {}, "old-command"); + await core.flushCheckpoint(); + const records = new Map(checkpoint.records); + const header = JSON.parse(records.get("header")!); + const cursor = JSON.parse(records.get("cursor")!); + records.delete("cursor"); + records.set("header", JSON.stringify({ ...header, schema: "paperclip.runner.controller-checkpoint.v1", + state: { ...header.state, ackedSourceSeq: cursor.ackedSourceSeq, replayDeliveries: cursor.replayDeliveries }, + commands: ["command:1"], events: [] })); + const upgraded = decodeDurablePrpCheckpoint(records); + expect(upgraded.records.get("command:1")).toContain("old-command"); + expect(JSON.parse(upgraded.records.get("header")!).schema).toBe("paperclip.runner.controller-checkpoint.v2"); + expect(upgraded.reset).toBe(true); + } finally { + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +function ordinaryOutput(sourceSeq: number): Record { + const envelope = semanticInputEvent(sourceSeq); + Object.assign(envelope.payload as Record, { + sourceEventId: `output-${sourceSeq}`, eventType: "item.delta", priority: 2, + payload: { text: "streaming output" }, + }); + return envelope; +} + +it("streams output without checkpoint writes and recovers its authenticated cursor on empty disk", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-control-checkpoint-")); + const options = { stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest }; + const core = new DurablePrpControlPlane(options); + const snapshots: DurablePrpCheckpoint[] = []; + core.setCheckpointPersistence(async saved => { snapshots.push(saved); }); + let restored: DurablePrpControlPlane | undefined; + try { + await core.start(); + const client = (await authenticate(core, core.issueBootstrapTicket()))!; + const before = snapshots.length; + const localState = readFileSync(core.store.path, "utf8"); + for (let seq = 1; seq <= 64; seq++) { + sendSecure(client, ordinaryOutput(seq)); + expect(await receiveSecure(client)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: seq } }); + } + await core.flushCheckpoint(); + expect(snapshots).toHaveLength(before); + expect(readFileSync(core.store.path, "utf8")).toBe(localState); + const saved = snapshots.at(-1)!; + expect(saved.recoveryEvidence).toEqual({ runnerInstanceId: identity.runnerInstanceId, + normalizedSessionId: identity.normalizedSessionId, authenticatedConnectionCount: 1, highestRecordedSourceSeq: 0 }); + expect(decodeDurablePrpCheckpoint(saved.records).recoveryEvidence).toEqual(saved.recoveryEvidence); + expect(JSON.parse(saved.records.get("cursor")!).ackedSourceSeq).toBe(0); + expect([...saved.records.keys()].filter(key => key.startsWith("event:"))).toEqual([]); + const token = client.leaseToken!; + client.socket.destroy(); + await core.stop(); + await core.drainPendingConnectionProcessing(); + rmSync(options.stateDirectory, { recursive: true }); + decodeDurablePrpCheckpoint(saved.records).restoreSessionDirectory(root); + restored = new DurablePrpControlPlane(options); + restored.setCheckpointPersistence(async saved => { snapshots.push(saved); }); + await restored.start(); + const replacement = (await authenticate(restored, token, identity, expectedRunnerDigest, undefined, false, 1, 64))!; + expect(replacement.welcome.payload).toMatchObject({ ackedSourceSeq: 64 }); + expect(restored.recoveredSourceSeq).toBe(64); + expect(snapshots.at(-1)!.recoveryEvidence).toMatchObject({ authenticatedConnectionCount: 2, highestRecordedSourceSeq: 64 }); + sendSecure(replacement, ordinaryOutput(65)); + expect(await receiveSecure(replacement)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 65 } }); + // Commands remain an immediate durable barrier, without copying text. + const count = snapshots.length; + restored.queueCommand("session.snapshot", {}, "after-reconnect"); + await restored.flushCheckpoint(); + expect(snapshots).toHaveLength(count + 1); + expect([...snapshots.at(-1)!.records.values()].join("\n")).not.toContain("streaming output"); + replacement.socket.destroy(); + } finally { + await restored?.stop(); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("excludes streamed output from checkpoints even when the live window is full", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-window-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + core.store.state.committedEvents = Array.from({ length: 4096 }, (_, index) => ({ + sourceSeq: index + 1, sourceEventId: `output-${index + 1}`, eventType: "item.delta", priority: 2 as const, + envelope: ordinaryOutput(index + 1), deliveryCount: 1, logicalEffectCount: 1 as const, + })); + const anchor = core.store.state.committedEvents[0]!; + anchor.eventType = "turn.started"; + (anchor.envelope.payload as Record).eventType = "turn.started"; + core.store.state.ackedSourceSeq = 4096; + core.store.save(); + const snapshots: DurablePrpCheckpoint[] = []; + core.setCheckpointPersistence(async saved => { snapshots.push(saved); }); + try { + await core.start(); + const client = (await authenticate(core, core.issueBootstrapTicket()))!; + const before = snapshots.length; + sendSecure(client, ordinaryOutput(4097)); + expect(await receiveSecure(client)).toMatchObject({ kind: "ack", payload: { ackedSourceSeq: 4097 } }); + expect(snapshots).toHaveLength(before); + const checkpoint = snapshots.at(-1)!; + expect(core.store.state.committedEvents[0]!.eventType).toBe("turn.started"); + expect(checkpoint.records.size).toBe(3); // authority + cursor + turn anchor, zero output rows. + expect(decodeDurablePrpCheckpoint(checkpoint.records).records).toEqual(checkpoint.records); + client.socket.destroy(); + } finally { + await core.stop(); + await core.drainPendingConnectionProcessing(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it.each(["unsupported", "invalid cursor", "invalid proof"])("does not advance recovery from an unauthenticated cursor: %s", async fault => { + const root = mkdtempSync(resolve(tmpdir(), "runner-recovery-cursor-auth-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + core.setCheckpointPersistence(async () => {}); + try { + await core.start(); + const material = credentialMaterial(core.issueBootstrapTicket()); + const { socket, reader } = await upgradeSocket(core.connectUrl); + const hello = authHello(material.credentialId); + const payload = hello.payload as Record; + payload.resume = { ackedSourceSeq: 50, nextSourceEventSeq: 51 }; + if (fault === "unsupported") delete payload.recoveryCursorVersion; + if (fault === "invalid cursor") payload.resume = { ackedSourceSeq: 50, nextSourceEventSeq: 50 }; + sendMaskedJson(socket, hello); + if (fault === "invalid proof") { + const challenge = (await reader.next())!.payload as Record; + expect(challenge.recoveryAckedSourceSeq).toBe(50); + expect(core.store.state.ackedSourceSeq).toBe(0); + sendMaskedJson(socket, { protocol: "paperclip.runner", version: 1, kind: "auth_response", payload: { + credentialId: material.credentialId, clientNonce: challenge.clientNonce, + serverNonce: challenge.serverNonce, clientProof: "0".repeat(64), + } }); + } + expect(await reader.next()).toBeNull(); + expect(core.store.state.ackedSourceSeq).toBe(0); + expect(core.recoveredSourceSeq).toBe(0); + socket.destroy(); + } finally { + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("persists authentication and commands before the runner can observe them", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-external-journal-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + let release!: () => void; + let pending: Promise = Promise.resolve(); + const snapshots: DurablePrpCheckpoint[] = []; + core.setCheckpointPersistence(async snapshot => { await pending; snapshots.push(snapshot); }); + try { + await core.start(); + const client = (await authenticate(core, core.issueBootstrapTicket()))!; + expect(snapshots.at(-1)).toMatchObject({ runnerVersion: expectedRunnerVersion, runId: identity.runId }); + expect(snapshots.at(-1)!.records.get("header")).toContain(client.welcome.connectionLeaseId); + pending = new Promise(resolveCommit => { release = resolveCommit; }); + core.queueCommand("session.snapshot", {}, "checkpoint-command", true); + let received = false; + const command = receiveSecure(client).then(value => { received = true; return value; }); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(received).toBe(false); + release(); + expect(await command).toMatchObject({ kind: "command", payload: { commandId: "checkpoint-command" } }); + expect([...snapshots.at(-1)!.records.values()].join("\n")).toContain("checkpoint-command"); + client.socket.destroy(); + } finally { + release?.(); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("does not dispatch a semantic effect when external journal persistence fails", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-external-journal-failure-")); + const dispatch = vi.fn(async () => ({ result: "written" })); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, + expectedRunnerDigest, onSemanticToolInput: dispatch }); + core.setCheckpointPersistence(async snapshot => { + if (JSON.parse(snapshot.records.get("cursor")!).ackedSourceSeq > 0) throw new Error("database unavailable"); + }); + try { + await core.start(); + const client = (await authenticate(core, core.issueBootstrapTicket()))!; + sendSecure(client, semanticInputEvent()); + await vi.waitFor(() => expect(core.activeRunnerConnectionCount()).toBe(0)); + expect(dispatch).not.toHaveBeenCalled(); + await expect(core.flushCheckpoint()).rejects.toThrow("database unavailable"); + client.socket.destroy(); + } finally { + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it.each([false, true])("recovers checkpoint persistence and the same connection lease after a failed write (committed=%s)", async committed => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-retry-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + const stored = new Map(); + const attempts: DurablePrpCheckpoint[] = []; + let fail = false; + core.setCheckpointPersistence(async snapshot => { + attempts.push(snapshot); + if (!fail || committed) { + if (snapshot.reset) stored.clear(); + for (const key of snapshot.removedRecords) stored.delete(key); + for (const [key, value] of snapshot.changedRecords) stored.set(key, value); + } + if (fail) throw new Error("checkpoint response lost"); + }); + try { + await core.start(); + const client = (await authenticate(core, core.issueBootstrapTicket()))!; + fail = true; + core.queueCommand("session.snapshot", {}, "retry-command", true); + await expect(core.flushCheckpoint()).rejects.toThrow("checkpoint response lost"); + expect(await receiveSecure(client)).toBeNull(); + fail = false; + await core.flushCheckpoint(); + expect(attempts.at(-1)!.reset).toBe(true); + expect(stored).toEqual(attempts.at(-1)!.records); + const settledAttempts = attempts.length; + await core.flushCheckpoint(); + expect(attempts).toHaveLength(settledAttempts); + const resumed = (await authenticate(core, client.leaseToken!))!; + expect(resumed).not.toBeNull(); + expect(resumed.welcome.payload).toMatchObject({ pendingCommands: [{ commandId: "retry-command" }] }); + expect(attempts.at(-1)!.reset).toBe(false); + expect(stored).toEqual(attempts.at(-1)!.records); + resumed.socket.destroy(); + client.socket.destroy(); + } finally { + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("rebuilds a queued checkpoint after the preceding write fails", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-checkpoint-queued-retry-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + let reject!: (error: Error) => void; + let pending: Promise = Promise.resolve(); + const snapshots: DurablePrpCheckpoint[] = []; + core.setCheckpointPersistence(async snapshot => { await pending; snapshots.push(snapshot); }); + try { + await core.flushCheckpoint(); + pending = new Promise((_resolve, rejectWrite) => { reject = rejectWrite; }); + core.queueCommand("session.snapshot", {}, "first-command"); + const first = core.flushCheckpoint(); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + core.queueCommand("session.snapshot", {}, "second-command"); + const second = core.flushCheckpoint(); + const failed = expect(first).rejects.toThrow("first write failed"); + reject(new Error("first write failed")); + pending = Promise.resolve(); + await failed; + await second; + expect(snapshots).toHaveLength(2); + expect(snapshots.at(-1)).toMatchObject({ reset: true }); + expect([...snapshots.at(-1)!.changedRecords.values()].join("\n")).toContain("first-command"); + expect([...snapshots.at(-1)!.changedRecords.values()].join("\n")).toContain("second-command"); + await core.flushCheckpoint(); + expect(snapshots).toHaveLength(2); + } finally { + reject?.(new Error("cleanup")); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it.each([true, false])("reconciles a saved semantic input only from an authoritative outcome (known=%s)", async known => { + const root = mkdtempSync(resolve(tmpdir(), "runner-recovered-semantic-")); + const dispatch = vi.fn(() => new Promise<{ result: unknown }>(() => undefined)); + const options = { stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }; + const first = new DurablePrpControlPlane({ ...options, onSemanticToolInput: dispatch }); + let recovered: DurablePrpControlPlane | undefined; + try { + await first.start(); + const client = (await authenticate(first, first.issueBootstrapTicket()))!; + sendSecure(client, semanticInputEvent()); + expect((await receiveSecure(client))!.kind).toBe("ack"); + expect(dispatch).toHaveBeenCalledTimes(1); + await first.stop(); + const recovery = vi.fn(async () => known ? { result: { savedReceipt: "exact-effect" } } : null); + const redispatch = vi.fn(async () => ({ result: "must not execute again" })); + recovered = new DurablePrpControlPlane({ ...options, onSemanticToolInput: redispatch, + recoverSemanticToolInput: recovery }); + await recovered.start(); + const next = (await authenticate(recovered, client.leaseToken!))!; + await recovered.recoverPendingSemanticInputs(); + expect(recovery).toHaveBeenCalledTimes(1); + expect(redispatch).not.toHaveBeenCalled(); + if (known) { + expect(await receiveSecure(next)).toMatchObject({ kind: "command", payload: { + type: "semantic_tool.result", payload: { result: { savedReceipt: "exact-effect" } }, + } }); + } else { + expect(recovered.store.state.commands).toHaveLength(0); + expect(recovered.semanticToolResultsSettled()).toBe(false); + } + next.socket.destroy(); + } finally { + await recovered?.stop(); + await first.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it("resolves a saved semantic input only once across overlapping recovery calls", async () => { + const root = mkdtempSync(resolve(tmpdir(), "runner-saved-semantic-race-")); + const options = { stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }; + const original = new DurablePrpControlPlane({ ...options, + onSemanticToolInput: () => new Promise<{ result: unknown }>(() => undefined) }); + let resolveOutcome!: (value: { result: unknown }) => void; + const outcome = new Promise<{ result: unknown }>(resolveResult => { resolveOutcome = resolveResult; }); + const recover = vi.fn(async () => outcome); + let replacement: DurablePrpControlPlane | undefined; + const pending: Promise[] = []; + try { + await original.start(); + const client = (await authenticate(original, original.issueBootstrapTicket()))!; + sendSecure(client, semanticInputEvent()); + expect((await receiveSecure(client))!.kind).toBe("ack"); + client.socket.destroy(); + await original.stop(); + replacement = new DurablePrpControlPlane({ ...options, recoverSemanticToolInput: recover }); + pending.push(replacement.recoverPendingSemanticInputs()); + await vi.waitFor(() => expect(recover).toHaveBeenCalledOnce()); + pending.push(replacement.recoverPendingSemanticInputs()); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(recover).toHaveBeenCalledOnce(); + resolveOutcome({ result: { receipt: "one-authoritative-outcome" } }); + await Promise.all(pending); + await replacement.recoverPendingSemanticInputs(); + expect(recover).toHaveBeenCalledOnce(); + expect(replacement.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(1); + } finally { + resolveOutcome({ result: { receipt: "one-authoritative-outcome" } }); + await Promise.allSettled(pending); + await replacement?.stop(); + await original.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); + +it.each([false, true])("dispatches a fresh semantic input once when recovery overlaps its checkpoint (saved outcome: %s)", async known => { + const root = mkdtempSync(resolve(tmpdir(), "runner-fresh-semantic-race-")); + let releaseWrite!: () => void; + const writeBarrier = new Promise(resolveWrite => { releaseWrite = resolveWrite; }); + let releaseRecovery!: () => void; + const recoveryBarrier = new Promise(resolveRecovery => { releaseRecovery = resolveRecovery; }); + let releaseDispatch!: () => void; + const dispatchBarrier = new Promise(resolveDispatch => { releaseDispatch = resolveDispatch; }); + const recover = vi.fn(async () => { await recoveryBarrier; return known ? { result: "stale recovery outcome" } : null; }); + const dispatch = vi.fn(async () => { await dispatchBarrier; return { result: "original live outcome" }; }); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, + expectedRunnerDigest, onSemanticToolInput: dispatch, recoverSemanticToolInput: recover }); + let writeHeld = false; + core.setCheckpointPersistence(async saved => { + if (JSON.parse(saved.records.get("cursor")!).ackedSourceSeq > 0) { writeHeld = true; await writeBarrier; } + }); + const pending: Promise[] = []; + let client: Awaited>; + try { + await core.start(); + client = (await authenticate(core, core.issueBootstrapTicket()))!; + sendSecure(client, semanticInputEvent()); + await vi.waitFor(() => expect(writeHeld).toBe(true)); + expect(core.store.state.committedEvents).toHaveLength(0); + pending.push(core.recoverPendingSemanticInputs()); + await new Promise(resolveTurn => setImmediate(resolveTurn)); + expect(dispatch).not.toHaveBeenCalled(); // The durability barrier still applies. + releaseWrite(); + expect((await receiveSecure(client))!.kind).toBe("ack"); + releaseRecovery(); + await Promise.all(pending); + expect(dispatch).toHaveBeenCalledOnce(); + expect(recover).not.toHaveBeenCalled(); + await core.recoverPendingSemanticInputs(); + expect(recover).not.toHaveBeenCalled(); + releaseDispatch(); + expect(await receiveSecure(client)).toMatchObject({ kind: "command", payload: { + type: "semantic_tool.result", payload: { result: "original live outcome" }, + } }); + expect(core.store.state.commands.filter(command => command.type === "semantic_tool.result")).toHaveLength(1); + } finally { + releaseWrite(); releaseRecovery(); releaseDispatch(); + await Promise.allSettled(pending); + client?.socket.destroy(); + await core.stop(); + rmSync(root, { recursive: true, force: true }); + } +}); +it.each(["turn.completed", "run.result.proposed", "tool.execution.progressed", "command_result"] as const)( + "publishes %s to transport readers only after its external checkpoint commits", async mode => { + for (const rejectWrite of [false, true]) { + const root = mkdtempSync(resolve(tmpdir(), "runner-published-checkpoint-")); + const core = new DurablePrpControlPlane({ stateDirectory: root, identity, expectedRunnerVersion, expectedRunnerDigest }); + let release!: () => void; + const gate = new Promise(resolveGate => { release = resolveGate; }); + let entered = false; + let hold = false; + let lastSaved!: DurablePrpCheckpoint; + core.setCheckpointPersistence(async checkpoint => { + if (hold) { + entered = true; + await gate; + if (rejectWrite) throw new Error("controlled checkpoint rejection"); + } + lastSaved = checkpoint; + }); + let client: AuthenticatedClient | null = null; + let pump: ReturnType | undefined; + let polls = 0; + const observed = vi.fn(); + try { + await core.start(); + client = (await authenticate(core, core.issueBootstrapTicket()))!; + core.queueCommand("session.snapshot", {}, "publication-command", true); + expect(await receiveSecure(client)).toMatchObject({ kind: "command" }); + await core.flushCheckpoint(); + hold = true; + pump = setInterval(() => { + polls++; + const visible = mode === "command_result" + ? core.getCommand("publication-command")?.status === "completed" + || core.store.state.commands.some(command => command.commandId === "publication-command" && command.status === "completed") + : core.store.state.committedEvents.some(event => event.eventType === mode); + if (visible && observed.mock.calls.length === 0) observed(); + }, 5); + if (mode === "command_result") { + sendSecure(client, { protocol: "paperclip.runner", version: 1, kind: "command_result", + payload: { commandId: "publication-command", commandType: "session.snapshot", controllerSeq: 1, + status: "completed", result: { snapshot: true } } }); + } else { + const event = semanticInputEvent(); + const payload = event.payload as Record; + payload.eventType = mode; + payload.priority = mode === "tool.execution.progressed" ? 1 : 0; + payload.payload = mode === "run.result.proposed" ? { + schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "complete", + completionClaim: { contractRevision: "test-v1", objectiveSatisfied: true, criteria: [], remainingWork: [] }, + evidence: [], verification: [], attentionRequests: [], artifacts: [], + } : mode === "tool.execution.progressed" ? { + schema: "paperclip.tool.execution.v1", executionId: "publication-progress", transport: "process", operation: "execute", + status: "running", output: "working", outputBytes: 7, outputTruncated: false, outputDigest: null, + } : { status: "completed" }; + expect(validatePrpEvent(payload)).toMatchObject({ ok: true }); + sendSecure(client, event); + } + await vi.waitFor(() => expect(entered).toBe(true)); + await vi.waitFor(() => expect(polls).toBeGreaterThan(2)); + expect(observed).not.toHaveBeenCalled(); + release(); + if (rejectWrite) { + await expect(core.flushCheckpoint()).rejects.toThrow("controlled checkpoint rejection"); + expect(observed).not.toHaveBeenCalled(); + hold = false; + await core.flushCheckpoint(); + } else await core.flushCheckpoint(); + await vi.waitFor(() => expect(observed).toHaveBeenCalledOnce()); + if (mode === "command_result") { + expect(JSON.parse(lastSaved.records.get("command:1")!).status).toBe("completed"); + } else expect(lastSaved.records.has("event:1")).toBe(true); + } finally { + hold = false; + release(); + if (pump) clearInterval(pump); + client?.socket.destroy(); + await core.stop(); + await core.drainPendingConnectionProcessing(); + rmSync(root, { recursive: true, force: true }); + } + } + }, +); + it.each(["event", "progress", "lifecycle", "lifecycle_priority_one", "usage", "credentials", "newer_command", "command_burst", "command_failure", "authority_conflict", "replacement", "write_failure", "usage_write_failure", "drain", "usage_drain", "semantic_replacement"] as const)( "keeps HTTP responsive during a durable event write and preserves %s", async (mode) => { if (mode === "command_failure" && process.platform === "win32") return; @@ -606,6 +1208,26 @@ it.skipIf(process.platform === "win32")( }, ); +it("retains process birth identity published after an external launch returns", async () => { + let startedAt: string | undefined; + const birth = "2026-10-26T12:00:00.000Z"; + const handle = spawnRunner({ + connection: { mode: "connect", connectUrl: "ws://127.0.0.1:43127" }, + stateDirectory: "/tmp/paperclip-runner-deferred-identity", identity, ticket: "bootstrap-ticket", + maxOutboxBytes: 256 * 1024, p0ReserveBytes: 64 * 1024, + runnerVersion: expectedRunnerVersion, runnerDigest: expectedRunnerDigest, + processLauncher: () => ({ + child: { pid: 42, exitCode: null, kill: () => true }, + completion: new Promise(() => {}), + get startedAt() { return startedAt; }, + }), + }); + expect(handle.startedAt).toBeUndefined(); + // The remote process marker arrives after the provider dispatch returns. + await Promise.resolve().then(() => { startedAt = birth; }); + expect(handle.startedAt).toBe(birth); +}); + it("does not release or replace a remote runner until its exit is confirmed", async () => { let exited = false; const launch = vi.fn(() => ({ @@ -1258,6 +1880,8 @@ function authHello( protocolMin: 1, protocolMax: 1, warmTransitionVersion: 1, + recoveryCursorVersion: 1, + resume: { ackedSourceSeq: 0, nextSourceEventSeq: 1 }, ...selectedIdentity, runnerVersion: expectedRunnerVersion, runnerDigest: expectedRunnerDigest, @@ -1273,11 +1897,13 @@ async function authenticate( warmTransitionId?: string, withoutWarmCapability = false, protocolMax = 1, + acknowledgedSourceSeq = 0, ): Promise { const { socket, reader } = await upgradeSocket(controlPlane.connectUrl); const material = credentialMaterial(token); const hello = authHello(material.credentialId, selectedIdentity); (hello.payload as Record).protocolMax = protocolMax; + (hello.payload as Record).resume = { ackedSourceSeq: acknowledgedSourceSeq, nextSourceEventSeq: acknowledgedSourceSeq + 1 }; (hello.payload as Record).runnerDigest = runnerDigest; if (warmTransitionId !== undefined) (hello.payload as Record).warmTransitionId = @@ -3194,12 +3820,15 @@ describe("DurablePrpControlPlane", () => { const root = mkdtempSync( resolve(tmpdir(), "paperclip-prp-attach-transition-"), ); + let externalCheckpoint: DurablePrpCheckpoint | undefined; let core = new DurablePrpControlPlane({ - stateDirectory: root, + stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest, }); + const usesExternalPersistence = mode === "activate" || mode === "activate-v2"; + if (usesExternalPersistence) core.setCheckpointPersistence(async checkpoint => { externalCheckpoint = checkpoint; }); const nextIdentity = { ...identity, runId: "00000000-0000-4000-8000-000000000002", @@ -3214,6 +3843,9 @@ describe("DurablePrpControlPlane", () => { peer = await authenticateVersion(core, core.issueBootstrapTicket()); const oldToken = peer!.leaseToken!; core.persistRunAttachTemplate({ provider: { kind: "mock", run: "prior" } }); + core.persistPreparedLaunchConfiguration(identity, "prior-configuration"); + core.persistPreparedLaunchConfiguration(nextIdentity, "next-configuration"); + if (usesExternalPersistence) rmSync(resolve(root, "control-plane"), { recursive: true }); const command = core.queueCommand( "run.attach", { @@ -3229,7 +3861,7 @@ describe("DurablePrpControlPlane", () => { await expect(receiveSecure(peer!)).resolves.toMatchObject({ kind: "command", }); - const beforeResult = readFileSync(core.store.path, "utf8"); + const beforeResult = usesExternalPersistence ? null : readFileSync(core.store.path, "utf8"); const result = { commandId: command.commandId, commandType: command.type, @@ -3271,9 +3903,10 @@ describe("DurablePrpControlPlane", () => { peer!.socket.destroy(); await core.stop(); if (mode === "bootstrap-before-result") - writeFileSync(core.store.path, beforeResult); + writeFileSync(core.store.path, beforeResult!); + externalCheckpoint?.restoreSessionDirectory(root); core = new DurablePrpControlPlane({ - stateDirectory: root, + stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest, @@ -3438,7 +4071,7 @@ describe("DurablePrpControlPlane", () => { expect(core.getCommand(command.commandId)?.status).toBe("pending"); await core.stop(); core = new DurablePrpControlPlane({ - stateDirectory: root, + stateDirectory: resolve(root, "control-plane"), identity, expectedRunnerVersion, expectedRunnerDigest, @@ -3562,6 +4195,10 @@ describe("DurablePrpControlPlane", () => { ); expect(successor).not.toBeNull(); expect(core.store.state.identity).toEqual(nextIdentity); + expect(core.preparedLaunchConfigurationId(nextIdentity)).toBe("next-configuration"); + core.persistPreparedLaunchConfiguration({ ...nextIdentity, runId: "third-run" }, "third-configuration"); + expect(core.preparedLaunchConfigurationId(identity)).toBeUndefined(); + expect(core.store.state.preparedLaunchConfigurations).toHaveLength(2); expect(core.store.state.runAttachTemplate).toEqual({ provider: { kind: "mock", run: "next" } }); expect( (successor!.welcome.payload as Record) @@ -3598,6 +4235,9 @@ describe("DurablePrpControlPlane", () => { await vi.waitFor(() => expect(core.store.state.warmTransition).toBeUndefined(), ); + if (mode === "activate" || mode === "activate-v2") { + expect(core.store.state.checkpointEventPolicy).toBe("control-v1"); + } expect(core.store.state.schema).toBe( "paperclip.runner.durable.control-plane-state.v1", ); @@ -3665,7 +4305,7 @@ describe("DurablePrpControlPlane", () => { successor!.socket.destroy(); await core.stop(); core = new DurablePrpControlPlane({ - stateDirectory: root, + stateDirectory: resolve(root, "control-plane"), identity: nextIdentity, expectedRunnerVersion, expectedRunnerDigest, diff --git a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts index 4f09db52dd9..7fb21c34431 100644 --- a/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts +++ b/packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts @@ -54,6 +54,7 @@ const protocolVersion = RUNNER_PRP_PROTOCOL_MAX; const secureFrameSchema = "paperclip.runner.secure-frame.v1"; const websocketGuid = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11"; const coreStateSchema = "paperclip.runner.durable.control-plane-state.v1"; +const externalCoreStateSchema = "paperclip.runner.external-control-plane-state.v1"; const transitionCoreStateSchema = "paperclip.runner.durable.control-plane-state.warm-transition.v1"; const maxFrameBytes = 1024 * 1024; @@ -142,6 +143,8 @@ interface ConnectionLeaseRecord { interface StoredCoreState { schema: typeof coreStateSchema | typeof transitionCoreStateSchema; identity: DurableRecoveryIdentity; + /** External checkpoints contain control state, not a duplicate output log. */ + checkpointEventPolicy?: "control-v1"; warmTransition?: { receipt: DurableWarmRunTransition; phase: "awaiting_result" | "prepared" | "activated"; @@ -161,6 +164,8 @@ interface StoredCoreState { * reconstruct another warm attachment. */ runAttachTemplate?: Record | null; + /** Opaque environment configuration bindings; bounded across a warm handoff. */ + preparedLaunchConfigurations?: Array<{ identity: DurableRecoveryIdentity; configurationId: string }>; tickets: Record; leases: Record; commands: DurableRecoveryCoreCommand[]; @@ -226,6 +231,7 @@ interface PendingChallenge { warmTransitionVersion?: 1; warmTransitionId?: string; requestedIdentity?: DurableRecoveryIdentity; + acknowledgedSourceSeq?: number; } interface SecureChannel { @@ -269,6 +275,9 @@ export interface DurablePrpControlPlaneOptions { }) => Promise<{ readonly result: unknown; readonly isError?: boolean }>; /** Persist the canonical event before the runner receives its cumulative ACK. */ onCommittedEvent?: (event: PrpEvent) => Promise; + recoverSemanticToolInput?: ( + input: Parameters>[0], + ) => Promise<{ readonly result: unknown; readonly isError?: boolean } | null>; /** Stop this exact owner after a proven, authenticated permanent integrity fault. */ onProtocolIntegrityError?: ( error: NativeSessionProtocolIntegrityError, @@ -276,6 +285,144 @@ export interface DurablePrpControlPlaneOptions { connectionLeaseTtlMs?: number; } +export interface DurablePrpCheckpoint { + /** Read-only protocol evidence. Does not grant attachment or execution authority. */ + readonly recoveryEvidence: { + readonly runnerInstanceId: string; + readonly normalizedSessionId: string; + readonly authenticatedConnectionCount: number; + readonly highestRecordedSourceSeq: number; + }; + readonly runnerVersion: string; + readonly runnerDigest: string; + readonly runId: string; + readonly predecessor: { readonly runId: string; readonly normalizedSessionId: string } | null; + readonly records: ReadonlyMap; + /** Atomic write set relative to the last successful callback for this run. */ + readonly changedRecords: ReadonlyMap; + readonly removedRecords: readonly string[]; + /** First write for a run, or retry after a failed write, replaces the checkpoint atomically. */ + readonly reset: boolean; + readonly workCommandIds: readonly string[]; + restoreSessionDirectory(root: string): void; +} + +type CheckpointHeader = { + schema: "paperclip.runner.controller-checkpoint.v2"; + runnerVersion: string; + runnerDigest: string; + state: Omit; +}; + +function checkpointRunId(state: StoredCoreState): string { + if (state.warmTransition) return state.warmTransition.receipt.newIdentity.runId; + const attachment = state.commands.findLast(command => command.type === "run.attach"); + const next = attachment?.payload.paperclipNextAuthority as { identity?: { runId?: string } } | undefined; + return next?.identity?.runId ?? state.identity.runId; +} + +function isStreamingEvent(event: Pick): boolean { + return event.priority !== 0 && ["item.delta", "usage.reported"].includes(event.eventType); +} + +function evictionCandidate(state: StoredCoreState): number { + // Keep the latest identity/turn/result anchors through arbitrarily long text + // streams. Unsettled semantic inputs retain their existing stronger fence. + const anchors = new Set(["harness.ready", "session.started", "session.resumed", "turn.started", + "turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted", "run.result.proposed", "usage.reported"]); + const latest = new Map(); + for (const event of state.committedEvents) if (anchors.has(event.eventType)) latest.set(event.eventType, event); + return state.committedEvents.findIndex(event => latest.get(event.eventType) !== event && !unsettledSemanticInput(event, state)); +} + +function checkpointChanges(records: ReadonlyMap, runId: string, previous?: DurablePrpCheckpoint) { + const reset = !previous || previous.runId !== runId; + return { + changedRecords: new Map([...records].filter(([key, value]) => reset || previous.records.get(key) !== value)), + removedRecords: reset ? [] : [...previous.records.keys()].filter(key => !records.has(key)), + reset, + }; +} + +function captureCheckpoint(state: StoredCoreState, runnerVersion: string, runnerDigest: string): DurablePrpCheckpoint { + const runId = checkpointRunId(state); + const records = new Map(); + const { commands, committedEvents, ackedSourceSeq: _cursor, replayDeliveries: _replays, ...metadata } = state; + const latestUsage = committedEvents.findLast(event => event.eventType === "usage.reported"); + const recoveryEvents = committedEvents.filter(event => !isStreamingEvent(event) || event === latestUsage); + for (const command of commands) records.set(`command:${command.controllerSeq}`, JSON.stringify(command)); + for (const event of recoveryEvents) records.set(`event:${event.sourceSeq}`, JSON.stringify(event)); + const header: CheckpointHeader = { schema: "paperclip.runner.controller-checkpoint.v2", + runnerVersion, runnerDigest, state: { ...metadata, checkpointEventPolicy: "control-v1" } }; + records.set("header", JSON.stringify(header)); + records.set("cursor", JSON.stringify({ ackedSourceSeq: state.ackedSourceSeq, replayDeliveries: state.replayDeliveries, + commandCount: commands.length, eventCount: recoveryEvents.length })); + const maintenance = new Set(["runner.drain", "runner.suspend", "session.snapshot", "session.close"]); + return { + runnerVersion, runnerDigest, runId, records, ...checkpointChanges(records, runId), + recoveryEvidence: { runnerInstanceId: state.identity.runnerInstanceId, normalizedSessionId: state.identity.normalizedSessionId, + authenticatedConnectionCount: state.connectionCount, + highestRecordedSourceSeq: state.committedEvents.reduce((highest, event) => Math.max(highest, event.sourceSeq), state.ackedSourceSeq) }, + predecessor: state.identity.runId === runId ? null + : { runId: state.identity.runId, normalizedSessionId: state.identity.normalizedSessionId }, + workCommandIds: commands.filter(command => !maintenance.has(command.type)).map(command => command.commandId), + restoreSessionDirectory(root) { + mkdirSync(root, { recursive: true, mode: 0o700 }); + verifyPrivateDirectory(root); + const directory = resolve(root, "control-plane"); + mkdirSync(directory, { recursive: true, mode: 0o700 }); + verifyPrivateDirectory(directory); + atomicPrivateWrite(resolve(directory, "control-plane-state.json"), JSON.stringify(checkpointState(records).state)); + }, + }; +} + +function checkpointState(records: ReadonlyMap) { + const header: unknown = JSON.parse(records.get("header") ?? "null"); + if (!isRecord(header) || !["paperclip.runner.controller-checkpoint.v1", "paperclip.runner.controller-checkpoint.v2"].includes(String(header.schema)) + || typeof header.runnerVersion !== "string" || !header.runnerVersion + || typeof header.runnerDigest !== "string" || !runnerDigestPattern.test(header.runnerDigest) + || !isRecord(header.state) || !exactIdentity(header.state.identity)) throw new Error("native_runner_checkpoint_invalid"); + const read = (key: string): unknown => { + const value = records.get(key); + if (value === undefined) throw new Error("native_runner_checkpoint_incomplete"); + return JSON.parse(value); + }; + const v2 = header.schema === "paperclip.runner.controller-checkpoint.v2"; + const keys = (prefix: string) => [...records.keys()].filter(key => key.startsWith(prefix)) + .sort((a, b) => Number(a.slice(prefix.length)) - Number(b.slice(prefix.length))); + const commandKeys = v2 ? keys("command:") : header.commands; + const eventKeys = v2 ? keys("event:") : header.events; + if (!Array.isArray(commandKeys) || commandKeys.length > maxCommands + || !Array.isArray(eventKeys) || eventKeys.length > maxCommittedEventWindow + || commandKeys.some(key => typeof key !== "string" || !/^command:\d+$/.test(key)) + || eventKeys.some(key => typeof key !== "string" || !/^event:\d+$/.test(key)) + || new Set(commandKeys).size !== commandKeys.length || new Set(eventKeys).size !== eventKeys.length) { + throw new Error("native_runner_checkpoint_invalid"); + } + const cursor = v2 ? read("cursor") : header.state; + if (!isRecord(cursor) || (v2 && (cursor.commandCount !== commandKeys.length || cursor.eventCount !== eventKeys.length + || records.size !== commandKeys.length + eventKeys.length + 2))) throw new Error("native_runner_checkpoint_incomplete"); + const state = { ...header.state, ackedSourceSeq: cursor.ackedSourceSeq, replayDeliveries: cursor.replayDeliveries, + commands: commandKeys.map(read), committedEvents: eventKeys.map(read) }; + if (!isStoredCoreState(state, header.state.identity)) throw new Error("native_runner_checkpoint_invalid"); + if (state.commands.some((command, index) => commandKeys[index] !== `command:${command.controllerSeq}`) + || state.committedEvents.some((event, index) => eventKeys[index] !== `event:${event.sourceSeq}`)) { + throw new Error("native_runner_checkpoint_invalid"); + } + return { state, runnerVersion: header.runnerVersion, runnerDigest: header.runnerDigest }; +} + +export function decodeDurablePrpCheckpoint(records: ReadonlyMap): DurablePrpCheckpoint { + try { + const decoded = checkpointState(records); + return captureCheckpoint(decoded.state, decoded.runnerVersion, decoded.runnerDigest); + } catch (error) { + if (error instanceof SyntaxError) throw new Error("native_runner_checkpoint_invalid", { cause: error }); + throw error; + } +} + export interface RunnerProcessResult { code: number | null; signal: NodeJS.Signals | null; @@ -645,6 +792,7 @@ function isStoredCoreState( (value.schema !== coreStateSchema && value.schema !== transitionCoreStateSchema) || canonicalJson(value.identity) !== canonicalJson(identity) || + (value.checkpointEventPolicy !== undefined && value.checkpointEventPolicy !== "control-v1") || !isRecord(value.tickets) || !isRecord(value.leases) || !Array.isArray(commands) || @@ -706,6 +854,13 @@ function isStoredCoreState( ) { return false; } + if (value.preparedLaunchConfigurations !== undefined && ( + !Array.isArray(value.preparedLaunchConfigurations) || value.preparedLaunchConfigurations.length > 2 || + !value.preparedLaunchConfigurations.every(binding => isRecord(binding) && exactIdentity(binding.identity) + && typeof binding.configurationId === "string" && binding.configurationId.trim().length > 0 + && binding.configurationId.length <= 1024) || + new Set(value.preparedLaunchConfigurations.map(binding => canonicalJson(binding.identity))).size !== value.preparedLaunchConfigurations.length + )) return false; if ( !commands.every( (command, index) => @@ -1203,7 +1358,9 @@ class DurableCoreStore { readonly path: string; #state: StoredCoreState; #writeIndeterminate = false; - #revision = 0; + #writeRevision = 0; + #externallyPersisted = false; + revision = 0; #pendingEvent: { buildCandidate: (state: StoredCoreState) => StoredCoreState | null; isCurrentOwner: () => boolean; @@ -1228,6 +1385,9 @@ class DurableCoreStore { const stored = readPrivateFile(this.path); if (stored !== null) { const parsed = JSON.parse(stored) as unknown; + if (isRecord(parsed) && parsed.schema === externalCoreStateSchema) { + throw new Error("Restore the external checkpoint before opening this controller."); + } if (!isStoredCoreState(parsed, identity)) { throw new Error( "Control-plane state is invalid or does not match the requested PRP identity.", @@ -1244,9 +1404,29 @@ class DurableCoreStore { return this.#state; } + useExternalPersistence(): void { + this.assertWritable(); + this.#pendingEvent = null; + ++this.#writeRevision; + // Leave a fail-closed marker, never a stale recovery snapshot. The external + // checkpoint must be restored before another controller opens this root. + try { + atomicPrivateWrite(this.path, JSON.stringify({ + schema: externalCoreStateSchema, + identity: this.#state.identity, + })); + } catch (error) { + this.#writeIndeterminate = true; + throw error; + } + this.#state.checkpointEventPolicy = "control-v1"; + this.#externallyPersisted = true; + this.revision += 1; + } + save(): void { this.assertWritable(); - ++this.#revision; + ++this.#writeRevision; try { const pending = this.#pendingEvent; // A command already needs a synchronous write. Include the admitted @@ -1254,36 +1434,48 @@ class DurableCoreStore { const withEvent = pending && !pending.committed && pending.isCurrentOwner() ? pending.buildCandidate(this.#state) : null; const candidate = withEvent ?? this.#state; - atomicPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`); + if (!this.#externallyPersisted) { + atomicPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`); + } this.#state = candidate; if (pending && withEvent) pending.committed = true; + this.revision += 1; } catch (error) { this.#writeIndeterminate = true; throw error; } } - /** Stage event state privately; publish it only after its durable commit. */ + /** File-backed events publish after the staged commit. External delivery + * waits for its checkpoint callback in the controller. */ async commitEvent( buildCandidate: (state: StoredCoreState) => StoredCoreState | null, isCurrentOwner: () => boolean, + checkpointRequired: boolean, ): Promise { if (!isCurrentOwner()) return false; const pending = { buildCandidate, isCurrentOwner, committed: false }; this.#pendingEvent = pending; try { this.assertWritable(); - const revision = this.#revision; + const revision = this.#writeRevision; const candidate = buildCandidate(this.#state); if (candidate === null) return false; + if (this.#externallyPersisted) { + this.#state = candidate; + ++this.#writeRevision; + if (checkpointRequired) this.revision += 1; + return true; + } await withStagedPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`, (temporary) => { this.assertWritable(); - if (pending.committed || !isCurrentOwner() || this.#revision !== revision) return; + if (pending.committed || !isCurrentOwner() || this.#writeRevision !== revision) return; renameSync(temporary, this.path); syncParentDirectory(this.path); this.#state = candidate; - ++this.#revision; + ++this.#writeRevision; pending.committed = true; + if (checkpointRequired) this.revision += 1; }); // Explicit authority commits do not absorb pending events. On conflict, // leave the event unacknowledged for replay instead of rewriting in a loop. @@ -1303,16 +1495,20 @@ class DurableCoreStore { ); } - /** Persist a complete candidate before publishing any new authority in memory. */ - commit(candidate: StoredCoreState): void { + /** Replace authority without allowing an older staged event to overwrite it. */ + commit(candidate: StoredCoreState, checkpointRequired = true): void { this.assertWritable(); // An explicit authority transaction owns its complete snapshot. Later // command writes must not absorb an event admitted before that boundary. this.#pendingEvent = null; - ++this.#revision; + ++this.#writeRevision; + candidate.checkpointEventPolicy ??= this.#state.checkpointEventPolicy; try { - atomicPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`); + if (!this.#externallyPersisted) { + atomicPrivateWrite(this.path, `${JSON.stringify(candidate)}\n`); + } this.#state = candidate; + if (checkpointRequired) this.revision += 1; } catch (error) { // Rename may already have succeeded before directory fsync failed. // Never overwrite that possibly durable receipt using stale memory. @@ -1558,10 +1754,23 @@ export class DurablePrpControlPlane { #port: number | null = null; #onSemanticToolInput?: DurablePrpControlPlaneOptions["onSemanticToolInput"]; #onCommittedEvent?: DurablePrpControlPlaneOptions["onCommittedEvent"]; + #recoverSemanticToolInput?: DurablePrpControlPlaneOptions["recoverSemanticToolInput"]; #beforeAuthenticatedConnection?: DurablePrpControlPlaneOptions["beforeAuthenticatedConnection"]; #onProtocolIntegrityError?: DurablePrpControlPlaneOptions["onProtocolIntegrityError"]; #protocolIntegrityError: NativeSessionProtocolIntegrityError | null = null; #connectionLeaseTtlMs: number; + #persistCheckpoint: ((checkpoint: DurablePrpCheckpoint) => Promise) | null = null; + #persistedRevision = -1; + #queuedRevision = -1; + #lastCheckpoint?: DurablePrpCheckpoint; + #publishedState?: StoredCoreState; + #publishedStore?: DurablePrpControlPlaneStore; + #publicationCommands = new Map(); + #visibleStateCache?: { working: StoredCoreState; published: StoredCoreState; revision: number; state: StoredCoreState }; + #checkpointPersistenceFailed = false; + #persistenceTail = Promise.resolve(); + #sendQueues = new Map>(); + #recoveredCursor: { runId: string; sourceSeq: number } | null = null; constructor(options: DurablePrpControlPlaneOptions) { if ( @@ -1582,6 +1791,9 @@ export class DurablePrpControlPlane { options.stateDirectory, options.identity, ); + if (this.#store.state.checkpointEventPolicy === "control-v1") { + this.#recoveredCursor = { runId: options.identity.runId, sourceSeq: this.#store.state.ackedSourceSeq }; + } this.#expectedRunnerVersion = options.expectedRunnerVersion; this.#expectedRunnerDigest = options.expectedRunnerDigest; const transition = this.#store.state.warmTransition; @@ -1596,16 +1808,135 @@ export class DurablePrpControlPlane { } this.#onSemanticToolInput = options.onSemanticToolInput; this.#onCommittedEvent = options.onCommittedEvent; + this.#recoverSemanticToolInput = options.recoverSemanticToolInput; this.#beforeAuthenticatedConnection = options.beforeAuthenticatedConnection; this.#onProtocolIntegrityError = options.onProtocolIntegrityError; this.#connectionLeaseTtlMs = options.connectionLeaseTtlMs ?? 60_000; } get store(): DurablePrpControlPlaneStore { - return this.#store; + return this.#publishedStore ?? this.#store; + } + + #capturePublishedState(snapshot: DurablePrpCheckpoint): StoredCoreState { + const header = JSON.parse(snapshot.records.get("header")!) as CheckpointHeader; + const cursor = JSON.parse(snapshot.records.get("cursor")!) as { ackedSourceSeq: number; replayDeliveries: number }; + const cached = new Map(); + for (const [part, record] of snapshot.records) { + if (!part.startsWith("command:")) continue; + const prior = this.#publicationCommands.get(part); + cached.set(part, prior?.record === record ? prior : { record, command: JSON.parse(record) }); + } + this.#publicationCommands = cached; + return { ...header.state, ...cursor, commands: [...cached.values()].map(value => value.command), + // Event records are immutable after insertion. Retain their array, not + // another deep copy of the streaming payload window. + committedEvents: this.#store.state.committedEvents }; + } + + #visibleState(): StoredCoreState { + const published = this.#publishedState!; + const working = this.#store.state; + const cached = this.#visibleStateCache; + if (cached?.working === working && cached.published === published && cached.revision === this.#store.revision) { + return cached.state; + } + // A warm handoff cannot expose its new authority before persistence. + if (canonicalJson(working.identity) !== canonicalJson(published.identity)) return published; + const barrier = this.#persistedRevision === this.#store.revision ? -1 + : working.committedEvents.findIndex(event => event.sourceSeq > published.ackedSourceSeq && !isStreamingEvent(event)); + const state = { ...published, + committedEvents: barrier < 0 ? working.committedEvents : working.committedEvents.slice(0, barrier), + ackedSourceSeq: barrier < 0 ? working.ackedSourceSeq : Math.min(working.ackedSourceSeq, working.committedEvents[barrier].sourceSeq - 1), + }; + this.#visibleStateCache = { working, published, revision: this.#store.revision, state }; + return state; + } + + /** Sparse, already-acknowledged history; never permits gaps in new delivery. */ + get recoveredSourceSeq(): number { + return this.#recoveredCursor?.runId === this.#identity.runId ? this.#recoveredCursor.sourceSeq : 0; + } + + /** + * Make the callback the durable authority. Local state becomes a marker; + * restart must restore its checkpoint before constructing the controller. + * Outbound frames and semantic dispatch still await the callback's commit. + */ + setCheckpointPersistence(persist: (checkpoint: DurablePrpCheckpoint) => Promise): void { + if (this.#connections.size > 0 || this.#persistCheckpoint !== null) { + throw new Error("runner_controller_persistence_already_active"); + } + this.#store.useExternalPersistence(); + this.#persistCheckpoint = persist; + this.#publishedState = this.#capturePublishedState(captureCheckpoint(this.#store.state, + this.#expectedRunnerVersion, this.#expectedRunnerDigest)); + const core = this; + this.#publishedStore = { path: this.#store.path, get state() { return core.#visibleState(); } }; + } + + flushCheckpoint(): Promise { + if (!this.#persistCheckpoint || this.#persistedRevision === this.#store.revision + || this.#queuedRevision === this.#store.revision) { + return this.#persistenceTail; + } + const snapshot = captureCheckpoint(this.#store.state, + this.#expectedRunnerVersion, this.#expectedRunnerDigest); + const published = this.#capturePublishedState(snapshot); + const revision = this.#store.revision; + this.#queuedRevision = revision; + const persist = this.#persistCheckpoint; + const pending = this.#persistenceTail.catch(() => undefined).then(async () => { + // The previous write may have failed after committing. Rebuild the delta + // at execution time, and reset after uncertainty instead of assuming it failed. + const checkpoint = { ...snapshot, ...checkpointChanges(snapshot.records, snapshot.runId, + this.#checkpointPersistenceFailed ? undefined : this.#lastCheckpoint) }; + try { + await persist(checkpoint); + this.#lastCheckpoint = checkpoint; + this.#publishedState = published; + this.#persistedRevision = revision; + this.#checkpointPersistenceFailed = false; + } catch (error) { + this.#checkpointPersistenceFailed = true; + throw error; + } + }); + this.#persistenceTail = pending; + void pending.then(() => { + if (this.#persistenceTail === pending) this.#queuedRevision = -1; + }, () => { + if (this.#persistenceTail === pending) this.#queuedRevision = -1; + this.disconnectActiveRunner(); + }); + return pending; + } + + #sendJson(connection: AuthorityConnection, value: unknown): void { + if (!this.#persistCheckpoint) { + connection.sendJson(value); + return; + } + const checkpoint = this.flushCheckpoint(); + const frame = structuredClone(value); + const sending = (this.#sendQueues.get(connection) ?? Promise.resolve()) + .then(() => checkpoint) + .then(() => connection.sendJson(frame)) + .catch(() => connection.close()); + this.#sendQueues.set(connection, sending); + void sending.finally(() => { + if (this.#sendQueues.get(connection) === sending) this.#sendQueues.delete(connection); + }); } getCommand(commandId: string): DurableRecoveryCoreCommand | undefined { + const state = this.store.state; + return state.commands.find(command => command.commandId === commandId) + ?? (state.warmTransition?.command.commandId === commandId ? state.warmTransition.command : undefined) + ?? (state.completedWarmTransition?.command.commandId === commandId ? state.completedWarmTransition.command : undefined); + } + + #getCommand(commandId: string): DurableRecoveryCoreCommand | undefined { return ( this.#store.state.commands.find( (command) => command.commandId === commandId, @@ -1620,6 +1951,36 @@ export class DurablePrpControlPlane { ); } + async recoverPendingSemanticInputs(): Promise { + if (!this.#recoverSemanticToolInput) return; + const identity = this.#identity; + for (const saved of [...this.#store.state.committedEvents]) { + if (!unsettledSemanticInput(saved, this.#store.state)) continue; + const event = saved.envelope.payload as PrpEvent; + const semantic = (event.payload as Record).semantic_tool as Record; + if (!semantic || event.runId !== identity.runId || event.normalizedSessionId !== identity.normalizedSessionId) continue; + if (typeof semantic.callId !== "string" || typeof semantic.operationId !== "string") continue; + const commandId = `command_tool_${createHash("sha256").update(`${identity.runId}\0${semantic.callId}`).digest("hex").slice(0, 32)}`; + if (this.#pendingSemanticCalls.has(commandId) || this.#getCommand(commandId)) continue; + this.#pendingSemanticCalls.add(commandId); + try { + const call = { callId: semantic.callId, operationId: semantic.operationId, input: semantic.input, + sourceEventId: event.sourceEventId, sourceEventType: event.eventType, + correlation: { runId: identity.runId, normalizedSessionId: identity.normalizedSessionId, + turnId: identity.turnId, itemId: event.itemId ?? identity.itemId } }; + const outcome = await this.#recoverSemanticToolInput(call); + if (this.#identity !== identity || this.#semanticCallbacksRetired) return; + if (!outcome || this.#getCommand(commandId)) continue; + const { input, ...metadata } = call; + this.queueCommand("semantic_tool.result", { ...metadata, inputDigest: canonicalDigest(input), + result: outcome.result, isError: outcome.isError === true }, commandId, true); + await this.flushCheckpoint(); + } finally { + this.#pendingSemanticCalls.delete(commandId); + } + } + } + get connectUrl(): string { if (this.#port === null) { throw new Error("Durable PRP control plane is not listening."); @@ -1884,6 +2245,28 @@ export class DurablePrpControlPlane { this.#store.save(); } + preparedLaunchConfigurationId(identity: DurableRecoveryIdentity): string | undefined { + return this.#store.state.preparedLaunchConfigurations?.find(binding => + canonicalJson(binding.identity) === canonicalJson(identity))?.configurationId; + } + + /** Called only after the environment has validated a prepared launch. */ + persistPreparedLaunchConfiguration(identity: DurableRecoveryIdentity, configurationId: string): void { + this.#store.assertWritable(); + if (!exactIdentity(identity) || typeof configurationId !== "string" || !configurationId.trim() || configurationId.length > 1024) { + throw new Error("runner_prepared_launch_configuration_invalid"); + } + const existing = this.preparedLaunchConfigurationId(identity); + if (existing !== undefined) { + if (existing !== configurationId) throw new Error("runner_prepared_launch_configuration_changed"); + return; + } + const current = this.#store.state.preparedLaunchConfigurations?.filter(binding => + canonicalJson(binding.identity) === canonicalJson(this.#identity)) ?? []; + this.#store.state.preparedLaunchConfigurations = [...current, { identity: structuredClone(identity), configurationId }]; + this.#store.save(); + } + issueBootstrapTicket(ttlMs = 5_000): string { this.#store.assertWritable(); if (this.#store.state.warmTransition) { @@ -2053,6 +2436,7 @@ export class DurablePrpControlPlane { throw new CommandJournalLimitError("command_payload_too_large"); this.#store.state.commands.push(command); this.#store.save(); + void this.flushCheckpoint().catch(() => undefined); if (deliverImmediately) { for (const connection of this.#connections) { if (connection.secureChannel !== null) { @@ -2235,6 +2619,7 @@ export class DurablePrpControlPlane { ) connection.replayOnly = true; this.#commandResult(connection, envelope); + await this.flushCheckpoint(); return; } if (kind === "warm_transition_activated") { @@ -2275,7 +2660,7 @@ export class DurablePrpControlPlane { this.#store.commit(candidate); connection.lease = this.#store.state.leases[transition.credentialId]!; } - connection.sendJson( + this.#sendJson(connection, this.#controlEnvelope( connection, `activation_ack_${receipt.transitionId}`, @@ -2335,7 +2720,7 @@ export class DurablePrpControlPlane { this.#store.commit(candidate); connection.lease = this.#store.state.leases[lease.credentialId]!; } - connection.sendJson( + this.#sendJson(connection, this.#controlEnvelope( connection, `lease_renewed_${expectedExpiry}`, @@ -2567,6 +2952,16 @@ export class DurablePrpControlPlane { connection.close(); return; } + const requiresRecoveryCursor = this.#persistCheckpoint !== null || this.#store.state.checkpointEventPolicy === "control-v1"; + const resume = payload.resume; + const acknowledgedSourceSeq = payload.recoveryCursorVersion === 1 && isRecord(resume) + && Number.isSafeInteger(resume.ackedSourceSeq) && (resume.ackedSourceSeq as number) >= 0 + && Number.isSafeInteger(resume.nextSourceEventSeq) && (resume.nextSourceEventSeq as number) > (resume.ackedSourceSeq as number) + ? resume.ackedSourceSeq as number : undefined; + if ((requiresRecoveryCursor || payload.recoveryCursorVersion !== undefined) && acknowledgedSourceSeq === undefined) { + connection.close(); + return; + } const serverNonce = randomUUID(); const selectedVersion = authorization.kind === "lease" @@ -2598,6 +2993,9 @@ export class DurablePrpControlPlane { ...(typeof payload.warmTransitionId === "string" ? { warmTransitionId: payload.warmTransitionId } : {}), + ...(acknowledgedSourceSeq !== undefined + ? { recoveryCursorVersion: 1, recoveryAckedSourceSeq: acknowledgedSourceSeq } + : {}), }; const canonicalChallenge = canonicalJson(challengePayload); const serverProof = domainHmac( @@ -2623,8 +3021,9 @@ export class DurablePrpControlPlane { ? { warmTransitionId: payload.warmTransitionId } : {}), requestedIdentity: structuredClone(authorization.identity), + acknowledgedSourceSeq, }; - connection.sendJson({ + this.#sendJson(connection, { protocol, version: selectedVersion, kind: "auth_challenge", @@ -2777,6 +3176,7 @@ export class DurablePrpControlPlane { } if (transition.phase === "prepared") { const candidate = initialCoreState(transition.receipt.newIdentity); + candidate.preparedLaunchConfigurations = structuredClone(this.#store.state.preparedLaunchConfigurations); candidate.schema = transitionCoreStateSchema; candidate.warmTransition = { ...structuredClone(transition), @@ -2790,6 +3190,16 @@ export class DurablePrpControlPlane { lease = this.#store.state.leases[lease.credentialId]!; } } + // The authenticated peer's durable ACK covers only events this authority + // previously released. Every control event was checkpointed before that ACK; + // only display/usage streaming may be absent from the restored event window. + if (pending.acknowledgedSourceSeq !== undefined + && (this.#persistCheckpoint !== null || this.#store.state.checkpointEventPolicy === "control-v1") + && canonicalJson(requestedIdentity) === canonicalJson(this.#store.state.identity)) { + const cursor = Math.max(this.#store.state.ackedSourceSeq, pending.acknowledgedSourceSeq); + this.#store.state.ackedSourceSeq = cursor; + this.#recoveredCursor = { runId: this.#identity.runId, sourceSeq: cursor }; + } connection.pendingChallenge = null; connection.lease = lease; connection.identity = structuredClone(requestedIdentity); @@ -2844,7 +3254,7 @@ export class DurablePrpControlPlane { (this.#store.state.commandDeliveryCounts[command.commandId] ?? 0) + 1; } this.#store.save(); - connection.sendJson({ + this.#sendJson(connection, { protocol, version: lease.protocolVersion, envelopeId: `welcome_${this.#store.state.connectionCount}`, @@ -2955,7 +3365,7 @@ export class DurablePrpControlPlane { this.#store.state.commandDeliveryCounts[command.commandId] = (this.#store.state.commandDeliveryCounts[command.commandId] ?? 0) + 1; this.#store.save(); - connection.sendJson( + this.#sendJson(connection, this.#controlEnvelope( connection, `command_${command.commandId}_${this.#store.state.commandDeliveryCounts[command.commandId]}`, @@ -3093,7 +3503,7 @@ export class DurablePrpControlPlane { connection: AuthorityConnection, receipt: DurableWarmRunTransition, ): void { - connection.sendJson( + this.#sendJson(connection, this.#controlEnvelope( connection, `command_result_ack_${receipt.controllerSeq}`, @@ -3122,7 +3532,7 @@ export class DurablePrpControlPlane { if (!this.#isTerminalLifecycleCommand(command)) { return; } - connection.sendJson( + this.#sendJson(connection, this.#controlEnvelope( connection, `command_result_ack_${command.controllerSeq}`, @@ -3272,10 +3682,7 @@ export class DurablePrpControlPlane { const eventToEvict = existing === undefined && this.#store.state.committedEvents.length >= maxCommittedEventWindow - ? this.#store.state.committedEvents.findIndex( - (candidate) => - !unsettledSemanticInput(candidate, this.#store.state), - ) + ? evictionCandidate(this.#store.state) : null; if (eventToEvict === -1) { connection.close(); @@ -3323,9 +3730,7 @@ export class DurablePrpControlPlane { if (committedEvents.length >= maxCommittedEventWindow) { // A command can settle an input while this snapshot is staged. Use // the current response ledger to select an event that can be evicted. - const currentEviction = committedEvents.findIndex( - (candidate) => !unsettledSemanticInput(candidate, state), - ); + const currentEviction = evictionCandidate(state); if (currentEviction < 0) return null; committedEvents.splice(currentEviction, 1); } @@ -3345,49 +3750,34 @@ export class DurablePrpControlPlane { connection.close(); return; } - this.#store.commit(candidate); - } else if (!await this.#store.commitEvent(buildCandidate, isCurrentOwner)) { + this.#store.commit(candidate, !isStreamingEvent(event)); + } else if (!await this.#store.commitEvent(buildCandidate, isCurrentOwner, !isStreamingEvent(event))) { if (isCurrentOwner()) connection.close(); return; } if (!isCurrentOwner()) return; - if ( - isSemanticInput && - this.#onSemanticToolInput && - semantic !== undefined && - typeof semantic.callId === "string" && - typeof semantic.operationId === "string" - ) { - const call = { - callId: semantic.callId, - operationId: semantic.operationId, - input: semantic.input, - sourceEventId, - sourceEventType: eventType, - correlation: { - runId: this.#identity.runId, - normalizedSessionId: this.#identity.normalizedSessionId, - turnId: this.#identity.turnId, - itemId: - typeof event.itemId === "string" - ? event.itemId - : this.#identity.itemId, - }, - }; - const commandId = `command_tool_${createHash("sha256") - .update(`${this.#identity.runId}\0${call.callId}`) - .digest("hex") - .slice(0, 32)}`; - const alreadyQueued = this.#store.state.commands.some( - (command) => command.commandId === commandId, - ); - // A committed input without a result after restart may already have - // changed the outside world. Never redispatch it on event replay. Only - // the original admitted callback may publish its result; missing outcome - // evidence stays pending for authoritative reconciliation. - if (existing === undefined && !alreadyQueued && !this.#pendingSemanticCalls.has(commandId)) { - this.#pendingSemanticCalls.add(commandId); + const liveCall = isSemanticInput && this.#onSemanticToolInput && semantic !== undefined + && typeof semantic.callId === "string" && typeof semantic.operationId === "string" + ? { callId: semantic.callId, operationId: semantic.operationId, input: semantic.input, + sourceEventId, sourceEventType: eventType, + correlation: { runId: identity.runId, normalizedSessionId: identity.normalizedSessionId, + turnId: identity.turnId, itemId: typeof event.itemId === "string" ? event.itemId : identity.itemId } } + : null; + const liveCommandId = liveCall ? `command_tool_${createHash("sha256") + .update(`${identity.runId}\0${liveCall.callId}`).digest("hex").slice(0, 32)}` : null; + const ownsLiveDispatch = existing === undefined && liveCommandId !== null + && !this.#getCommand(liveCommandId) && !this.#pendingSemanticCalls.has(liveCommandId); + // Reserve the original callback before exposing its checkpoint wait to + // recovery. A missing saved outcome must never suppress first dispatch. + if (ownsLiveDispatch) this.#pendingSemanticCalls.add(liveCommandId!); + let dispatched = false; + try { + await this.flushCheckpoint(); + if (this.#eventOwner !== connection || this.#identity !== identity) return; + if (ownsLiveDispatch && liveCall && liveCommandId && this.#onSemanticToolInput && !this.#getCommand(liveCommandId)) { + const call = liveCall; + const commandId = liveCommandId; const recordFailure = (stage: "persist_result" | "dispatch", error: unknown) => { this.#semanticResultPersistenceFailed = true; // Keep diagnostics bounded and content-free. Exception messages can @@ -3420,7 +3810,9 @@ export class DurablePrpControlPlane { recordFailure("persist_result", error); } }; - void this.#onSemanticToolInput(call) + const outcome = this.#onSemanticToolInput(call); + dispatched = true; + void outcome .then((outcome) => queueResult(outcome.result, outcome.isError === true), ) @@ -3439,18 +3831,19 @@ export class DurablePrpControlPlane { }) .finally(() => this.#pendingSemanticCalls.delete(commandId)); } + this.#sendJson(connection, + this.#controlEnvelope( + connection, + `ack_${this.#store.state.ackedSourceSeq}`, + "ack", + { + ackedSourceSeq: this.#store.state.ackedSourceSeq, + }, + ), + ); + } finally { + if (ownsLiveDispatch && !dispatched) this.#pendingSemanticCalls.delete(liveCommandId!); } - - connection.sendJson( - this.#controlEnvelope( - connection, - `ack_${this.#store.state.ackedSourceSeq}`, - "ack", - { - ackedSourceSeq: this.#store.state.ackedSourceSeq, - }, - ), - ); } } @@ -3722,6 +4115,8 @@ export function spawnRunner(options: { const environment = runnerEnvironment(options.ticket, options.identity.normalizedSessionId, options.environment); const withRestart = (handle: RunnerProcessHandle): RunnerProcessHandle => handle.restart === null ? handle : ({ ...handle, + // Remote owners publish this identity after dispatch; keep it live through the wrapper. + get startedAt() { return handle.startedAt; }, restart: (ticket) => { if (handle.isExitConfirmed?.() === false) { throw new Error("runner_process_termination_unconfirmed"); diff --git a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts index 7a55b20b843..f94fa464082 100644 --- a/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts +++ b/packages/paperclip-runner/src/control-plane/runner-execution-driver.ts @@ -11,6 +11,12 @@ import type { DurableRecoveryIdentity } from "../contracts/durable-recovery.js"; export interface RunnerExecutionDriver { /** Own provider configuration and file preparation in the execution environment. */ prepareLaunch?: RunnerLaunchPreparer; + /** Reauthorize retained resources before adoption without preparing them again. + * The ID is bound to the exact execution in the durable controller checkpoint. + * Omission preserves prepareLaunch's existing recovery/authorization path. + * Adoption with this hook requires prepareLaunch and its retained configuration binding. + */ + authorizeRunnerAdoption?: (configurationId: string, identity: DurableRecoveryIdentity) => Promise; /** Prepare resources and register or acquire the connection before launch. */ controlPlaneRegistration?: ( authority: DurablePrpControlPlane, @@ -57,9 +63,9 @@ export interface RunnerExecutionDriver { * never calls the process launcher while the process remains alive. */ adoptExistingRunner?: { - pid: number; + pid: number | null; processGroupId: number | null; - startedAt: string; + startedAt: string | null; isAlive: () => Promise | boolean; signal?: (signal: NodeJS.Signals) => Promise | boolean; }; diff --git a/packages/paperclip-runner/src/control-plane/runner-recovery-fixture.test.ts b/packages/paperclip-runner/src/control-plane/runner-recovery-fixture.test.ts new file mode 100644 index 00000000000..e346eee823f --- /dev/null +++ b/packages/paperclip-runner/src/control-plane/runner-recovery-fixture.test.ts @@ -0,0 +1,142 @@ +import { execFileSync, spawn, type ChildProcess } from "node:child_process"; +import { once } from "node:events"; +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import { beforeAll, expect, it, vi } from "vitest"; +import { + DurablePrpControlPlane, decodeDurablePrpCheckpoint, type DurablePrpCheckpoint, +} from "./durable-prp-control-plane.js"; + +const packageRoot = fileURLToPath(new URL("../..", import.meta.url)); +const binary = resolve(packageRoot, "runner/target/debug/examples/recovery-fixture"); +const identity = { + runnerInstanceId: "fixture-runner", environmentLeaseId: "fixture-lease", + runId: "fixture-run", normalizedSessionId: "fixture-session", turnId: "fixture-turn", itemId: "fixture-item", +}; +const expectedRunnerVersion = "0.3.0"; +const expectedRunnerDigest = `sha256:${"c".repeat(64)}`; + +beforeAll(() => { + execFileSync("cargo", ["build", "--locked", "--manifest-path", resolve(packageRoot, "runner/Cargo.toml"), + "-p", "paperclip-runner-core", "--example", "recovery-fixture"], { timeout: 120_000, stdio: "pipe" }); +}, 120_000); + +async function fixture(externalPersistence: boolean, graceMs = 20_000) { + const root = mkdtempSync(resolve(tmpdir(), "runner-in-process-recovery-")); + const stateDirectory = resolve(root, "control-plane"); + const receiptFile = resolve(root, "work.json"); + const releaseFile = resolve(root, "release"); + const scriptFile = resolve(root, "script.json"); + writeFileSync(scriptFile, JSON.stringify({ steps: 64, waitAfter: 32, retentionLimit: 8, receiptFile, releaseFile })); + const delivered: Array<{ sourceEventId: string; text: string }> = []; + let checkpoint: DurablePrpCheckpoint | undefined; + const create = () => { + const core = new DurablePrpControlPlane({ stateDirectory, identity, expectedRunnerVersion, expectedRunnerDigest, + onCommittedEvent: async event => { + if (event.eventType === "item.delta") delivered.push({ sourceEventId: event.sourceEventId, text: (event.payload as { text: string }).text }); + }, + }); + if (externalPersistence) core.setCheckpointPersistence(async saved => { checkpoint = saved; }); + return core; + }; + let core = create(); + let child: ChildProcess | undefined; + let exit: Promise | undefined; + const receipt = () => JSON.parse(readFileSync(receiptFile, "utf8")) as { + pid: number; phase: string; completedSteps: number; retainedEvents: number; peakRetainedEvents: number; acknowledgedEvents: number; + }; + await core.start(); + const port = Number(new URL(core.connectUrl).port); + const ticket = core.issueBootstrapTicket(); + await core.flushCheckpoint(); + child = spawn(binary, ["--fixture-script", scriptFile, "--connect-url", core.connectUrl, + "--state-dir", resolve(root, "runner"), "--runner-id", identity.runnerInstanceId, + "--environment-lease-id", identity.environmentLeaseId, "--run-id", identity.runId, + "--session-id", identity.normalizedSessionId, "--turn-id", identity.turnId, "--item-id", identity.itemId, + "--runner-version", expectedRunnerVersion, "--runner-digest", expectedRunnerDigest, + "--max-outbox-bytes", "65536", "--p0-reserve-bytes", "16384", "--reconnect-grace-ms", String(graceMs), + ], { env: { ...process.env, PAPERCLIP_RUNNER_BOOTSTRAP_TICKET: ticket }, stdio: ["ignore", "ignore", "pipe"] }); + let diagnostics = ""; + child.stderr!.on("data", bytes => { diagnostics = (diagnostics + String(bytes)).slice(-4096); }); + exit = once(child, "exit"); + async function command(type: string, commandId: string) { + core.queueCommand(type, {}, commandId, true); + await vi.waitFor(() => expect(core.getCommand(commandId)?.status, `${commandId}: exit=${child?.exitCode}; ${diagnostics}`).toBe("completed"), { timeout: 10_000 }); + } + return { + root, receipt, child, exit, delivered, command, + checkpoint: () => checkpoint, + release: () => writeFileSync(releaseFile, "compute"), + stopController: async () => { + await core.stop(); + await core.drainPendingConnectionProcessing(); + await core.flushCheckpoint(); + }, + replaceController: async () => { + if (externalPersistence) { + rmSync(stateDirectory, { recursive: true }); + decodeDurablePrpCheckpoint(checkpoint!.records).restoreSessionDirectory(root); + } + core = create(); + await core.start(port); + }, + cleanup: async () => { + await core.stop(); + if (child?.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); + await exit; + rmSync(root, { recursive: true, force: true }); + }, + }; +} + +it.each([false, true])("preserves useful in-process work through repeated replacement (external persistence: %s)", async external => { + const run = await fixture(external); + try { + await run.command("turn.start", "start-once"); + expect(run.receipt().completedSteps).toBe(0); + await run.stopController(); + run.release(); + await vi.waitFor(() => expect(run.receipt()).toMatchObject({ phase: "backpressured", completedSteps: 8, retainedEvents: 8 }), { timeout: 5000 }); + expect(run.receipt().pid).toBe(run.child.pid); + expect(run.child.exitCode).toBeNull(); + expect(run.delivered).toHaveLength(0); + await run.replaceController(); + await vi.waitFor(() => expect(run.delivered).toHaveLength(32), { timeout: 10_000 }); + expect(run.receipt()).toMatchObject({ phase: "awaiting_controller", completedSteps: 32 }); + await run.stopController(); + expect(run.receipt().completedSteps).toBe(32); + await run.replaceController(); + await run.command("session.snapshot", "after-second-replacement"); + expect(run.receipt().completedSteps).toBe(32); + await run.command("turn.steer", "controller-dependent-continuation"); + await vi.waitFor(() => expect(run.delivered).toHaveLength(64), { timeout: 10_000 }); + expect(new Set(run.delivered.map(event => event.sourceEventId)).size).toBe(64); + expect(run.delivered.map(event => Number(event.text.match(/^step=(\d+)/)?.[1]))).toEqual(Array.from({ length: 64 }, (_, i) => i + 1)); + // Controller delivery precedes the acknowledgement reaching the runner. + // Prove backlog settlement at the runner before ending the experiment. + await vi.waitFor(() => expect(run.receipt()).toMatchObject({ + phase: "completed", pid: run.child.pid, completedSteps: 64, acknowledgedEvents: 64, retainedEvents: 0, + }), { timeout: 10_000 }); + expect(run.receipt().peakRetainedEvents).toBeLessThanOrEqual(8); + if (external) expect([...run.checkpoint()!.records.values()].join("\n")).not.toContain("step="); + await run.command("runner.shutdown", "shutdown"); + expect((await run.exit)[0]).toBe(0); + } finally { await run.cleanup(); } +}, 35_000); + +it("stops after the original reconnect deadline while retaining recovery state", async () => { + const run = await fixture(true, 1500); + try { + await run.command("turn.start", "start-once"); + await run.stopController(); + run.release(); + await run.exit; + const state = JSON.parse(readFileSync(resolve(run.root, "runner/runner-state.json"), "utf8")); + expect(state.recoverableFailure).toBe("transport_reconnect_grace_exceeded"); + expect(run.receipt()).toMatchObject({ phase: "stopped", pid: run.child.pid }); + expect(run.receipt().completedSteps).toBeGreaterThan(0); + expect(run.receipt().peakRetainedEvents).toBeLessThanOrEqual(8); + } finally { await run.cleanup(); } +}, 15_000); diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index c8743107d1f..35d3225b5f5 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,64 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(66); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(68); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63"); + }); + + it("preserves both lineages while binding the merged implementation in v68", async () => { + const published = readFileSync(new URL("../../../test/fixtures/copilot-profile-v66-identity.json", import.meta.url)); + const candidate = readFileSync(new URL("../../../test/fixtures/copilot-profile-v67-identity.json", import.meta.url)); + expect(createHash("sha256").update(published).digest("hex")).toBe("9eb0bfd31334210fd41ddb677a165ef7432a2951922e642dddff087e4db8b6bc"); + expect(createHash("sha256").update(candidate).digest("hex")).toBe("6c818541933b97bea896ca29179ad54c887db9b7d563021da1676a8126f960dd"); + const prior = JSON.parse(published.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v68-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "providerLaunchSourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + for (const history of [prior, JSON.parse(candidate.toString("utf8"))]) { + expect(isSupportedAcpxProfileVersion("copilot", history.declaration.agentProfileVersion)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: history.declaration.agentProfileVersion, commandDigest: history.commandDigest })).rejects.toThrow("exact pinned"); + } + }); + + it("preserves published v65 while binding hot-upgrade adoption in v67", async () => { + const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v65-identity.json", import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe("ca1d95c21b3f49c99b6d7c6bd3c2bf149715db3263927ad653cc0c6f33fb4462"); + const prior = JSON.parse(bytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v67-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "providerLaunchSourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + expect(isSupportedAcpxProfileVersion("copilot", 65)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 65, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + }); + + it("retains candidate v66 identity when completion recovery changes in v67", async () => { + const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v66-hot-upgrade-candidate-historical-identity.json", import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe("430f7a3bb4f51a22d6f18a515fb32eb4defa5bab60a6a4901f6750e59f2536c6"); + const prior = JSON.parse(bytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v67-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeRequestBridgeSourceSha256", + ]); + expect(isSupportedAcpxProfileVersion("copilot", 66)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 66, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); + }); + + it("retains the hot-upgrade v59 development identity without replacing published history", async () => { + const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v59-hot-upgrade-candidate-historical-identity.json", import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe("b61fe644383acbbaacd209836878243cb96b143807d90c521ef1fbee2a2fcb0c"); + const prior = JSON.parse(bytes.toString("utf8")); + const sorted = Object.fromEntries(Object.entries(prior.declaration).sort(([a], [b]) => a.localeCompare(b))); + expect(`sha256:${createHash("sha256").update(JSON.stringify(sorted)).digest("hex")}`).toBe(prior.commandDigest); + expect(isSupportedAcpxProfileVersion("copilot", 59)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 59, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); }); it("retains v65 history while binding the shared native Grok effort transport in v66", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index e8810030a7d..7b94907809f 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 66, + "agentProfileVersion": 68, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:0cddc219af25b741b7138f38faa7514dd5ee3427b1d3168e8a4bfbe5180983e8", + "commandDigest": "sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 6d397e70882..25804b55c31 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/dot/runnerd-dot-driver.test.ts b/packages/paperclip-runner/src/drivers/dot/runnerd-dot-driver.test.ts index 3cf534e40d5..9e14a1bdd2b 100644 --- a/packages/paperclip-runner/src/drivers/dot/runnerd-dot-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/dot/runnerd-dot-driver.test.ts @@ -132,7 +132,7 @@ it.each([ } }, 45000); -it.each(["local", "custom local directory", "external launcher", "grok external launcher"])("reattaches the %s Rust bridge without duplicate operations and refuses lost checkpoints", async launch => { +it.each(["local", "local pending finish", "custom local directory", "external launcher", "grok external launcher", "external launcher empty controller"])("reattaches the %s Rust bridge without duplicate operations and refuses lost checkpoints", async launch => { const root = await mkdtemp(join(tmpdir(), "dot-driver-recovery-")); await writeFile(join(root, "AGENTS.md"), "Use only the synthetic counter."); const input = execution(root, launch === "grok external launcher" ? "grok_bot" : "openai_dot"); @@ -142,6 +142,11 @@ it.each(["local", "custom local directory", "external launcher", "grok external const settlements = new Map(); const dispatches: unknown[] = []; let controllerPort = 0; + const emptyController = launch === "external launcher empty controller"; + const pendingFinish = launch === "local pending finish"; + const firstCompletion = vi.fn(() => new Promise(() => {})); + const recoveredCompletion = vi.fn(async () => "Completion accepted"); + let checkpoint: controlPlane.DurablePrpCheckpoint | undefined; const options: RunnerdDotDriverOptions = { execution: input, stateDirectory: join(root, "state"), identity: { runnerInstanceId: randomUUID(), environmentLeaseId: randomUUID(), runId: input.binding.runId, @@ -149,18 +154,23 @@ it.each(["local", "custom local directory", "external launcher", "grok external runnerBinary: resolve("runner/target/debug/paperclip-runnerd"), onSpawn: process => { spawned = process; }, controlPlaneRegistration: async authority => { + if (emptyController) { + authority.setCheckpointPersistence(async saved => { checkpoint = controlPlane.decodeDurablePrpCheckpoint(new Map(saved.records)); }); + await authority.flushCheckpoint(); + } await authority.start(controllerPort); controllerPort = Number(new URL(authority.connectUrl).port); return { connectUrl: authority.connectUrl, release: () => authority.stop() }; }, dynamicTools: [{ name: "synthetic_counter", description: "Read a synthetic counter", inputSchema: { type: "object", properties: {}, additionalProperties: false } }], dynamicToolHandler: async () => ({ count: ++calls }), + ...(pendingFinish ? { completionFeedback: firstCompletion } : {}), port: { dispatch: async event => { dispatches.push(event); }, settle: async event => { settlements.set(String(event.payload.requestId), event.payload.outcome); }, attach: async callback => { send = callback; return async () => { send = undefined; }; } }, }; - const runnerState = launch === "local" ? join(root, "state/runner") : join(root, "external-runtime/runner"); + const runnerState = launch === "local" || pendingFinish ? join(root, "state/runner") : join(root, "external-runtime/runner"); if (launch === "custom local directory") options.runnerStateDirectory = runnerState; - if (["external launcher", "grok external launcher"].includes(launch)) { + if (launch.includes("external launcher")) { await mkdir(runnerState, { recursive: true }); options.runnerStateDirectory = runnerState; options.readProviderState = async () => { @@ -168,6 +178,7 @@ it.each(["local", "custom local directory", "external launcher", "grok external catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return null; throw error; } }; options.runnerProcessLauncher = spec => { + if (emptyController) expect(Object.keys(JSON.parse(checkpoint!.records.get("header")!).state.tickets)).toHaveLength(1); expect(spec.environment).not.toHaveProperty("OPENAI_API_KEY"); expect(spec.args).toContain(runnerState); const child = spawn(spec.command, [...spec.args], { cwd: runnerState, env: spec.environment, stdio: "ignore" }); @@ -196,13 +207,37 @@ it.each(["local", "custom local directory", "external launcher", "grok external await send!(renewal); await vi.waitFor(() => expect(settlements.get(renewal.requestId)).toMatchObject({ status: "renewed" }), { timeout: 10000 }); } + const finish = operation("tool", { name: "paperclip_finish", arguments: { + schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "Read the counter", + completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [{ criterionId: "objective", status: "satisfied", evidenceRefs: [] }], remainingWork: [] }, + evidence: [], verification: [], attentionRequests: [], artifacts: [], + } }); + if (pendingFinish) { + await send!(finish); + await vi.waitFor(() => expect(firstCompletion).toHaveBeenCalledOnce(), { timeout: 10000 }); + expect(settlements.has(finish.requestId)).toBe(false); + } const snapshot = await session.snapshot(); expect(snapshot).toMatchObject({ providerSessionId: null, activeTurnId: options.identity.turnId, providerRecoveryPolicy: "same_session_only" }); await session.detachControllerForRestart!(); - const recoveredDriver = new RunnerdDotDriver({ ...options, adoptExistingRunner: { ...spawned!, isAlive: () => true } }); + if (emptyController) { + await rm(options.stateDirectory, { recursive: true }); + checkpoint!.restoreSessionDirectory(options.stateDirectory); + } + const recoveredDriver = new RunnerdDotDriver({ ...options, + ...(pendingFinish ? { completionFeedback: recoveredCompletion } : {}), + ...(emptyController ? { runnerBinary: join(root, "unavailable-old-runner"), + runnerArtifact: { version: checkpoint!.runnerVersion, digest: checkpoint!.runnerDigest } } : {}), + adoptExistingRunner: { ...spawned!, isAlive: () => true } }); const recovered = await recoveredDriver.recoverSession(snapshot, { signal: new AbortController().signal }); expect(recovered.recovered).toBe(true); session = recovered.session!; + if (pendingFinish) { + expect(options.recoverSemanticToolInput).toBeUndefined(); + await vi.waitFor(() => expect(settlements.get(finish.requestId)).toMatchObject({ status: "completed", isError: false }), { timeout: 10000 }); + expect(recoveredCompletion).toHaveBeenCalledOnce(); + expect(firstCompletion).toHaveBeenCalledOnce(); + } await send!(call); expect(calls).toBe(1); expect(dispatches).toHaveLength(1); diff --git a/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts b/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts index 6de8688950b..3daecd15d60 100644 --- a/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts +++ b/packages/paperclip-runner/src/drivers/external/runnerd-external-driver.ts @@ -7,7 +7,7 @@ import type { NativeExecutionInputV6, NativeExecutionInputV8 } from "../../contr import type { ExternalProviderPort } from "../../contracts/external-provider.js"; import type { PrpEvent } from "../../protocol/replay-contract.js"; import { validatePrpStructuredRunResult } from "../../protocol/replay-contract.js"; -import { DurablePrpControlPlane, spawnRunner, type RunnerProcessHandle } from "../../control-plane/durable-prp-control-plane.js"; +import { DurablePrpControlPlane, spawnRunner, type RunnerProcessHandle, type DurablePrpControlPlaneOptions } from "../../control-plane/durable-prp-control-plane.js"; import type { DurableRecoveryIdentity } from "../../contracts/durable-recovery.js"; import { authorizedToolSetForProvider, defaultCapabilityRunnerdBinary, readRunnerdArtifactBinding, type CapabilityRunnerdCodexTransportOptions } from "../../live/runnerd-codex-transport.js"; import { codexSemanticToolSpecs } from "../codex/codex-driver-values.js"; @@ -21,6 +21,9 @@ export interface RunnerdExternalDriverOptions { identity: DurableRecoveryIdentity; port: ExternalProviderPort; runnerBinary?: string; + runnerArtifact?: { version: string; digest: string }; + runnerReconnectGraceMs?: number; + recoverSemanticToolInput?: DurablePrpControlPlaneOptions["recoverSemanticToolInput"]; runnerStateDirectory?: string; runnerProcessLauncher?: CapabilityRunnerdCodexTransportOptions["runnerProcessLauncher"]; /** Reads the checkpoint from the execution target. Missing state cannot authorize replay. */ @@ -172,7 +175,32 @@ class RunnerdExternalSession implements HarnessSession { const runnerState = o.runnerStateDirectory ?? resolve(o.stateDirectory, "runner"); if (!o.runnerProcessLauncher) mkdirSync(runnerState, { recursive: true, mode: 0o700 }); const binary = o.runnerBinary ?? defaultCapabilityRunnerdBinary(); - const artifact = readRunnerdArtifactBinding(binary); + const artifact = o.runnerArtifact ?? readRunnerdArtifactBinding(binary); + const handleSemanticInput: NonNullable = async call => { + if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") { + let completionInput: unknown; + try { + completionInput = await resolveCompletionContinuation(call.input, o.identity.runId, + o.dynamicToolHandler ? () => o.dynamicToolHandler!({ + tool: "get_task_context", callId: `${call.callId}:continuation`, + threadId: o.identity.normalizedSessionId, turnId: o.identity.turnId, arguments: {}, + }) : undefined); + } catch (error) { + return { result: { error: error instanceof Error ? error.message : "Completion rejected" }, isError: true }; + } + const result = validatePrpStructuredRunResult(completionInput); + if (!result.ok || (call.operationId === "paperclip_block") !== (result.ok && result.result.reportedWorkDisposition === "blocked")) { + return { result: { error: "Invalid completion report. Use the admitted completion contract and the correct completion tool." }, isError: true }; + } + try { return { result: { accepted: true, completionReport: result.result, feedback: await o.completionFeedback?.(result.result) ?? "Completion accepted; finish the external turn." } }; } + catch (error) { return { result: { error: error instanceof Error ? error.message : "Completion rejected" }, isError: true }; } + } + if (!o.dynamicToolHandler) throw new Error("dot_semantic_authority_unavailable"); + const result = await o.dynamicToolHandler({ tool: call.operationId, callId: call.callId, + threadId: o.identity.normalizedSessionId, turnId: o.identity.turnId, arguments: call.input }); + const outcome = result && typeof result === "object" ? (result as Record).outcome : undefined; + return { result, isError: typeof outcome === "string" && !["completed", "succeeded", "success"].includes(outcome) }; + }; const core = this.#core = new DurablePrpControlPlane({ identity: o.identity, stateDirectory: resolve(o.stateDirectory, "control-plane"), expectedRunnerVersion: artifact.version, expectedRunnerDigest: artifact.digest, @@ -188,31 +216,10 @@ class RunnerdExternalSession implements HarnessSession { if (!this.#events.some(e => e.sourceEventId === event.sourceEventId)) this.#events.push(event); this.#wake(); }, - onSemanticToolInput: async call => { - if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") { - let completionInput: unknown; - try { - completionInput = await resolveCompletionContinuation(call.input, o.identity.runId, - o.dynamicToolHandler ? () => o.dynamicToolHandler!({ - tool: "get_task_context", callId: `${call.callId}:continuation`, - threadId: o.identity.normalizedSessionId, turnId: o.identity.turnId, arguments: {}, - }) : undefined); - } catch (error) { - return { result: { error: error instanceof Error ? error.message : "Completion rejected" }, isError: true }; - } - const result = validatePrpStructuredRunResult(completionInput); - if (!result.ok || (call.operationId === "paperclip_block") !== (result.ok && result.result.reportedWorkDisposition === "blocked")) { - return { result: { error: "Invalid completion report. Use the admitted completion contract and the correct completion tool." }, isError: true }; - } - try { return { result: { accepted: true, completionReport: result.result, feedback: await o.completionFeedback?.(result.result) ?? "Completion accepted; finish the external turn." } }; } - catch (error) { return { result: { error: error instanceof Error ? error.message : "Completion rejected" }, isError: true }; } - } - if (!o.dynamicToolHandler) throw new Error("dot_semantic_authority_unavailable"); - const result = await o.dynamicToolHandler({ tool: call.operationId, callId: call.callId, - threadId: o.identity.normalizedSessionId, turnId: o.identity.turnId, arguments: call.input }); - const outcome = result && typeof result === "object" ? (result as Record).outcome : undefined; - return { result, isError: typeof outcome === "string" && !["completed", "succeeded", "success"].includes(outcome) }; - }, + onSemanticToolInput: handleSemanticInput, + recoverSemanticToolInput: async call => + call.operationId === "paperclip_finish" || call.operationId === "paperclip_block" + ? handleSemanticInput(call) : o.recoverSemanticToolInput?.(call) ?? null, }); // Rehydrate the projection from the authenticated transport journal. Broker // persistence happens before ACK, so these events have durable projections. @@ -222,15 +229,17 @@ class RunnerdExternalSession implements HarnessSession { if (!registration) await core.start(); signal?.throwIfAborted(); if (!o.adoptExistingRunner) { + const ticket = core.issueBootstrapTicket(60_000); + await core.flushCheckpoint(); this.#process = spawnRunner({ processLauncher: o.runnerProcessLauncher, connectUrl: registration?.connectUrl ?? (registration?.connection ? undefined : core.connectUrl), connection: registration?.connection, stateDirectory: runnerState, identity: o.identity, - ticket: core.issueBootstrapTicket(60_000), runnerBinaryPath: binary, + ticket, runnerBinaryPath: binary, runnerVersion: artifact.version, runnerDigest: artifact.digest, maxOutboxBytes: 16 * 1024 * 1024, p0ReserveBytes: 1024 * 1024, // A managed controller rollout can outlast a minute. Task authority // remains gated by the current controller lease throughout the gap. - maxRuntimeMs: 0, reconnectGraceMs: o.runnerProcessLauncher ? 300_000 : 60_000, + maxRuntimeMs: 0, reconnectGraceMs: o.runnerReconnectGraceMs ?? (o.runnerProcessLauncher ? 300_000 : 60_000), // No agent, OAuth, ChatGPT or provider API credentials are inherited. environment: { PATH: process.env.PATH }, }); @@ -275,6 +284,7 @@ class RunnerdExternalSession implements HarnessSession { criterionIds: o.execution.completionContract.contract.criteria.map(c => c.id) }, }, `${this.codec.commandPrefix}_prepare`); await this.#command("session.open", {}, `${this.codec.commandPrefix}_open`); + await core.recoverPendingSemanticInputs(); signal?.throwIfAborted(); } diff --git a/packages/paperclip-runner/src/index.ts b/packages/paperclip-runner/src/index.ts index a4d3769bb4c..45ad697cbc6 100644 --- a/packages/paperclip-runner/src/index.ts +++ b/packages/paperclip-runner/src/index.ts @@ -38,6 +38,8 @@ export { RUNNER_PRP_PROTOCOL_MAX, inspectWarmRunTransition, type DurablePrpControlPlaneOptions, + type DurablePrpCheckpoint, + decodeDurablePrpCheckpoint, type PrpWireConnection, type PrpWireAttachment, type TransportCloseReason, diff --git a/packages/paperclip-runner/src/live/runner-provider-launch.ts b/packages/paperclip-runner/src/live/runner-provider-launch.ts index 6f2a815e3d3..39fa635f1d6 100644 --- a/packages/paperclip-runner/src/live/runner-provider-launch.ts +++ b/packages/paperclip-runner/src/live/runner-provider-launch.ts @@ -157,7 +157,7 @@ export function validatePreparedRunnerLaunch( * Providers without a filesystem working directory return null. */ export function preparedRunnerLaunchWorkingDirectory( - prepared: PreparedRunnerProviderLaunch, + prepared: Pick, ): string | null { return prepared.provider.cwd === undefined ? null : validatePreparedLaunchWorkingDirectory(prepared.provider.cwd); diff --git a/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts b/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts index 14e6351c9c9..f45407609d5 100644 --- a/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-acpx-permission-forwarding.test.ts @@ -10,6 +10,7 @@ import type { HarnessRuntimeRequest } from "../contracts/harness-driver.js"; // event pump, registered handler, and request emission below all run unchanged. const peer = vi.hoisted(() => ({ current: null as TestPeer | null, readiness: {} as Record })); interface TestPeer { + readonly recoveredSourceSeq: number; store: { state: { identity: DurableRecoveryIdentity; committedEvents: DurableRecoveryCommittedEvent[]; commands: Array> } }; emit(type: string, payload: Record): void; } @@ -19,6 +20,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) ...actual, spawnRunner: vi.fn(() => ({ child: { exitCode: null }, completion: new Promise(() => {}) })), DurablePrpControlPlane: class implements TestPeer { + readonly recoveredSourceSeq = 0; store: TestPeer["store"]; connectUrl = "ws://fixture.invalid"; constructor(options: { identity: DurableRecoveryIdentity }) { @@ -30,7 +32,10 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) async drainPendingConnectionProcessing() {} disconnectActiveRunner() {} persistRunAttachTemplate() {} + persistPreparedLaunchConfiguration() {} issueBootstrapTicket() { return {}; } + async flushCheckpoint() {} + async recoverPendingSemanticInputs() {} getCommand(id: string) { return this.store.state.commands.find(c => c.commandId === id); } queueCommand(type: string, payload: Record, id = type) { this.store.state.commands.push({ type, commandId: id, status: "completed", result: { result: { providerTurnId: payload.turnId } } }); diff --git a/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts b/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts index 1dcd3de3875..bdab53a9376 100644 --- a/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-acpx-permission-ordering.test.ts @@ -8,6 +8,7 @@ import type { DurableRecoveryCommittedEvent, DurableRecoveryIdentity } from "../ // event pump, registered handler, and request emission below all run unchanged. const peer = vi.hoisted(() => ({ current: null as TestPeer | null })); interface TestPeer { + readonly recoveredSourceSeq: number; store: { state: { identity: DurableRecoveryIdentity; committedEvents: DurableRecoveryCommittedEvent[]; commands: Array> } }; emit(type: string, payload: Record): void; } @@ -17,6 +18,7 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) ...actual, spawnRunner: vi.fn(() => ({ child: { exitCode: null }, completion: new Promise(() => {}) })), DurablePrpControlPlane: class implements TestPeer { + readonly recoveredSourceSeq = 0; store: TestPeer["store"]; connectUrl = "ws://fixture.invalid"; constructor(options: { identity: DurableRecoveryIdentity }) { @@ -28,7 +30,10 @@ vi.mock("../control-plane/durable-prp-control-plane.js", async (importOriginal) async drainPendingConnectionProcessing() {} disconnectActiveRunner() {} persistRunAttachTemplate() {} + persistPreparedLaunchConfiguration() {} issueBootstrapTicket() { return {}; } + async flushCheckpoint() {} + async recoverPendingSemanticInputs() {} getCommand(id: string) { return this.store.state.commands.find(c => c.commandId === id); } queueCommand(type: string, payload: Record, id = type) { this.store.state.commands.push({ type, commandId: id, status: "completed", result: { result: { providerTurnId: payload.turnId } } }); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts index 26a3e0b20b4..855c41492a3 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.test.ts @@ -608,6 +608,24 @@ it("requires an explicit retained state directory before adopting a runner", () expect(signal).not.toHaveBeenCalled(); }); +it("rejects an adoption authorization hook without launch preparation", () => { + const launch = vi.fn(); + const registration = vi.fn(); + const authorizeRunnerAdoption = vi.fn(); + expect(() => createCapabilityRunnerdCodexTransport({ + stateDirectory: tmpdir(), + runnerProcessLauncher: launch, + controlPlaneRegistration: registration, + authorizeRunnerAdoption, + adoptExistingRunner: { + pid: 123, processGroupId: 123, startedAt: null, isAlive: () => true, + }, + })).toThrow("runner_adoption_authorization_requires_launch_preparation"); + expect(launch).not.toHaveBeenCalled(); + expect(registration).not.toHaveBeenCalled(); + expect(authorizeRunnerAdoption).not.toHaveBeenCalled(); +}); + it("reads valid control-plane history above 64 MiB and rejects it above 256 MiB", async () => { const root = await mkdtemp(join(tmpdir(), "runnerd-large-control-state-")); const stateDirectory = join(root, "control-plane"); @@ -1103,6 +1121,44 @@ it("refuses a reusable close checkpoint when the local provider snapshot is unre } }, 15_000); +it("uses the same API result envelope for live dispatch and recovered receipts", async () => { + const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-recovered-api-")); + const OriginalCore = durableControlPlane.DurablePrpControlPlane; + let callbacks!: ConstructorParameters[0]; + let core!: DurablePrpControlPlane; + const coreSpy = vi.spyOn(durableControlPlane, "DurablePrpControlPlane").mockImplementation(function (options) { + callbacks = options; + return core = new OriginalCore(options); + }); + let outcome: { result: unknown; isError?: boolean } | null = { + result: { ok: true, apiOperationId: "POST /api/example", data: { saved: true } }, isError: false, + }; + const bundle = createCapabilityRunnerdCodexTransport({ + runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, + codexArgs: fakeCodexArgs(stateDirectory), stateDirectory, + recoverSemanticToolInput: async () => outcome, + }); + bundle.transport.setServerRequestHandler(async () => ({ success: !outcome?.isError, + contentItems: [{ type: "inputText", text: JSON.stringify(outcome?.result) }] })); + try { + await bundle.transport.request("thread/start", { cwd: tmpdir() }); + const call = { callId: "recovered-api", operationId: "call_api", input: { operationId: "POST /api/example" }, + sourceEventId: "saved-event", sourceEventType: "semantic_tool.input", correlation: core.store.state.identity }; + const live = await callbacks.onSemanticToolInput!(call); + expect(live).toMatchObject({ isError: false, result: { ok: true, operationId: "call_api", callId: call.callId, result: outcome!.result } }); + expect(await callbacks.recoverSemanticToolInput!(call)).toMatchObject({ result: live.result, isError: live.isError }); + outcome = { result: { error: "saved failure" }, isError: true }; + const liveError = await callbacks.onSemanticToolInput!(call); + expect(await callbacks.recoverSemanticToolInput!(call)).toMatchObject({ result: liveError.result, isError: liveError.isError }); + outcome = null; + expect(await callbacks.recoverSemanticToolInput!(call)).toBeNull(); + } finally { + await bundle.transport.close(); + coreSpy.mockRestore(); + await rm(stateDirectory, { recursive: true, force: true }); + } +}, 20_000); + it.each(["after_budget", "within_budget", "interrupted_within_budget", "persistence_failure"] as const)( "fences reusable suspension against late semantic completion (%s)", async (mode) => { @@ -8374,9 +8430,19 @@ async function verifyLiveRunnerAdoption( goalMidTurn = false, detachBeforeCleanup = false, startWithoutCheckpoint = false, + preparedTask = false, + adoptionAuthorization: "allow" | "deny" | "legacy" | "legacy-missing" | "missing-binding" = "allow", + toolHandlerOrder?: "before" | "after" | "completion-before" | "completion-after", uncertainAtClose = false, ) { const stateDirectory = await mkdtemp(join(tmpdir(), "runnerd-live-adopt-")); + const pendingCompletion = toolHandlerOrder?.startsWith("completion-") === true; + const completionPath = join(stateDirectory, "pending-completion.json"); + if (pendingCompletion) await writeFile(completionPath, JSON.stringify({ + schema: "paperclip.run_result.v1", reportedWorkDisposition: "done", summary: "Completed the synthetic work", + completionClaim: { contractRevision: "1", objectiveSatisfied: true, criteria: [], remainingWork: [] }, + evidence: [], verification: [], attentionRequests: [], artifacts: [], + })); const server = createServer(); let authority: DurablePrpControlPlane | null = null; const checkpoint = vi.fn(async () => undefined); @@ -8393,7 +8459,7 @@ async function verifyLiveRunnerAdoption( const address = server.address(); if (!address || typeof address === "string") throw new Error("Expected adoption test listener"); - const registration = async (next: DurablePrpControlPlane) => { + const registration = vi.fn(async (next: DurablePrpControlPlane) => { authority = next; return { connectUrl: `ws://127.0.0.1:${address.port}/runner`, @@ -8402,7 +8468,7 @@ async function verifyLiveRunnerAdoption( if (authority === next) authority = null; }, }; - }; + }); const identity = { runnerInstanceId: "runner-live-adopt", environmentLeaseId: "lease-live-adopt", @@ -8414,14 +8480,36 @@ async function verifyLiveRunnerAdoption( const sharedOptions = { runnerBinary: defaultCapabilityRunnerdBinary(), codexCommand: fakeCodex, - codexArgs: fakeCodexArgs(stateDirectory, ...(goalMidTurn ? ["--goal-autostart", "--goal-item-trigger", join(stateDirectory, "emit-goal-item")] : [])), + codexArgs: fakeCodexArgs(stateDirectory, ...(pendingCompletion ? ["--terminal-result-file", completionPath] : toolHandlerOrder ? ["--emit-tool-call"] : []), ...(goalMidTurn ? ["--goal-autostart", "--goal-item-trigger", join(stateDirectory, "emit-goal-item")] : [])), stateDirectory, prpIdentity: identity, lifecyclePolicy: { mode: "warm" as const, idleTimeoutMs: 60_000 }, controlPlaneRegistration: registration, }; - const first = createCapabilityRunnerdCodexTransport(sharedOptions); + const prepareLaunch = vi.fn(async (requirements: RunnerProviderLaunchRequirements) => ({ + configurationId: "adopted-preparation", requirementsDigest: runnerLaunchRequirementsDigest(requirements), + provider: createRunnerProviderLaunch(requirements, { options: { codexArgs: sharedOptions.codexArgs }, + root: stateDirectory, cwd: tmpdir(), codexHome: join(stateDirectory, "codex-home"), + codexCommand: fakeCodex, providerNodeCommand: process.execPath, opencodeProxyPath: "", acpxSidecarPath: "" }), + })); + const options = { ...sharedOptions, ...(preparedTask ? { + prepareLaunch, runnerArtifact: readRunnerdArtifactBinding(defaultCapabilityRunnerdBinary()), + runnerProcessLauncher: (spec: durableControlPlane.RunnerProcessLaunchSpec) => { + const child = spawn(spec.command, [...spec.args], { cwd: spec.cwd, env: { ...process.env, ...spec.environment }, + detached: process.platform !== "win32", stdio: "ignore" }); + return { child, processGroupId: child.pid ?? null, completion: new Promise<{ code: number | null; signal: NodeJS.Signals | null; stdout: string; stderr: string }>((resolveExit, reject) => { + child.once("error", reject); + child.once("exit", (code, signal) => resolveExit({ code, signal, stdout: "", stderr: "" })); + }) }; + }, + } : {}) }; + const first = createCapabilityRunnerdCodexTransport(options); + const pendingTool = vi.fn(() => new Promise>(() => undefined)); + if (toolHandlerOrder) first.transport.setServerRequestHandler(pendingTool); + const recoveredTool = vi.fn(async () => ({ result: { receipt: "known-outcome" }, isError: false })); + const redispatchedTool = vi.fn(async () => ({ success: true, contentItems: [] })); let runnerPid: number | null = null; + let runnerStartedAt: string | null = null; let adopted: ReturnType | null = null; let restoreIntervalSpies: () => void = () => undefined; @@ -8430,7 +8518,7 @@ async function verifyLiveRunnerAdoption( try { opened = await first.transport.request("thread/start", { cwd: tmpdir(), - dynamicTools: codexSemanticToolSpecs(), + dynamicTools: toolHandlerOrder && !pendingCompletion ? [{ name: "get_task_context", description: "Read current task", inputSchema: { type: "object", properties: {}, additionalProperties: false } }] : codexSemanticToolSpecs(), }); } catch (error) { const stderr = await readFile( @@ -8442,6 +8530,7 @@ async function verifyLiveRunnerAdoption( ); } runnerPid = first.evidence().runnerPid; + runnerStartedAt = first.transport.processInfo().startedAt; expect(runnerPid).toEqual(expect.any(Number)); if (goalMidTurn) { @@ -8451,6 +8540,10 @@ async function verifyLiveRunnerAdoption( } } + if (toolHandlerOrder) { + await first.transport.request("turn/start", { input: [{ type: "text", text: "Read current task" }] }); + await vi.waitFor(() => expect(pendingTool).toHaveBeenCalledOnce(), { timeout: 5_000 }); + } await first.detachControllerForRestart(); expect(() => process.kill(runnerPid!, 0)).not.toThrow(); if (goalMidTurn) { @@ -8487,6 +8580,20 @@ async function verifyLiveRunnerAdoption( }; if (!startWithoutCheckpoint) await compactProviderIdentityEvents(); + if (preparedTask) { + if (adoptionAuthorization !== "legacy") await rm(join(stateDirectory, "launch-preparation.json")); + prepareLaunch.mockClear(); + registration.mockClear(); + if (adoptionAuthorization === "missing-binding") { + const retained = JSON.parse(await readFile(controlPlaneStatePath, "utf8")); + delete retained.preparedLaunchConfigurations; + await writeFile(controlPlaneStatePath, JSON.stringify(retained), { mode: 0o600 }); + } + } + const authorizeRunnerAdoption = vi.fn(async () => { + expect(registration).not.toHaveBeenCalled(); + if (adoptionAuthorization === "deny") throw new Error("project grant revoked"); + }); const duplicateLauncher = vi.fn(() => { throw new Error("duplicate runner spawn attempted"); }); @@ -8498,7 +8605,8 @@ async function verifyLiveRunnerAdoption( const clearedIntervals = uncertainAtClose ? vi.spyOn(globalThis, "clearInterval") : null; restoreIntervalSpies = () => { intervals?.mockRestore(); clearedIntervals?.mockRestore(); }; adopted = createCapabilityRunnerdCodexTransport({ - ...sharedOptions, + ...options, + ...(preparedTask && !adoptionAuthorization.startsWith("legacy") ? { authorizeRunnerAdoption } : {}), ...(uncertainAtClose ? { closeGraceMs: 2_000 } : {}), // Hash different stable bytes without replacing the real runner artifact // used by concurrent tests. Adoption must never execute this path. @@ -8508,7 +8616,8 @@ async function verifyLiveRunnerAdoption( runnerReconnectGraceMs: 150, } : {}), - resumeDynamicTools: [], + ...(toolHandlerOrder && !pendingCompletion ? { recoverSemanticToolInput: recoveredTool } : {}), + resumeDynamicTools: pendingCompletion ? codexSemanticToolSpecs() : toolHandlerOrder ? [{ name: "get_task_context", description: "Read current task", inputSchema: { type: "object", properties: {}, additionalProperties: false } }] : [], resumeProviderSession: startWithoutCheckpoint ? undefined : { driverSessionId: String(openedThread.id), providerSessionId: mismatchedCheckpoint @@ -8519,7 +8628,7 @@ async function verifyLiveRunnerAdoption( adoptExistingRunner: { pid: runnerPid!, processGroupId: runnerPid, - startedAt: new Date().toISOString(), + startedAt: runnerStartedAt, signal, isAlive: () => { if (rejectLiveness) return Promise.reject(probeError); @@ -8532,6 +8641,18 @@ async function verifyLiveRunnerAdoption( }, }, }); + if (preparedTask && ["deny", "legacy-missing", "missing-binding"].includes(adoptionAuthorization)) { + await expect(adopted.transport.request("thread/read", {})).rejects.toThrow( + adoptionAuthorization === "deny" ? "project grant revoked" : adoptionAuthorization === "legacy-missing" + ? "runner_launch_recovery_requirements_unavailable" : "runner_launch_recovery_configuration_unavailable"); + expect(registration).not.toHaveBeenCalled(); + expect(prepareLaunch).not.toHaveBeenCalled(); + expect(duplicateLauncher).not.toHaveBeenCalled(); + expect(signal).not.toHaveBeenCalled(); + expect(() => process.kill(runnerPid!, 0)).not.toThrow(); + expect(authorizeRunnerAdoption).toHaveBeenCalledTimes(adoptionAuthorization === "deny" ? 1 : 0); + return; + } if (mismatchedArtifact) { await expect( adopted.transport.request("thread/read", {}), @@ -8576,12 +8697,26 @@ async function verifyLiveRunnerAdoption( expect(() => process.kill(runnerPid!, 0)).not.toThrow(); return; } + if (toolHandlerOrder?.endsWith("before")) adopted.transport.setServerRequestHandler(redispatchedTool); await expect(adopted.transport.request(startWithoutCheckpoint ? "thread/start" : "thread/read", {})).resolves.toEqual( expect.objectContaining({ thread: expect.objectContaining({ id: "codex-thread-1" }), }), ); + if (toolHandlerOrder?.endsWith("after")) adopted.transport.setServerRequestHandler(redispatchedTool); + if (toolHandlerOrder) { + await vi.waitFor(() => { + expect(pendingCompletion ? redispatchedTool : recoveredTool).toHaveBeenCalledOnce(); + const result = authority!.store.state.commands.find(command => command.type === "semantic_tool.result"); + expect(result?.status).toBe("completed"); + }, { timeout: 5_000 }); + if (pendingCompletion) { + expect(recoveredTool).not.toHaveBeenCalled(); + expect(redispatchedTool).toHaveBeenCalledWith(expect.objectContaining({ params: expect.objectContaining({ tool: "paperclip_finish" }) })); + } else expect(redispatchedTool).not.toHaveBeenCalled(); + } expect(adopted.evidence().runnerPid).toBe(runnerPid); + expect(adopted.transport.processInfo().startedAt).toBe(runnerStartedAt); if (startWithoutCheckpoint) { const retained = JSON.parse(await readFile(controlPlaneStatePath, "utf8")); for (const type of ["run.prepare", "session.open"]) { @@ -8601,6 +8736,16 @@ async function verifyLiveRunnerAdoption( expect(observed.params).toMatchObject({ threadId: "codex-thread-1", turnId: "provider-goal-turn-1" }); } expect(duplicateLauncher).not.toHaveBeenCalled(); + if (preparedTask) { + if (adoptionAuthorization === "legacy") { + expect(prepareLaunch).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({ identity }), expect.objectContaining({ configurationId: "adopted-preparation" })); + expect(authorizeRunnerAdoption).not.toHaveBeenCalled(); + } else { + expect(prepareLaunch).not.toHaveBeenCalled(); + expect(authorizeRunnerAdoption).toHaveBeenCalledExactlyOnceWith("adopted-preparation", identity); + } + expect((await adopted.transport.request("thread/read", {})).thread).toMatchObject({ cwd: tmpdir() }); + } if (uncertainAtClose) { const before = JSON.parse(await readFile(controlPlaneStatePath, "utf8")); const retainedAuthority = authority as DurablePrpControlPlane | null; @@ -8680,7 +8825,12 @@ it( 30_000, ); -it("retains authenticated adopted-runner ownership when a close probe rejects", () => verifyLiveRunnerAdoption(false, false, false, false, false, true), 30_000); +it("adopts a prepared runner after losing its controller launch cache without preparing resources again", + () => verifyLiveRunnerAdoption(false, false, false, false, false, true), 30_000); + +it.each(["deny", "legacy", "legacy-missing", "missing-binding"] as const)("preserves adoption authorization with %s", + mode => verifyLiveRunnerAdoption(false, false, false, false, false, true, mode), 30_000); +it("retains authenticated adopted-runner ownership when a close probe rejects", () => verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", undefined, true), 30_000); it( "blocks adopted runner artifact drift without duplicate launch, checkpoint replacement, or process signals", @@ -9027,9 +9177,10 @@ it.each([true, false])("preserves prepared OpenCode cleanup errors (primary fail }); it.each([ - { model: "openrouter/deepseek/deepseek-v4-flash-0731", supportsReasoning: true }, - { model: "anthropic/claude-sonnet-4", supportsReasoning: false }, -])("preserves prepared input and completion feedback through runnerd and the real OpenCode proxy boundary ($model)", async ({ model, supportsReasoning }) => { + { model: "openrouter/deepseek/deepseek-v4-flash-0731", supportsReasoning: true, adoptAfterTurn: false }, + { model: "openrouter/deepseek/deepseek-v4-flash-0731", supportsReasoning: true, adoptAfterTurn: true }, + { model: "anthropic/claude-sonnet-4", supportsReasoning: false, adoptAfterTurn: false }, +])("preserves prepared input and completion feedback through runnerd and the real OpenCode proxy boundary ($model, live adoption: $adoptAfterTurn)", async ({ model, supportsReasoning, adoptAfterTurn }) => { const root = await mkdtemp(join(tmpdir(), "runnerd-prepared-opencode-")); // GitHub-hosted Linux toolcache Node can be group-writable, unlike the AWS // fleet. Qualify an owned copy with strict permissions, never chmod the host @@ -9062,7 +9213,30 @@ it.each([ await writeFile(proxy, proxyBytes, { mode: 0o755 }); const digest = (file: string) => `sha256:${createHash("sha256").update(readFileSync(file)).digest("hex")}`; const runtime = join(root, "opencode"); + const server = createServer(); + let authority: DurablePrpControlPlane | null = null; + server.on("upgrade", (request, socket, head) => { + if (authority) authority.handleUpgrade(request, socket, "/runner", head); + else socket.destroy(); + }); + await new Promise(resolveListen => server.listen(0, "127.0.0.1", resolveListen)); + const address = server.address(); + if (!address || typeof address === "string") throw new Error("Expected OpenCode adoption listener"); + const registration = async (next: DurablePrpControlPlane) => { + authority = next; + return { + connectUrl: `ws://127.0.0.1:${address.port}/runner`, + release: async () => { if (authority === next) authority = null; }, + }; + }; + const identity = { + runnerInstanceId: "runner-opencode-adopt", environmentLeaseId: "lease-opencode-adopt", + runId: "prepared-opencode", normalizedSessionId: "prepared-opencode", + turnId: "turn-opencode-adopt", itemId: "item-opencode-adopt", + }; const bundle = createCapabilityRunnerdCodexTransport({ + prpIdentity: identity, + controlPlaneRegistration: registration, provider: "opencode", runnerBinary: defaultCapabilityRunnerdBinary(), stateDirectory: join(root, "runner-state"), @@ -9097,6 +9271,8 @@ it.each([ }, }); let session: Awaited> | undefined; + let adopted: ReturnType | undefined; + let survivingPid: number | null = null; const prepared = JSON.stringify({ schema: "paperclip.native-model-envelope.v3", task: { prompt: "Keep this completion-feedback request unchanged." }, @@ -9139,6 +9315,54 @@ it.each([ expect(outcomes).toHaveLength(2); expect(outcomes[0].result).toMatchObject({ isError: true, content: [{ text: expect.stringContaining("required document link") }] }); expect(outcomes[1].result).toMatchObject({ content: [{ text: expect.stringContaining(feedback) }] }); + if (adoptAfterTurn) { + const snapshot = await session.snapshot(); + const runnerStartedAt = bundle.transport.processInfo().startedAt; + survivingPid = bundle.evidence().runnerPid; + expect(survivingPid).toEqual(expect.any(Number)); + const artifact = readRunnerdArtifactBinding(defaultCapabilityRunnerdBinary()); + await bundle.detachControllerForRestart(); + // These controller launch assets are gone. The original native proxy + // and harness remain alive; only authenticated adoption is allowed. + await rm(proxy); + await rm(executable); + const duplicateLauncher = vi.fn(() => { throw new Error("duplicate OpenCode runner launch"); }); + adopted = createCapabilityRunnerdCodexTransport({ + provider: "opencode", + stateDirectory: join(root, "runner-state"), + runnerFilesystemRoot: join(root, "retired-controller"), + runnerBinary: join(root, "retired-controller", "runnerd"), + runnerArtifact: artifact, + prpIdentity: identity, + controlPlaneRegistration: registration, + resumeProviderSession: { + driverSessionId: snapshot.driverSessionId, + providerSessionId: snapshot.providerSessionId, + providerIdentity: snapshot.providerIdentity, + }, + runnerProcessLauncher: duplicateLauncher, + adoptExistingRunner: { + pid: survivingPid!, processGroupId: survivingPid, startedAt: runnerStartedAt, + signal: () => true, + isAlive: () => { try { process.kill(survivingPid!, 0); return true; } catch { return false; } }, + }, + }); + const recoveryDriver = new CodexAppServerDriver({ + taskEnvelope: task, conversationMode: "prepared", model, + transportFactory: () => adopted!.transport, + workingDirectoryAuthority: "remote_runner", + environment: { PAPERCLIP_WORKSPACE_CWD: root }, + completionFeedback: async () => feedback, + }); + const recovery = await recoveryDriver.recoverSession(snapshot); + if (!recovery.recovered) throw new Error(`OpenCode live adoption failed: ${recovery.reason}`); + expect(recovery.session).toBeDefined(); + session = recovery.session!; + expect(adopted.evidence().runnerPid).toBe(survivingPid); + expect(adopted.transport.processInfo().startedAt).toBe(runnerStartedAt); + expect((await session.snapshot()).providerSessionId).toBe(snapshot.providerSessionId); + expect(duplicateLauncher).not.toHaveBeenCalled(); + } await session.startTurn({ message: { role: "user", text: "Reply Hi." } }); for await (const event of session.events()) if (event.eventType === "turn.completed") break; const allRequests = (await readFile(join(runtime, sessionRoots[0]!.name, "data/fake-prompt-requests.ndjson"), "utf8")).trim().split("\n").map((line) => JSON.parse(line)); @@ -9146,9 +9370,18 @@ it.each([ ? ["paperclip-no-reasoning", "paperclip-default"] : [undefined, undefined]); }, closeSession: async () => { await session?.close(); }, - closeTransport: () => bundle.transport.close(), - removeRoot: () => rm(root, { recursive: true, force: true }), - evidence: () => bundle.evidence(), + closeTransport: async () => { + try { await adopted?.transport.close(); } + finally { + await bundle.transport.close(); + if (survivingPid !== null) { try { process.kill(survivingPid, "SIGKILL"); } catch {} } + } + }, + removeRoot: async () => { + await new Promise((resolveClose, reject) => server.close(error => error ? reject(error) : resolveClose())); + await rm(root, { recursive: true, force: true }); + }, + evidence: () => adopted?.evidence() ?? bundle.evidence(), }); }, 30_000); @@ -9261,3 +9494,10 @@ it("launches and closes native Codex without materializing or editing the precon await rm(root, { force: true, recursive: true }); } }, 30_000); + + +it.each(["before", "after"] as const)("settles a saved tool outcome with the handler installed %s recovery startup", order => + verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", order), 30_000); + +it.each(["completion-before", "completion-after"] as const)("recovers a pending finish without an ordinary-tool lookup (%s)", order => + verifyLiveRunnerAdoption(false, false, false, false, false, false, "allow", order), 30_000); diff --git a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts index c05c6a14894..912fb0de809 100644 --- a/packages/paperclip-runner/src/live/runnerd-codex-transport.ts +++ b/packages/paperclip-runner/src/live/runnerd-codex-transport.ts @@ -1238,6 +1238,7 @@ export interface CapabilityRunnerdCodexTransportOptions extends RunnerExecutionD provider?: "codex" | "opencode" | "claude_managed" | "aws_agentcore" | "acpx"; observeCredentials?: boolean; onCredentialsUpdated?: (event: PrpEvent) => Promise; + recoverSemanticToolInput?: DurablePrpControlPlaneOptions["recoverSemanticToolInput"]; opencodePermissionMode?: NativeOpenCodePermissionMode; acpxAgent?: QualifiedAcpxAgent; /** Explicit evaluation-only candidate selection, never derived from persisted session input. */ @@ -3603,6 +3604,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { readonly #bridgedRuntimeInputs = new Map(); constructor(readonly options: CapabilityRunnerdCodexTransportOptions) { + if (options.adoptExistingRunner && options.authorizeRunnerAdoption && !options.prepareLaunch) { + throw new Error("runner_adoption_authorization_requires_launch_preparation"); + } if (options.prepareLaunch && (!options.runnerProcessLauncher || !options.controlPlaneRegistration || !options.runnerArtifact)) { throw new Error("runner_launch_preparation_requires_process_owner_and_artifact"); } @@ -4025,6 +4029,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#handler = handler; this.#handlerInstalled = true; this.#resolveHandlerReady(); + this.#recoverPendingSemanticInputsWhenReady(); + } + + #recoverPendingSemanticInputsWhenReady(): void { + if (this.#startupComplete && this.#handlerInstalled) { + void Promise.resolve().then(() => this.#core?.recoverPendingSemanticInputs()) + .catch(error => this.#failTransport(error instanceof Error ? error : new Error(String(error)))); + } } async #awaitWarmRunAttachmentReady(): Promise { @@ -4312,7 +4324,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { return { pid: this.#evidence.runnerPid, processGroupId: this.#evidence.runnerProcessGroupId, - startedAt: this.#startedAt, + startedAt: this.options.adoptExistingRunner?.startedAt ?? this.#handle?.startedAt ?? this.#startedAt, exited: this.#evidence.runnerExited, exitCode: this.#evidence.runnerExitCode, signal: this.#evidence.runnerSignal, @@ -4327,7 +4339,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { await this.options.onSpawn?.({ pid: handle.child.pid, processGroupId: handle.processGroupId ?? null, - startedAt: this.#startedAt, + startedAt: handle.startedAt ?? this.#startedAt, }); } } @@ -4837,12 +4849,13 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#assertStartupOpen(); template.provider = prepared.provider; new RunnerLaunchPreparationStore(this.#root).save(requirements, prepared, this.#core!.store.state.identity); + this.#core!.persistPreparedLaunchConfiguration(identity, prepared.configurationId); return prepared; } // The prepared launch is the only source of the provider directory. Core // records it in session metadata; the session cannot move to another one. - #recordPreparedWorkingDirectory(prepared: PreparedRunnerProviderLaunch): void { + #recordPreparedWorkingDirectory(prepared: Pick): void { const workingDirectory = preparedRunnerLaunchWorkingDirectory(prepared); if (workingDirectory === null) { throw new Error("runner_prepared_launch_working_directory_unavailable"); @@ -4876,6 +4889,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#recordPreparedWorkingDirectory(prepared); this.#assertStartupOpen(); new RunnerLaunchPreparationStore(this.#root).save(requirements, prepared, this.#core!.store.state.identity); + this.#core!.persistPreparedLaunchConfiguration(identity, prepared.configurationId); const completionContract = record(params.completionContract); return { runAttachTemplate: { observeCredentials: this.options.observeCredentials !== false, @@ -5090,6 +5104,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { expectedRunnerDigest: runnerArtifact.digest, onProtocolIntegrityError: (error) => this.#failTransport(error), onSemanticToolInput: (call) => this.#handleSemanticToolInput(call), + recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { if (event.eventType === "auth.credentials.updated") { await this.options.onCredentialsUpdated?.(event); @@ -5134,6 +5149,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { } this.#assertStartupOpen(); const admissionDeadline = this.#coldAdmissionDeadline(); + const bootstrapTicket = core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS); + await core.flushCheckpoint(); + this.#assertStartupOpen(); const handle = spawnRunner({ connection: registration?.connection ?? { mode: "connect", @@ -5142,7 +5160,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { stateDirectory: this.options.runnerStateDirectory ?? resolve(this.#root, "runner"), identity, - ticket: core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS), + ticket: bootstrapTicket, maxOutboxBytes: RUNNERD_MAX_OUTBOX_BYTES, p0ReserveBytes: RUNNERD_P0_RESERVE_BYTES, maxRuntimeMs: 0, @@ -5444,6 +5462,9 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.#durableTurnId = identity.turnId; this.#eventIdentity = structuredClone(identity); const provider = this.options.provider ?? "codex"; + // Adoption authenticates the original artifact and process. Launch profiles + // belong only to a new spawn and may reference a retired controller build. + const preparesProviderLaunch = !this.options.prepareLaunch && !this.options.adoptExistingRunner; if (!this.options.prepareLaunch && provider === "codex" && (this.options.homeMode === "preconfigured" || record(record(controlPlaneState!.runAttachTemplate).provider).preconfiguredHome !== undefined)) { const seed = rotatedRunAttachPayload(controlPlaneState!, desiredIdentity, this.#authorizedTools, this.options.resumeCompletionContract); const priorHome = record(seed.provider).preconfiguredHome; @@ -5465,14 +5486,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ? resolve(this.options.runnerFilesystemRoot, "codex-home") : localCodexHome; const opencodeProxyPath = - (this.options.prepareLaunch ? "" : undefined) ?? this.options.opencodeProxyPath ?? + (preparesProviderLaunch ? undefined : "") ?? this.options.opencodeProxyPath ?? (provider === "opencode" && !this.options.runnerFilesystemRoot ? resolveBuildOwnedCliArtifact("opencode-app-server-proxy.cjs") : fileURLToPath( new URL("../cli/opencode-app-server-proxy.cjs", import.meta.url), )); const acpxSidecarPath = - (this.options.prepareLaunch ? "" : undefined) ?? this.options.acpxSidecarPath ?? + (preparesProviderLaunch ? undefined : "") ?? this.options.acpxSidecarPath ?? (provider === "acpx" && !this.options.runnerFilesystemRoot ? resolveBuildOwnedCliArtifact("acpx-runtime-sidecar.cjs") : fileURLToPath( @@ -5481,11 +5502,11 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { const providerNodeCommand = this.options.providerNodeCommand ?? process.execPath; const opencodeExecutable = - provider === "opencode" && !this.options.prepareLaunch + provider === "opencode" && preparesProviderLaunch ? resolveRunnerOpenCodeExecutable(this.options) : null; const runnerAcpxLaunchProfile = - provider === "acpx" && !this.options.prepareLaunch + provider === "acpx" && preparesProviderLaunch ? acpxRunnerLaunchProfile( this.options, providerNodeCommand, @@ -5493,7 +5514,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ) : undefined; const runnerOpenCodeLaunchProfile = - provider === "opencode" && !this.options.prepareLaunch + provider === "opencode" && preparesProviderLaunch ? opencodeRunnerLaunchProfile( this.options, providerNodeCommand, @@ -5508,6 +5529,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { expectedRunnerDigest: runnerArtifact.digest, onProtocolIntegrityError: (error) => this.#failTransport(error), onSemanticToolInput: (call) => this.#handleSemanticToolInput(call), + recoverSemanticToolInput: call => this.#recoverSemanticToolInput(call), onCommittedEvent: async (event) => { if (event.eventType === "auth.credentials.updated") { await this.options.onCredentialsUpdated?.(event); @@ -5535,8 +5557,19 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { connectionLeaseTtlMs: 60 * 60 * 1_000, }); this.#core = core; + // Reusing resources still requires current environment authorization. The + // optional hook avoids re-preparation after controller storage is lost. + // Existing consumers retain the preparation callback's authorization path. + const authorizeAdoption = this.options.adoptExistingRunner && this.options.authorizeRunnerAdoption; + if (this.options.prepareLaunch && authorizeAdoption) { + const configurationId = core.preparedLaunchConfigurationId(desiredIdentity); + if (!configurationId) throw new Error("runner_launch_recovery_configuration_unavailable"); + await authorizeAdoption(configurationId, structuredClone(desiredIdentity)); + this.#assertStartupOpen(); + this.#recordPreparedWorkingDirectory({ provider: record(record(controlPlaneState!.runAttachTemplate).provider) }); + } let environmentPrepared: PreparedRunnerProviderLaunch | undefined; - if (this.options.prepareLaunch) { + if (this.options.prepareLaunch && !authorizeAdoption) { const preparations = new RunnerLaunchPreparationStore(this.#root); const saved = preparations.load(desiredIdentity); const previous = preparations.load(controlPlaneIdentity(controlPlaneState!)); @@ -5812,6 +5845,7 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { RUNNER_BOOTSTRAP_TICKET_TTL_MS, ) : core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS); + await core.flushCheckpoint(); if (warmRecovery) { await this.options.authorizeWarmTransitionRecovery?.("before_spawn"); } @@ -5963,6 +5997,15 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { ? "runnerd attached the durable provider session to a fresh PRP run authority" : "runnerd restored its durable PRP session and provider thread", ); + this.#recoverPendingSemanticInputsWhenReady(); + } + + async #recoverSemanticToolInput(call: Parameters>[0]) { + if (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block") { + return this.#handleSemanticToolInput(call); + } + const outcome = await this.options.recoverSemanticToolInput?.(call); + return outcome ? this.#bindSemanticToolOutcome(call, outcome) : null; } async #handleSemanticToolInput( @@ -6042,15 +6085,16 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { this.options.provider === "acpx" && (call.operationId === "paperclip_finish" || call.operationId === "paperclip_block"), ); + return this.#bindSemanticToolOutcome(call, outcome); + } + + #bindSemanticToolOutcome(call: { operationId: string; callId: string }, outcome: { result: unknown; isError?: boolean }) { if (call.operationId === "call_api") { const result = record(outcome.result); if ( result.operationId !== call.operationId || result.callId !== call.callId ) { - // HTTP receipts also contain `ok` and `operationId` (the HTTP route). - // Bind that application value inside a real semantic envelope so the - // runner cannot mistake the route for the provider tool's identity. return { ...outcome, result: { @@ -6378,8 +6422,14 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { const event = deferredEvent ?? events.find((candidate) => candidate.sourceSeq > this.#eventSourceSeq); - if (event === undefined) return; + if (event === undefined) { + this.#eventSourceSeq = Math.max(this.#eventSourceSeq, core.recoveredSourceSeq); + return; + } const fromDeferredQueue = deferredEvent !== undefined; + if (!fromDeferredQueue && event.sourceSeq > this.#eventSourceSeq + 1) { + this.#eventSourceSeq = Math.max(this.#eventSourceSeq, Math.min(event.sourceSeq - 1, core.recoveredSourceSeq)); + } if (!fromDeferredQueue && event.sourceSeq !== this.#eventSourceSeq + 1) { throw new Error( `PRP provider event window advanced past source sequence ${this.#eventSourceSeq + 1}`, @@ -7118,9 +7168,10 @@ class DurablePrpCodexTransport implements CodexAppServerTransport { const priorConnectionCount = this.#core.store.state.connectionCount; let recoveredHandle: RunnerProcessHandle; try { - recoveredHandle = restart( - this.#core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS), - ); + const ticket = this.#core.issueBootstrapTicket(RUNNER_BOOTSTRAP_TICKET_TTL_MS); + await this.#core.flushCheckpoint(); + if (this.#closed || this.#failure !== null) return; + recoveredHandle = restart(ticket); } catch (error) { lastDetail = error instanceof Error ? error.message : String(error); attempt += 1; diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v59-hot-upgrade-candidate-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v59-hot-upgrade-candidate-historical-identity.json new file mode 100644 index 00000000000..a623aa0d80c --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v59-hot-upgrade-candidate-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:4418740ed8d2ada631094e37b25df5e2fa374fda3c7b66d5045c35fd79ca19b1", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 59, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "3343df6362252d44d12b079ff78c8f6539d1544f3ce5ec36ff36db70ab0fac31", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "103b83066da72434602e9e461e6278dfb40bb993d2f3b49b2e93df81bbb08b72", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "972c08e6e320d1384221e180391fe817ea1ba632db687201e457d64e68ef4360", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "fb6305de1473a51efa25eff83dcea2ab124dd60cfb8036e5ba56ad6785fc1929", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "2e940cf2d55211a46e08fb8575cf6157c438d5acd5e0a46cb72d8dd0ad8951c4", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "fc7c6568feb0d8d709e9baa51839420c9b90336b4865a9b8405aba06e4d1e9c4", + "profileCompatibilitySourceSha256": "6a26b102b78f282137ff93b0f75eb8788c603fc177802c58715c5ad7b2f3b9e3", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "d248e47a0fe7f2536d1e64cea0b0b410f34104800cd643d995b4473104ecc8e0", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v66-hot-upgrade-candidate-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v66-hot-upgrade-candidate-historical-identity.json new file mode 100644 index 00000000000..d6b3c24d98d --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v66-hot-upgrade-candidate-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:5e9c2d1ca9c1a915ab449871eabd679bc6ac7b62afd96109073b9bbcd8183351", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 66, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "63c14082a11f9b02dea9563408bf22fff9f73b8cec0f9f78e6147572190fd2b4", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "99295623f20124617ba96a2ae9f10a39771de1c6afd540c05e82d6b8a3c6def1", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "d96c3b809d083f9d43936af05e6bc14b1755c55f255c8e5967b8c16661f34ffa", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v67-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v67-identity.json new file mode 100644 index 00000000000..d105cf07a6e --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v67-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:7f5897a354d5cd8bdcd165d3e3bab4d6908aaca3981af384f4dfeb4bc0838fd8", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 67, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "17361d1782c0dcebb2b7972635f45c890f2a4081f8bc0ef2d2276550e0b91629", + "semanticDirectDriverSourceSha256": "ff572638f2cb5d9f8249ff8e2f412311c1e3af209ef0bf7e5ea9f10a21d91103", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "016391a087cc217aafed4ba1f4efafc6e4881a7525952dce824b7f4d24530603", + "semanticSidecarProtocolSourceSha256": "03c0c95b900f9dc18f29f6be27b8c79bc0bcab29780190d46a74e95328081236", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "03f4cd7f76f34f6d82e009b66aa51f18f6e23dc235b660778060e35b6ea07412", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "e7b12d4418c7a71f5f2474b046697d3009eae3613b284bec381df5e9965a1c37", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "9173ebccf09cb56a781d2bf5c7eb27fdafb13d3450d99342c6aff86b42d6b43a", + "providerLaunchSourceSha256": "63c14082a11f9b02dea9563408bf22fff9f73b8cec0f9f78e6147572190fd2b4", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "5f2255933fb9974063c9f0cf312333f2d283cc3ec879a217834759c7d00c37b0", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "3a30d52bffad68c7f02349193be84761050eb99039fc0556c2b449891a36e918", + "profileCompatibilitySourceSha256": "94aaea13c8a6957097cfcf5794a6fd5035a00a3f5c7aabee1019a1ca6b9a1670", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "94a73749267f5a80b5df19413a65b4b9263c00a9bf7e98fda00f00b8f9fd32ec", + "runtimeAdapterSourceSha256": "e4c764c2c3c7dd655a965b39472bd1ad8d577d1e7258b25594b171c9429752b8", + "runtimeSandboxSourceSha256": "92dfa53f5a3ed0062913ae417eeccec243b430d444495efde2a8e9c4c02e78ea", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v68-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v68-identity.json new file mode 100644 index 00000000000..397ee98de59 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v68-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 68, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62", + "semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "4cae579aac836762fdbf92c5447425d8b756e87a5766bc0c7a1885a90541c8d6", + "semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b", + "providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "93a631b8439df1590b0cf442cf84a9b05f004f12bfee90291651ccfc0861effe", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297", + "profileCompatibilitySourceSha256": "7b4706f8af55f0359b28d2f941a7774f2932bd8ca4f0aa883559f7aa9e781253", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "976fef21ec9f3ae9dd5eabb133a55ae578f1c0729c657c30e42562a80db237d8", + "runtimeAdapterSourceSha256": "e2dd18de4b134a5492de93c88b4add479f63fb650c655d60a17c04af722c7563", + "runtimeSandboxSourceSha256": "981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} From b5b9491f95782672fca17daa6a08ffdecdc9a314 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Tue, 27 Oct 2026 15:21:12 -0700 Subject: [PATCH 21/41] test(heartbeat): cover first resume with default workspace settings (#17948) ## Thinking Path > - Paperclip manages AI agents and their tasks. > - Task continuations reuse provider sessions when effective configuration is unchanged. > - Repository projects can inherit isolated workspace defaults from a storage plugin. > - Initial workspace binding must not change those defaults. > - PR #17924 fixes policy resolution after binding. > - Its regression supplied an explicit task strategy. > - This pull request also tests unset task settings and an explicit isolated mode. ## Linked Issues or Issue Description Refs #17924 and #17447. **What happened?** On Core revision `e71a23c01c8af0e0b4cc5ac983d86886ba324239`, the first continuation of a task with an inherited repository workspace reset its provider session for `workspaceConfig`. The workspace was reused and its configuration fingerprint did not change. Session preparation runs before workspace binding. The first run resolved the storage plugin's effective project policy. After binding, `!boundSourceWorkspace` skipped that resolver and parsed the raw policy instead. This dropped inherited provider and strategy fields from the session fingerprint. The persisted change from unset task settings to `reuse_existing` and `mode: isolated_workspace` exposed this path; redundant task mode values are already normalized. **Expected behavior** Initial binding and two unchanged continuations retain the session and workspace. Real effective policy changes still reset the session. Explicit task settings and source-free tasks retain their behavior. **Steps to reproduce** 1. Configure a default storage plugin with isolated Git worktrees. 2. Create a repository project with no explicit project policy. 3. Start a task with unset workspace preference and settings. 4. Continue the task twice. **Paperclip version or commit** The runtime fix is already merged in #17924 at `0b65926822aacde48729f4b82a5c333f4cadf375`. This PR adds regression coverage on that fix. It does not duplicate or relax runtime freshness checks. ## What Changed - Run the existing repository heartbeat regression with unset settings, an explicit isolated mode, and an explicit strategy. - Assert that binding persists the expected workspace preference and mode. Each case checks two unchanged continuations and a real policy-change reset. - Make synthetic storage inspection return the branch prepared for that workspace, so generated default branch names pass the existing branch validation. - Update the session verification recipe. ## Verification - All 240 tests pass across `heartbeat-session-preparation.test.ts`, `heartbeat-workspace-session.test.ts`, `project-workspace-provider.integration.test.ts`, `plugin-workspaces.integration.test.ts`, and `plugin-fresh-workspaces.integration.test.ts`. Run them from `server/` with `node ../node_modules/vitest/vitest.mjs run src/__tests__/`. - Restoring the benchmark revision's `!boundSourceWorkspace` condition makes the unset-settings case fail on its first continuation: the dispatched provider session is null. Restoring #17924 makes the complete eight-case session suite pass. - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full `pnpm -r typecheck` and `pnpm build` were attempted. Both stop at Runner Rust checks because this host has no `cargo`. Full `pnpm test:run` was started, then interrupted before its first group completed. It is not counted as a pass; complete suite coverage is left to CI. A separate server typecheck also stops at the missing Runner Rust toolchain. - Initial local attempts found stale cross-checkout dependency links and sandbox restrictions on PostgreSQL/IPC. Isolated workspace links and permitted local PostgreSQL resolved those prerequisites. The first new fixture attempt failed branch validation because inspection returned a hardcoded branch; the fixture now reports its prepared branch. - Greptile scores current head `5185e5e6443e0432b205a8ef63c5c878b9b74bdf` 5/5. Its check concluded success and found no actionable issues. CI is still running. - Tests use real PostgreSQL and heartbeat persistence with synthetic provider execution. No live deployment or TTFT result is claimed. ## Risks Low risk: this PR changes tests and the verification recipe only. Runtime behavior, authorization, and configuration freshness remain owned by the existing implementation. The local full checks are incomplete as stated above. The benchmark deployment needs a revision containing #17924 to receive the runtime fix. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, code editing, and local test execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- maps/feature-map/runs-adapters.md | 4 +++- .../__tests__/heartbeat-session-preparation.test.ts | 10 ++++++++-- server/src/__tests__/helpers/plugin-workspaces.ts | 4 +++- 3 files changed, 14 insertions(+), 4 deletions(-) diff --git a/maps/feature-map/runs-adapters.md b/maps/feature-map/runs-adapters.md index 9432c9a792b..25744903964 100644 --- a/maps/feature-map/runs-adapters.md +++ b/maps/feature-map/runs-adapters.md @@ -75,7 +75,9 @@ mutations. Qualify a useful model task separately from saving the account. ### `session` For a repository project that inherits storage-plugin defaults, run a task and -send two follow-ups. Binding the workspace must preserve the effective policy, +send two follow-ups. Start with unset task workspace settings, then repeat with +an explicit isolated mode or strategy. Binding must save `reuse_existing` and +the resolved mode without changing the effective policy, session fingerprint, and workspace identity. An actual project-policy change must still reset the session and preserve the stored workspace owner. The [session admission regression](../../server/src/__tests__/heartbeat-session-preparation.test.ts) diff --git a/server/src/__tests__/heartbeat-session-preparation.test.ts b/server/src/__tests__/heartbeat-session-preparation.test.ts index 13d0544016b..2d117a6855c 100644 --- a/server/src/__tests__/heartbeat-session-preparation.test.ts +++ b/server/src/__tests__/heartbeat-session-preparation.test.ts @@ -112,7 +112,11 @@ describePostgres("heartbeat session preparation through adapter dispatch", () => } finally { await db.delete(plugins).where(eq(plugins.id, storage.pluginId)); } }); - it("keeps inherited repository policy stable through binding and retains the provider on policy changes", async () => { + it.each([ + null, + { mode: "isolated_workspace" }, + { workspaceStrategy: { type: "git_worktree", branchTemplate: "work" } }, + ])("keeps inherited repository policy stable through binding with settings %j and retains the provider on policy changes", async (settings) => { const f = await seed(); const storage = await createPluginWorkspaceFixture(db, f.companyId); const [plugin] = await db.select().from(plugins).where(eq(plugins.id, storage.pluginId)); @@ -121,7 +125,7 @@ describePostgres("heartbeat session preparation through adapter dispatch", () => } } }).where(eq(plugins.id, storage.pluginId)); await db.update(projects).set({ executionWorkspacePolicy: null }).where(eq(projects.id, storage.input.projectId)); await db.update(issues).set({ projectId: storage.input.projectId, projectWorkspaceId: storage.input.projectWorkspaceId, - executionWorkspaceSettings: { workspaceStrategy: { type: "git_worktree", branchTemplate: "work" } }, + executionWorkspaceSettings: settings, }).where(eq(issues.id, f.issueId)); await instanceSettingsService(db).updateExperimental({ enableIsolatedWorkspaces: true, enableIsolatedWorkspacesByDefault: true }); const cwd = await mkdtemp(join(tmpdir(), "session-provider-fixture-")); @@ -140,6 +144,8 @@ describePostgres("heartbeat session preparation through adapter dispatch", () => const first = await execute(f); const [bound] = await db.select().from(issues).where(eq(issues.id, f.issueId)); expect(bound.executionWorkspaceId).toEqual(expect.any(String)); + expect(bound.executionWorkspacePreference).toBe("reuse_existing"); + expect(bound.executionWorkspaceSettings).toEqual({ ...settings, mode: "isolated_workspace" }); const [workspace] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, bound.executionWorkspaceId!)); expect([...captured.values()].filter(input => input.context.issueId === f.issueId)).toHaveLength(1); expect(workspace).toMatchObject({ providerType: "plugin", strategyType: "git_worktree" }); diff --git a/server/src/__tests__/helpers/plugin-workspaces.ts b/server/src/__tests__/helpers/plugin-workspaces.ts index 31cc69c1546..0ebb4af7a03 100644 --- a/server/src/__tests__/helpers/plugin-workspaces.ts +++ b/server/src/__tests__/helpers/plugin-workspaces.ts @@ -100,6 +100,7 @@ export async function createPluginWorkspaceFixture( status: "running", }); const calls: any[] = []; + const preparedBranches = new Map(); let loseResponse = false, dirty = false, removed = false, @@ -120,6 +121,7 @@ export async function createPluginWorkspaceFixture( } if (params.operation.kind === "prepare") { removed = false; + preparedBranches.set(params.executionWorkspaceId, params.operation.branchName ?? null); return { kind: "prepared", binding: { @@ -141,7 +143,7 @@ export async function createPluginWorkspaceFixture( return { kind: "inspection", headSha: "a".repeat(40), - branchName: "work", + branchName: preparedBranches.get(params.executionWorkspaceId) ?? "work", hasDirtyTrackedFiles: dirty, hasUntrackedFiles: false, hasLocalOnlyData: false, From 3f8149066c2732fb54aafe32f87f6c9bed362374 Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Tue, 27 Oct 2026 15:24:29 -0700 Subject: [PATCH 22/41] Fix connection controls for inherited hiring environments (#17944) https://entire.com/gh/paperclipai/-/trails/106/changes/paperclip/106 ## Thinking Path > - Paperclip lets people hire agents and select execution environments. > - The name and adapter dialog saves an inactive draft before model connection. > - That draft can inherit its environment through a null defaultEnvironmentId. > - Managed-only and forced Kubernetes policy resolve that inheritance to an environment ID. > - The readiness gate compared the stored null with that ID and treated the selection as unsaved. > - This change accepts unchanged inheritance and keeps preparation and save-failure gates. > - People can select auth, open Advanced, and test a saved account before finishing setup. ## Linked Issues or Issue Description **What happened?** The Connect a model screen disabled all auth controls for a draft from the name and adapter dialog under managed-only or forced Kubernetes policy. Reloading did not recover it. The draft had no explicit environment, which is a valid inherited selection. **Expected behavior** Keep connection controls usable once the draft and environment are ready. Test the saved agent against its resolved environment before hiring finishes. Continue to block tests during preparation or an unsaved environment change. **Steps to reproduce** 1. Enable managed-only execution with a valid instance default, or force Kubernetes. 2. Open New agent, enter a name, and choose Claude or Codex. 3. Reach Connect a model with the saved draft's defaultEnvironmentId still null. 4. Try the saved subscription, API-key tile, or Advanced gateway. This follows #17761. The roadmap and related PR search found no existing fix for this readiness regression. ## What Changed - Accept the inherited selection when neither the draft nor the user specifies an environment. - Retain the existing preparation, loading, selection-save, and connection-test gates. - Cover Claude and Codex under managed plugin, managed sandbox, and forced Kubernetes policy. - Extend preparation coverage to both explicit and inherited environments. - Add the inherited-draft journey to the feature map. ## Verification - Hosted reproduction on Core e71a23c01 from a macOS browser: one owned draft was created through the real name/adapter UI. Its persisted environment was null. Auth controls were disabled before and after reload. The instance required managed execution. - Cleanup used the normal Terminate action. Readback confirmed lifecycleState=terminated with no lifecycle error. The roster returned to its original two active agents. No existing agent, task, account, or instance setting was changed. - Before the fix, four inherited-connection cases and the inherited preparation case failed. The explicit-environment preparation control passed. - After the fix, 140 tests passed across NewAgent.test.tsx, test-agent-setup.test.ts, and qualify-claude-setup.test.ts. Coverage includes same-agent probes before Finish, preparation waits, account preservation, failed-save retry, stale reads, and runner qualification. - npm --prefix ui run typecheck passed, including Storybook catalog generation and tsc -b. Direct UI TypeScript, token gates, map checks, and git diff --check also passed. - The pnpm entry point stalled while resolving its pinned version on this host; both owned attempts were stopped. Running the same UI typecheck script through npm passed. Full local workspace typecheck/build were not repeated because Cargo is absent. The exact-head CI run supplies the full workspace validation. - Automated CI on f2a5a2df7 passed all checks. Standard review found only the explanatory source comment, which contributor policy forbids. The comment was removed in 8f43a6d80, and its thread was resolved. The runtime expression and tests are unchanged. Fresh CI passes on 8f43a6d80: 55 successful checks and two intentional skips. The full workflow includes build, typecheck, all test shards, Rust, and end-to-end checks: https://github.com/paperclipai/paperclip/actions/runs/40591187819. Standard Greptile returned 5/5, then Apex returned 5/5 with zero new comments on the same head: https://github.com/paperclipai/paperclip/pull/17944#issuecomment-6454535506. No review threads remain unresolved; the PR is conflict-free. - The diff was reviewed for secrets, PII, and run artifacts. Evidence stays outside git. - No deployment, live post-fix provider test, or VM provisioning was performed. Component tests simulate API responses; the hosted check proves the original failure. ## Risks This changes only how the wizard recognizes an inherited environment. It does not change the environment resolver, lifecycle, approval policy, or lease admission. The existing preparation and failed-save guards remain active. No migration is required. Hosted acceptance of the fix requires a later authorized deployment. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, tool use, browser inspection, and code execution. The exact serving model ID and context window size are not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report fields - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] I have run the targeted tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All Paperclip CI gates are green - [x] Standard Greptile and Apex are 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- maps/feature-map/agents.md | 3 ++ ui/src/components/new-agent/NewAgentSetup.tsx | 4 +- ui/src/pages/NewAgent.test.tsx | 41 ++++++++++++++++++- 3 files changed, 44 insertions(+), 4 deletions(-) diff --git a/maps/feature-map/agents.md b/maps/feature-map/agents.md index f5fce683f77..909e3d1e084 100644 --- a/maps/feature-map/agents.md +++ b/maps/feature-map/agents.md @@ -99,6 +99,9 @@ For an agent-owned plugin environment, verify preparation before connecting, same-agent probes, account preservation after an environment change, failed selection-save retry, and switching back while the computer stops. Confirm the setup hold still blocks ordinary work and API key creation before finishing. +Start through the name/adapter dialog with a managed-only or forced Kubernetes +policy and a draft that inherits its environment. Once preparation completes, +confirm the auth choices, Advanced gateway, and saved subscription remain usable. The [setup lifecycle tests](../../server/src/modules/agent-lifecycle/__tests__/setup.test.ts) exercise durable transitions with synthetic plugin resources; they do not prove live VM provisioning. diff --git a/ui/src/components/new-agent/NewAgentSetup.tsx b/ui/src/components/new-agent/NewAgentSetup.tsx index 7571b2d06ac..3e22e012835 100644 --- a/ui/src/components/new-agent/NewAgentSetup.tsx +++ b/ui/src/components/new-agent/NewAgentSetup.tsx @@ -414,8 +414,8 @@ function Setup({ ); const needsCurrentSetup = ["codex_local", "claude_local", "opencode_local", "pi_local"].includes(adapterType); const preparingComputer = created?.pauseReason === "setup" && ["preparing", "pausing"].includes(created.lifecycleState ?? ""); - const environmentSelectionSaved = (created?.defaultEnvironmentId ?? null) - === (environmentOverride || (forced.forced || managedOnly ? environmentId : null)); + const environmentSelectionSaved = (!environmentOverride && created?.defaultEnvironmentId == null) + || (created?.defaultEnvironmentId ?? null) === (environmentOverride || (forced.forced || managedOnly ? environmentId : null)); const ready = Boolean(created) && environmentSelectionSaved && !environmentSaving && !preparingComputer && connectionSettingsReady && (!needsCurrentSetup || ( envs.isSuccess && !envs.isFetching && settings.isSuccess && !settings.isFetching diff --git a/ui/src/pages/NewAgent.test.tsx b/ui/src/pages/NewAgent.test.tsx index 91fd659420f..16f663819e1 100644 --- a/ui/src/pages/NewAgent.test.tsx +++ b/ui/src/pages/NewAgent.test.tsx @@ -10,6 +10,42 @@ import { } from "./NewAgent.test-support"; describe("New agent setup", () => { + it.each([ + ["claude_local", "Claude", "plugin"], ["codex_local", "OpenAI", "plugin"], + ["claude_local", "Claude", "sandbox"], ["codex_local", "OpenAI", "sandbox"], + ["claude_local", "Claude", "kubernetes"], ["codex_local", "OpenAI", "kubernetes"], + ] as const)("connects an inherited %s draft through the saved %s account under %s policy", async (adapter, provider, policy) => { + const id = "00000000-0000-4000-8000-000000000093"; + const draft = { id, companyId: "company-1", name: "Atlas", role: "general", adapterType: "paperclip_runner", + adapterConfig: adapter === "claude_local" ? { provider: "acpx", acpxAgent: "claude" } : { provider: "codex" }, + runtimeConfig: {}, lifecycleState: "paused", pauseReason: "setup", defaultEnvironmentId: null, status: "paused" }; + state.params = new URLSearchParams({ name: "Atlas", adapterType: adapter, runner: "auto", setupAgentId: id }); + api.get.mockResolvedValue(draft); + api.update.mockImplementation(async (_id, patch) => ({ ...draft, ...patch })); + settings.get.mockResolvedValue({ defaultEnvironmentId: "remote" }); + settings.getExperimental.mockResolvedValue({ enableManagedSandboxOnly: policy !== "kubernetes" }); + settings.getGeneral.mockResolvedValue({ executionMode: policy === "kubernetes" ? "kubernetes" : "any" }); + envApi.list.mockResolvedValue([{ id: "remote", name: "Remote", status: "active", + driver: policy === "plugin" ? "plugin" : "sandbox", config: policy === "plugin" ? {} : { provider: policy === "sandbox" ? "daytona" : "kubernetes" } }]); + managedApi.list.mockResolvedValue({ currentUserId: "user-1", connections: [{ + id: "00000000-0000-4000-8000-000000000021", grantId: "00000000-0000-4000-8000-000000000022", + companyId: "company-1", provider: adapter === "claude_local" ? "anthropic" : "openai", method: "subscription", + name: "Saved subscription", ownership: "shared", isDefault: false, status: "connected", + }] } as any); + await renderCurrentParams(); + await click("AdvancedCustom Gateway"); + expect(container.querySelector('[role="combobox"][aria-label="Connection"]')).toBeTruthy(); + await click(`${provider}API key`); + expect(container.querySelector('[aria-label="API key"]')).toBeTruthy(); + await click(`${provider}Subscription`); + await click("Use saved subscription"); + expect(container.textContent).toContain("Configure your agent"); + expect(api.testEnvironment).toHaveBeenCalledExactlyOnceWith("company-1", adapter, expect.objectContaining({ agentId: id, environmentId: "remote" })); + await click("Finish setup"); + expect(api.hire).toHaveBeenCalledExactlyOnceWith("company-1", expect.objectContaining({ setupAgentId: id, defaultEnvironmentId: "remote" })); + expect(api.create).not.toHaveBeenCalled(); + }); + it("retains the selected account while a company requires approval before preparing the computer", async () => { settings.get.mockResolvedValue({ defaultEnvironmentId: "computer" }); envApi.list.mockResolvedValue([{ id: "computer", name: "Agent computer", status: "active", driver: "plugin", config: {} }]); @@ -92,13 +128,14 @@ describe("New agent setup", () => { expect(api.hire).toHaveBeenCalledWith("company-1", expect.objectContaining({ setupAgentId: id, defaultEnvironmentId: "computer" })); }); - it("waits for background preparation and tests the same saved computer after it completes", async () => { + it.each([null, "computer"])("waits for background preparation and tests the same saved computer (environment=%s)", async (defaultEnvironmentId) => { const id = "00000000-0000-4000-8000-000000000092"; const draft = { id, companyId: "company-1", name: "Atlas", adapterType: "paperclip_runner", adapterConfig: { provider: "codex" }, - runtimeConfig: {}, lifecycleState: "preparing", pauseReason: "setup", defaultEnvironmentId: "computer", status: "paused" }; + runtimeConfig: {}, lifecycleState: "preparing", pauseReason: "setup", defaultEnvironmentId, status: "paused" }; state.params = new URLSearchParams({ name: "Atlas", adapterType: "codex_local", setupAgentId: id }); api.get.mockResolvedValue(draft); settings.get.mockResolvedValue({ defaultEnvironmentId: "computer" }); + settings.getExperimental.mockResolvedValue({ enableManagedSandboxOnly: true }); envApi.list.mockResolvedValue([{ id: "computer", name: "Agent computer", status: "active", driver: "plugin", config: {} }]); await renderCurrentParams(); expect(container.textContent).toContain("Preparing your agent computer"); From 5785090ff17e3a0537b2022333ee74ff786e0cf2 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:26:54 -0500 Subject: [PATCH 23/41] fix(server): wake archive notification delivery from committed changes (#17475) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Auto-archive writes an existing notification outbox in its archive transaction. > - The maintenance sweep polls that outbox even when it is empty. > - The committing producer can wake delivery. Saved rows and claim timestamps can restore unfinished work. > - This PR separates delivery from retention scanning and uses the existing coordinator for retries and deadlines. > - Settled delivery becomes quiet while archive and notification behavior remain intact. ## Linked Issues or Issue Description **What happened?** The retention maintenance sweep calls archive notification delivery on every tick. An empty outbox and retained delivered history still cause database work. **What did you expect?** Committed auto-archive should wake delivery. Pending work and existing claims should retain retry or expiry deadlines. Settled history should cause no recurring delivery SQL. **Steps to reproduce** Start a configured instance with an empty or fully delivered archive outbox. Leave it idle. The old maintenance tick still checks the outbox. This is slice 08 of the sleep migration. It remains stacked on projection repair #17468. Retention eligibility and scanning, decision TTL expiry and decision continuation recovery have separate owners and PRs. ## What Changed - Notify archive delivery before the existing outbox insert in the auto-archive transaction. Preserve outer transaction commit, rollback and uncertainty handling. - Register a delivery worker with the existing coordinator. Remove notification delivery from retention sweeps. - Keep claim recovery, selection, company/agent grouping and timestamp-fenced settlement in the delivery service. Keep the existing service API and payload. - Count a notified group only when it settles rows. Use one timestamp for failure reset. - Add generated concurrent migration `0387_boring_professor_monster.sql`. Its partial index matches pending order: `last_attempt_at NULLS FIRST, created_at, id`. Add no table. - Test real startup registration and prove a retention tick does not send notifications. Preserve the disabled-scheduler no-notification contract. ## Verification Current candidate: `457c70fc6acbd681fcccd90d08b6391efb202bce`, normally merged with pinned master `8161f2a813` after projection #17468 landed. All 39 independent files from that master integration are preserved exactly. Archive owner behavior and migration 0387 are unchanged. - **58 archive/startup/scoped-deadline cases pass** after master reconciliation. Server typecheck and build pass. The preceding projection-parent integration passed all **239** gateway/projection/archive/startup/deadline cases. - All 385 parent journal entries remain an exact prefix, with own migration 0387 appended; the five newest generated snapshots remain. Generated index apply, replay, schema drift and real pending-order query measurements passed in the prior owner verification. - Failure accounting: the broader local command also attempted six upstream heartbeat-session-preparation and 55 native control-plane cases. The embedded PostgreSQL runtime was unavailable: the six heartbeat cases skipped, and native setup failed before its 55 cases could execute. This is an incomplete additional local check, not a passing full run. The owning 58 cases passed with the existing external fixture where applicable. CI must exercise its supported embedded environment. - The first independent-master comparison used the shared mutable remote ref, which another worktree advanced during verification. Pinning the actual merged master SHA confirms all 39 independent files match exactly. No incoming source change was dropped. The behavior audit in `doc/idle-decision-archive-delivery.md` records earlier fixture failures, the stale-claim regression and the index probe. Local tests do not prove deployed suspension or fleet savings. Fresh current-head CI and both reviews remain required. ## Risks - Delivery remains at least once. A wake accepted before an acknowledgement is lost can be retried. - Each pass keeps the existing 500-row limit. Failed or remaining pending work retains the existing thirty-second retry. Saved delivery claims retain their five-minute deadline. - Auto-archive is the existing outbox producer. Manual archive/revive and reviewed-manifest actions do not gain notifications. - Signals use the existing single-active-writer database owner. Independent SQL or another active server needs a wake contract. - Concurrent index creation uses the existing top-level executor and invalid-index retry path. Existing SQL and migration receipts are unchanged. - Cloud sleep guards and external-wake coverage do not change. Local worker quiet is not proof of database or host suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge The final projection efficiency parent is integrated normally. All 239 affected gateway/projection, archive, startup and scoped-deadline cases pass, with server typecheck/build. The archive owner behavior and index remain unchanged. Fresh reviews and CI are required on this parent integration. --------- Co-authored-by: Paperclip --- doc/idle-decision-archive-delivery.md | 127 ++++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 26 ++- packages/db/src/client.ts | 2 + .../0387_boring_professor_monster.sql | 1 + ...{0382_snapshot.json => 0387_snapshot.json} | 113 +++++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/decision_queues.ts | 3 + .../decision-retention-service.test.ts | 195 +++++++++++++++++- .../server-startup-feedback-export.test.ts | 39 +++- server/src/index.ts | 7 +- .../decision-archive-delivery-work.ts | 17 ++ .../src/services/decision-archive-delivery.ts | 115 +++++++++++ server/src/services/decision-retention.ts | 94 +-------- .../services/delivery-work-notifications.ts | 2 + 14 files changed, 649 insertions(+), 99 deletions(-) create mode 100644 doc/idle-decision-archive-delivery.md create mode 100644 packages/db/src/migrations/0387_boring_professor_monster.sql rename packages/db/src/migrations/meta/{0382_snapshot.json => 0387_snapshot.json} (99%) create mode 100644 server/src/services/decision-archive-delivery-work.ts create mode 100644 server/src/services/decision-archive-delivery.ts diff --git a/doc/idle-decision-archive-delivery.md b/doc/idle-decision-archive-delivery.md new file mode 100644 index 00000000000..4378bb00847 --- /dev/null +++ b/doc/idle-decision-archive-delivery.md @@ -0,0 +1,127 @@ +# Decision-archive notification delivery + +Auto-archive already writes `decision_archive_notification_outbox` in the same +transaction that archives its retention row. The old retention sweep called +notification delivery on every maintenance tick, including an empty outbox. +Delivery is now its own coordinator worker. Retention scanning remains a separate +migration; this PR does not change what is archived or when. + +## Ownership and producer audit + +`decision-retention.ts` owns eligibility, current-version checks, source authority, +origin resolution, and archive writes. Its sole outbox producer is `autoArchive`. +It now awaits the delivery notification immediately before inserting the outbox +row in the existing transaction. Caller-owned transactions and savepoints retain +outer commit/rollback ownership through the shared database signal mechanism. +Manual archive/revive and reviewed-manifest archive do not create these +notifications today; this change does not add notifications to those actions. + +`decision-archive-delivery.ts` owns claim recovery, selection, grouping, delivery, +and settlement. `decisionRetentionService.deliverNotifications` remains a +compatible delegate. Its notification callback and batch type retain their +existing contract. The scheduler only sees the delivery operation and next +work deadline. Claim and acknowledgement writes belong to this worker; they do +not broadcast self-generated wakes. No table is added. A concurrent partial index supports the pending delivery +order on the existing outbox. + +The server registers the worker in enabled-heartbeat startup housekeeping and +removes notification delivery from `runRetentionSweep`. Registration scans saved +work before the retention pass can enqueue more. The disabled-heartbeat branch +continues not to dispatch agent notifications. A service without a callback +leaves outbox rows intact and does not attempt notification delivery. + +## Work and deadlines + +Pending outbox rows are due for delivery. Each pass claims up to 500 rows and +groups them by company and origin agent, preserving company scope and the +existing notification payload. Failed or remaining pending rows retain a +30-second retry. Never-attempted rows sort first, then the oldest attempts, so +one failing batch cannot repeatedly hide later rows. + +A delivering row retains its existing five-minute claim deadline. Startup +reconstructs that deadline from `lastAttemptAt`; a legacy missing timestamp is +immediately recoverable. A worker waiting only for a future claim deadline does +not repeatedly query until that deadline. Settlement and failure reset require +the same claim timestamp, so a delayed old result cannot settle or release a +replacement claim. Delivered counts include only rows actually settled. + +An empty outbox or retained delivered history has no worker deadline and no +recurring SQL. The coordinator coalesces writes and prevents overlapping passes. +The shared deadline worker preserves first enqueue during drain, changes during +delivery, and failed retries across drain/resume. Coordinator shutdown joins +in-flight work. These guarantees do not replace the durable rows. + +## Recovery and limits + +A failed enqueue rolls back both the archive mutation and its outbox insertion. +An uncertain transaction retains the shared intent until its outcome is known. +A notification failure returns the same rows to pending. A process stopping +while delivering leaves the claim for deadline recovery on restart. + +Delivery remains at least once. A wake accepted before a lost acknowledgement +can be retried. This change does not promise exactly-once agent wakeups or add +new deduplication storage. The five-minute stale threshold is unchanged. + +Supported producers share the process-local database signal owner. Independent +SQL or another active server needs a separate wake contract. The existing Cloud +sleep proof still blocks this outbox conservatively; no external-wake coverage +or sleep guard is relaxed. Worker quiet does not prove actual database or host +suspension. Retention scanning and decision continuations remain separate duties. + +## Verification + +The original migration selection passed 54 tests: 15 real-PostgreSQL retention/delivery cases +and 39 startup cases. Twelve new cases cover configured idle, retained delivered +history, real enqueue, restart, exact stale-claim timing, legacy missing timestamps, +first enqueue during drain, failed-delivery retry through drain, outer commit and +rollback, enqueue failure rollback, writes during delivery, more than 500 rows, +and stale-result fencing. Quiet assertions observe no reads or writes across an +hour of controlled time. Workspace typecheck and build pass. + +The first 52-case selection passed. Adding fault/concurrency tests then exposed +two fixture errors: duplicate globally unique issue identifiers across two seeded +companies, and asserting quiet before rollback reconciliation had run. Seeded +identifiers now use their company prefix; quiet checks advance the retained retry +and await authoritative settlement. All 54 cases passed on rerun. No production +hold, recovery deadline, or timeout was weakened. Full repository CI and fresh +review are required before this PR is ready. + + +## Review repairs: startup, claim results and backlog selection + +Startup regressions now exercise `startServer` itself. Enabled startup must +register `decisionArchiveDelivery` and invoke the delivery owner; calling the +captured heartbeat interval still performs retention reads without another +notification delivery. Calling the registered worker invokes delivery again. +The disabled-heartbeat case explicitly preserves the existing no-notification +contract. An initial expanded test incorrectly expected that disabled mode to +register the worker and failed; inspection confirmed the documented disabled +contract, so the test now asserts it rather than enabling new background work. + +The real stale-claim regression first returned `notifiedAgents: 1` after the +replacement claim prevented any settlement. The counter now increments only +for a group that actually settles rows, while the existing timestamp fence and +at-least-once delivery contract remain intact. Failure reset uses one timestamp +for `lastAttemptAt` and `updatedAt`. + +Generated migration `0387_boring_professor_monster.sql` creates a concurrent +partial index on `(last_attempt_at NULLS FIRST, created_at, id)` for pending rows. +The existing top-level migration executor handles concurrent creation and +invalid-index recovery. No history migration or existing journal entry changes; +the generated snapshots retain the five newest versions. + +A real migration and query-plan check used a separate database with 50,000 +outbox rows, including 30,000 pending. The existing claim-selection query returned +500 rows in both cases. Without the new index PostgreSQL sorted the pending +backlog with a top-N heapsort and touched 944 shared buffers; with the index it +used an index-only scan without sorting and touched 505 buffers. The baseline +index removal was rolled back inside that dedicated database, and the migrated +index was confirmed valid afterwards. These local measurements establish the +access path, not production latency or cost savings. + +All 58 cases across real PostgreSQL delivery/retention, startup and the shared +deadline worker pass. Generated snapshot drift, migration numbering/safety, +server types and build pass. The earlier stale-counter failure and invalid +scheduler-disabled expectation are retained above. The existing 500-row batch +limit and thirty-second retry for remaining pending work are unchanged. Fresh +current-head CI, Greptile and Ponytail remain required before merge. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 25d3ebf70bc..2345ab9bb90 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -15,7 +15,7 @@ Do not mark an item done from a timer removal or a green check alone. | 01 | Custom-image setup expiry | Session writes, existing expiry, startup/retry/drain | [#17443](https://github.com/paperclipai/paperclip/pull/17443), merged after CI and passing reviews | | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | -| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Pending | +| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; notification delivery implemented in slice 08, PR pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | | 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), merged after current-head CI and both 5/5 reviews; interrupted claims: [#17458](https://github.com/paperclipai/paperclip/pull/17458), merged after current-head CI and both 5/5 reviews; projection: [#17468](https://github.com/paperclipai/paperclip/pull/17468), interaction-link index verified and awaiting fresh CI/review | @@ -335,3 +335,27 @@ gateway/startup/deadline selection passes 222 cases; server types/build pass. The initial invalid one-wake-per-approval assertion and corrected isolated-hint measurement are retained in the behavior document. Current-head CI/reviews remain required before this slice can merge. + + +## Slice 08: decision-archive notification delivery + +Projection repair is #17468. Its bounded master integration passes local +acceptance and awaits fresh checks and review. Slice 08 builds on that branch +and separates archive notification delivery from retention scanning. +The existing auto-archive transaction is the sole outbox producer. A committed +enqueue wakes delivery; pending rows retain retries and delivering rows retain +their saved five-minute claim deadline. Delivered history is quiet. + +The implementation preserves the delivery API and payload. It fences stale +completion by claim timestamp and orders unattempted/oldest-attempted pending +rows so failed batches do not starve later work. No table is added. Generated +migration `0387_boring_professor_monster.sql` adds the concurrent partial index +for pending delivery order. See [archive delivery](../idle-decision-archive-delivery.md). + +All 58 selected tests pass: 15 real-PostgreSQL retention/delivery cases, 42 startup +cases and the shared deadline worker case. Migration checks, generated snapshot +drift, server typecheck and build pass. The initial fixture failures, stale-claim +counter regression and corrected disabled-mode expectation are recorded in the +behavior document. Full current-head CI and both reviews remain required. +Retention eligibility/scanning, decision expiry, effect recovery, and decision +continuation delivery remain independent unfinished reasons in the full ledger. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 342256ac462..1d98d929d7a 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -505,6 +505,8 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0382_narrow_the_executioner.sql", "0385_open_madame_web.sql", "0386_steady_epoch.sql", + "0387_boring_professor_monster.sql", + "0384_funny_adam_warlock.sql", "0383_tool_review_company_expiry.sql", ]); diff --git a/packages/db/src/migrations/0387_boring_professor_monster.sql b/packages/db/src/migrations/0387_boring_professor_monster.sql new file mode 100644 index 00000000000..9398f9e44bc --- /dev/null +++ b/packages/db/src/migrations/0387_boring_professor_monster.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "decision_archive_notification_outbox_delivery_idx" ON "decision_archive_notification_outbox" USING btree ("last_attempt_at" NULLS FIRST,"created_at","id") WHERE "decision_archive_notification_outbox"."status" = 'pending'; diff --git a/packages/db/src/migrations/meta/0382_snapshot.json b/packages/db/src/migrations/meta/0387_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0382_snapshot.json rename to packages/db/src/migrations/meta/0387_snapshot.json index 6c27dd9aa22..7fd3f6cbe93 100644 --- a/packages/db/src/migrations/meta/0382_snapshot.json +++ b/packages/db/src/migrations/meta/0387_snapshot.json @@ -1,6 +1,6 @@ { - "id": "ecec138e-5312-4ad6-b051-bd4ee9ed4e34", - "prevId": "2e93deb7-f24c-42d1-8c8e-30b21cff90bc", + "id": "de27fdf8-430a-443a-9502-725f92c48892", + "prevId": "da3229b5-4eab-4d8b-ba6b-d756e6c42430", "version": "7", "dialect": "postgresql", "tables": { @@ -22110,6 +22110,34 @@ "concurrently": false, "method": "btree", "with": {} + }, + "decision_archive_notification_outbox_delivery_idx": { + "name": "decision_archive_notification_outbox_delivery_idx", + "columns": [ + { + "expression": "last_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "first" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decision_archive_notification_outbox\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -52834,6 +52862,66 @@ "method": "btree", "with": {} }, + "tool_action_requests_company_pending_expiry_idx": { + "name": "tool_action_requests_company_pending_expiry_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_executing_deadline_idx": { + "name": "tool_action_requests_executing_deadline_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'executing'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_approved_idx": { + "name": "tool_action_requests_approved_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'approved'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_invocation_idx": { "name": "tool_action_requests_invocation_idx", "columns": [ @@ -52849,6 +52937,27 @@ "method": "btree", "with": {} }, + "tool_action_requests_company_interaction_idx": { + "name": "tool_action_requests_company_interaction_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "interaction_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_issue_idx": { "name": "tool_action_requests_issue_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 578d8027ed9..4e9bf240449 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2696,6 +2696,13 @@ "when": 1793135146893, "tag": "0386_steady_epoch", "breakpoints": true + }, + { + "idx": 387, + "version": "7", + "when": 1793136115698, + "tag": "0387_boring_professor_monster", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/decision_queues.ts b/packages/db/src/schema/decision_queues.ts index 667c67f8210..12215c34652 100644 --- a/packages/db/src/schema/decision_queues.ts +++ b/packages/db/src/schema/decision_queues.ts @@ -248,6 +248,9 @@ export const decisionArchiveNotificationOutbox = pgTable( table.originAgentId, ), pendingIdx: index("decision_archive_notification_outbox_pending_idx").on(table.status, table.createdAt), + pendingDeliveryIdx: index("decision_archive_notification_outbox_delivery_idx") + .on(table.lastAttemptAt.asc().nullsFirst(), table.createdAt, table.id) + .where(sql`${table.status} = 'pending'`), statusCheck: check( "decision_archive_notification_outbox_status_check", sql`${table.status} IN ('pending', 'delivering', 'delivered')`, diff --git a/server/src/__tests__/decision-retention-service.test.ts b/server/src/__tests__/decision-retention-service.test.ts index 07be9a27859..d03796a5b3e 100644 --- a/server/src/__tests__/decision-retention-service.test.ts +++ b/server/src/__tests__/decision-retention-service.test.ts @@ -1,5 +1,10 @@ +import { sql } from "drizzle-orm"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { registerDecisionArchiveDeliveryWork } from "../services/decision-archive-delivery-work.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; import { randomUUID } from "node:crypto"; -import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { activityLog, agents, @@ -13,19 +18,23 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro import { attentionService } from "../services/attention.js"; import { decisionRetentionService } from "../services/decision-retention.js"; -const support = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_RETENTION_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describePg = support.supported ? describe : describe.skip; describePg("decision retention", () => { let tempDb: Awaited> | null = null; let db: ReturnType; + let stopWorker: (() => Promise) | undefined; + beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-decision-retention-"); - db = createDb(tempDb.connectionString); + if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-decision-retention-"); + db = createDb(externalUrl ?? tempDb!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); }, 30_000); afterEach(async () => { + await stopWorker?.(); stopWorker = undefined; await db.delete(decisionArchiveNotificationOutbox); await db.delete(decisionRetention); await db.delete(activityLog); @@ -34,7 +43,7 @@ describePg("decision retention", () => { await db.delete(companies); }); - afterAll(async () => tempDb?.cleanup()); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); async function seedReviewItems(count: number) { const companyId = randomUUID(); @@ -61,7 +70,7 @@ describePg("decision retention", () => { await db.insert(issues).values(issueIds.map((id, index) => ({ id, companyId, - identifier: `RET-${index + 1}`, + identifier: `R${companyId.slice(0, 6)}-${index + 1}`, title: `Old review ${index + 1}`, status: "in_review", priority: "medium", @@ -90,6 +99,180 @@ describePg("decision retention", () => { responsibleUserId: "board-user", }; + describe("archive delivery worker", () => { + let coordinator: ReturnType | undefined; + let fixture: Awaited>; + let service: ReturnType; + let draining: boolean; + const errors: unknown[] = []; + const notify = vi.fn(async (_batch: unknown) => {}); + + beforeEach(async () => { + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + fixture = await seedReviewItems(1); draining = false; errors.length = 0; notify.mockReset(); + service = decisionRetentionService(db, { notifyOriginAgent: notify }); + stopWorker = async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.execute(sql`drop trigger if exists archive_delivery_fail on decision_archive_notification_outbox`); + await db.execute(sql`drop function if exists archive_delivery_fail()`); + expect(errors).toEqual([]); + }; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerDecisionArchiveDeliveryWork(coordinator, service, () => !draining).ready; + } + async function archive(source = db) { + const feed = await attentionService(source).list(fixture.companyId, { limit: 100 }); + return decisionRetentionService(source).autoArchive({ companyId: fixture.companyId, items: feed.items }); + } + async function settle(ms = 60_000) { + await vi.advanceTimersByTimeAsync(ms); + if (!draining) await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0), { timeout: 5000 }); + } + async function quiet() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + const reads = vi.spyOn(db, "select"), executes = vi.spyOn(db, "execute"), writes = vi.spyOn(db, "update"); + await settle(60 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); executes.mockRestore(); writes.mockRestore(); + } + async function delivered() { + const rows = await db.select().from(decisionArchiveNotificationOutbox); + expect(rows.length).toBeGreaterThan(0); expect(rows.every(row => row.status === "delivered")).toBe(true); + } + + it("is quiet with configuration, wakes from real auto-archive, and leaves delivered history quiet", async () => { + await start(); await quiet(); + expect(await archive()).toBe(1); await settle(); await delivered(); await quiet(); + expect(notify).toHaveBeenCalledTimes(1); + expect(notify.mock.calls[0][0]).toMatchObject({ companyId: fixture.companyId, agentId: fixture.agentId, + items: [expect.objectContaining({ issueId: fixture.issueIds[0], archiveVersion: 1 })] }); + }); + + it("reconstructs pending delivery after restart", async () => { + await archive(); await start(); await quiet(); await delivered(); expect(notify).toHaveBeenCalledTimes(1); + }); + + it("waits until an existing delivery claim expires", async () => { + await archive(); const attemptedAt = new Date(); + await db.update(decisionArchiveNotificationOutbox).set({ status: "delivering", lastAttemptAt: attemptedAt, attemptCount: 1 }); + await start(); expect(notify).not.toHaveBeenCalled(); + expect(coordinator!.nextWakeAt()).toBe(attemptedAt.getTime() + 5 * 60_000); + const reads = vi.spyOn(db, "select"); await settle(4 * 60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await settle(2 * 60_000); await delivered(); await quiet(); expect(notify).toHaveBeenCalledTimes(1); + }); + + it("recovers a legacy delivering row with no attempt timestamp", async () => { + await archive(); await db.update(decisionArchiveNotificationOutbox).set({ status: "delivering" }); + await start(); await delivered(); await quiet(); + }); + + it("retains first enqueue during drain without passive SQL", async () => { + await start(); await quiet(); draining = true; + await archive(); await settle(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"); + await settle(60 * 60_000); expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); expect(notify).not.toHaveBeenCalled(); + draining = false; await settle(); await delivered(); await quiet(); + }); + + it("retries failed delivery through drain and then becomes quiet", async () => { + await archive(); notify.mockRejectedValueOnce(new Error("wake unavailable")); + await start(); expect(notify).toHaveBeenCalledTimes(1); expect(coordinator!.nextWakeAt()).not.toBeNull(); + draining = true; await settle(60 * 60_000); expect(notify).toHaveBeenCalledTimes(1); + draining = false; + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); await delivered(); }, { timeout: 5000 }); + await quiet(); expect(notify).toHaveBeenCalledTimes(2); + }); + + it.each([false, true])("waits for the archive's outer transaction (rollback %s)", async rollback => { + await start(); await quiet(); + let entered!: () => void, release!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const aborted = new Error("fixture rollback"); + const running = db.transaction(async tx => { + await archive(tx as unknown as typeof db); entered(); await gate; if (rollback) throw aborted; + }).catch(error => { if (error !== aborted) throw error; }); + await ready; + try { await vi.advanceTimersByTimeAsync(60_000); expect(notify).not.toHaveBeenCalled(); } + finally { release(); await running; } + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); expect(idleWorkSnapshot().active).toBe(0); + if (rollback) expect(await db.select().from(decisionArchiveNotificationOutbox)).toEqual([]); + else await delivered(); + }, { timeout: 5000 }); + await quiet(); expect(notify).toHaveBeenCalledTimes(rollback ? 0 : 1); + }); + + it("rolls back archive state when enqueue fails and wakes on a later successful archive", async () => { + await start(); await quiet(); + await db.execute(sql`create function archive_delivery_fail() returns trigger language plpgsql as $$ + begin raise exception 'archive enqueue fixture failure'; end $$`); + await db.execute(sql`create trigger archive_delivery_fail before insert on decision_archive_notification_outbox + for each row execute function archive_delivery_fail()`); + await expect(archive()).rejects.toThrow(); + await db.execute(sql`drop trigger archive_delivery_fail on decision_archive_notification_outbox`); + await quiet(); expect(notify).not.toHaveBeenCalled(); + expect(await db.select().from(decisionArchiveNotificationOutbox)).toEqual([]); + expect((await db.select().from(decisionRetention))[0].archivedAt).toBeNull(); + await archive(); await settle(); await delivered(); await quiet(); + }); + + it("keeps an enqueue committed during delivery and drains it on the next pass", async () => { + await start(); await quiet(); + let entered!: () => void, release!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + notify.mockImplementationOnce(async () => { entered(); await gate; }); + await archive(); + await vi.advanceTimersByTimeAsync(30_000); await ready; + try { + fixture = await seedReviewItems(1); await archive(); + expect(notify).toHaveBeenCalledTimes(1); + } finally { release(); } + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); await delivered(); }, { timeout: 5000 }); + await quiet(); expect(notify).toHaveBeenCalledTimes(2); + }); + + it("does not let an older failed batch starve rows beyond the batch limit", async () => { + await archive(); const [row] = await db.select().from(decisionArchiveNotificationOutbox); + await db.insert(decisionArchiveNotificationOutbox).values(Array.from({ length: 501 }, (_, index) => ({ + ...row, id: randomUUID(), sourceId: `additional-${index}` }))); + notify.mockRejectedValueOnce(new Error("first batch unavailable")); + await start(); expect(notify.mock.calls[0][0]).toMatchObject({ items: expect.any(Array) }); + expect((notify.mock.calls[0][0] as { items: unknown[] }).items).toHaveLength(500); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); await delivered(); }, { timeout: 5000 }); + await quiet(); + expect((notify.mock.calls[1][0] as { items: unknown[] }).items).toHaveLength(500); + expect((notify.mock.calls[2][0] as { items: unknown[] }).items).toHaveLength(2); + }); + + it("does not settle a replacement claim from an older delivery result", async () => { + await archive(); + let entered!: () => void, release!: () => void; + const ready = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + notify.mockImplementationOnce(async () => { entered(); await gate; }); + const first = service.deliverNotifications(); await ready; + const replacementAt = new Date(Date.now() + 6 * 60_000); + try { await db.update(decisionArchiveNotificationOutbox).set({ lastAttemptAt: replacementAt, attemptCount: 2 }); } + finally { release(); } + expect(await first).toEqual({ notifiedAgents: 0, delivered: 0 }); + expect((await db.select().from(decisionArchiveNotificationOutbox))[0]).toMatchObject({ status: "delivering", lastAttemptAt: replacementAt }); + await start(); expect(coordinator!.nextWakeAt()).toBe(replacementAt.getTime() + 5 * 60_000); + await settle(12 * 60_000); await delivered(); await quiet(); + }); + }); + it("auto-archives, excludes by default, returns archived rows, and revives", async () => { const { companyId, issueIds } = await seedReviewItems(1); const now = new Date("2026-08-02T00:00:00.000Z"); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index cb7da55e61a..2f454b226e2 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -6,8 +6,8 @@ import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; import * as idleLocalWork from "../services/idle-local-work.js"; import { idleBackupWakeMarker, markIdleBackupWakeRequired } from "../services/idle-database-backup.js"; import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; -import { reconcileBuiltInAgentsOnStartup, reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup } from "../services/index.js"; -import { runDatabaseBackup } from "@paperclipai/db"; +import { decisionRetentionService, reconcileBuiltInAgentsOnStartup, reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup } from "../services/index.js"; +import { companies, runDatabaseBackup, type Db } from "@paperclipai/db"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; import { gzipSync } from "node:zlib"; @@ -352,6 +352,7 @@ vi.mock("../services/index.js", () => ({ decisionRetentionService: vi.fn(() => ({ autoArchive: vi.fn(async () => 0), deliverNotifications: vi.fn(async () => ({ notifiedAgents: 0, delivered: 0 })), + nextArchiveNotificationAt: vi.fn(async () => null), })), feedbackService: feedbackServiceFactoryMock, bootstrapExecutionPolicyFromEnv: vi.fn(async () => null), @@ -521,6 +522,40 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true])("registers archive delivery separately from retention sweeps (scheduler %s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled, heartbeatSchedulerIntervalMs: 30000 })); + let tick: (() => void) | undefined; + const timer = vi.spyOn(globalThis, "setInterval").mockImplementation(((callback: () => void, interval: number) => { + if (interval === 30000) tick = callback; + return 1 as unknown as ReturnType; + }) as typeof setInterval); + try { + await startServer(); await settleStartupHousekeeping(); + const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.decisionArchiveDelivery); + if (!enabled) { + expect(registration).toBeUndefined(); + expect(decisionRetentionService).not.toHaveBeenCalled(); + expect(tick).toBeDefined(); tick!(); await settleStartupHousekeeping(); + expect(decisionRetentionService).not.toHaveBeenCalled(); + return; + } + expect(registration).toBeDefined(); + const retention = vi.mocked(decisionRetentionService).mock.results[0]!.value; + expect(retention.deliverNotifications).toHaveBeenCalledOnce(); + expect(retention.nextArchiveNotificationAt).toHaveBeenCalledOnce(); + const db = createDbMock.mock.results[0]!.value as Db; + const companyReads = () => vi.mocked(db.select).mock.results.filter(({ value }) => + vi.mocked(value.from).mock.calls.some(([table]) => table === companies)).length; + const before = companyReads(); + expect(before).toBeGreaterThan(0); expect(tick).toBeDefined(); + tick!(); await settleStartupHousekeeping(); + expect(companyReads()).toBeGreaterThan(before); + expect(retention.deliverNotifications).toHaveBeenCalledOnce(); + await registration![1].run(new AbortController().signal); + expect(retention.deliverNotifications).toHaveBeenCalledTimes(2); + } finally { timer.mockRestore(); } + }); + it("registers durable case handoffs independently of the heartbeat scheduler", async () => { await startServer(); expect(caseHandoffDeliveryServiceFactoryMock).toHaveBeenCalledWith(createDbMock.mock.results[0]!.value, { heartbeat: heartbeatServiceMock }); diff --git a/server/src/index.ts b/server/src/index.ts index bef68b441e7..37661c37c0e 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,5 @@ +import { registerDecisionArchiveDeliveryWork } from "./services/decision-archive-delivery-work.js"; + import { registerToolReviewProjectionWork } from "./services/tool-review-projection-work.js"; import { registerToolReviewInterruptionWork } from "./services/tool-review-interruption-work.js"; import { registerToolReviewRecoveryWork } from "./services/tool-review-recovery-work.js"; @@ -1809,8 +1811,7 @@ async function startServerWithDatabaseTeardown( }); archived += await retentionExecutor.autoArchive({ companyId: company.id, items: page.items }); } - const notifications = await retentionExecutor.deliverNotifications(); - return { archived, ...notifications }; + return { archived }; }; // Startup housekeeping: provider and connection sweeps that every timer tick // repeats and that neither serving requests nor run recovery depends on. @@ -1843,6 +1844,8 @@ async function startServerWithDatabaseTeardown( // Retry any orphan sandbox teardown left by a failed acquire before a // server restart, so a leaked sandbox does not stay allocated. await step("environment_lease_cleanup", startEnvironmentCleanup); + await step("archive_notification_worker", () => registerDecisionArchiveDeliveryWork(deliveryWork, retentionExecutor, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("retention_sweep", runRetentionSweep); }; diff --git a/server/src/services/decision-archive-delivery-work.ts b/server/src/services/decision-archive-delivery-work.ts new file mode 100644 index 00000000000..7b75cd921be --- /dev/null +++ b/server/src/services/decision-archive-delivery-work.ts @@ -0,0 +1,17 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import { DECISION_ARCHIVE_RETRY_MS } from "./decision-archive-delivery.js"; +import type { decisionRetentionService } from "./decision-retention.js"; + +export function registerDecisionArchiveDeliveryWork( + coordinator: ReturnType, + service: Pick, "deliverNotifications" | "nextArchiveNotificationAt">, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.decisionArchiveDelivery, { + retryMs: DECISION_ARCHIVE_RETRY_MS, canRun, + run: () => service.deliverNotifications(), + next: service.nextArchiveNotificationAt, + }); +} diff --git a/server/src/services/decision-archive-delivery.ts b/server/src/services/decision-archive-delivery.ts new file mode 100644 index 00000000000..df7bd7ee1f2 --- /dev/null +++ b/server/src/services/decision-archive-delivery.ts @@ -0,0 +1,115 @@ +import { and, asc, eq, inArray, isNull, lte, or, sql } from "drizzle-orm"; +import { decisionArchiveNotificationOutbox, type Db } from "@paperclipai/db"; +import type { AttentionSourceKind } from "@paperclipai/shared"; + +const CLAIM_MS = 5 * 60_000; +export const DECISION_ARCHIVE_RETRY_MS = 30_000; + +export type ArchiveNotificationBatch = { + companyId: string; + agentId: string; + items: Array<{ + sourceKind: AttentionSourceKind; + sourceId: string; + archiveVersion: number; + issueId: string; + }>; +}; + +export function decisionArchiveDeliveryService( + db: Db, + options: { notifyOriginAgent?: (batch: ArchiveNotificationBatch) => Promise }, +) { + async function deliverNotifications(limit = 500) { + if (!options.notifyOriginAgent) return { notifiedAgents: 0, delivered: 0 }; + const claimAt = new Date(); + const staleClaimAt = new Date(claimAt.getTime() - CLAIM_MS); + await db.update(decisionArchiveNotificationOutbox).set({ status: "pending", updatedAt: claimAt }) + .where(and( + eq(decisionArchiveNotificationOutbox.status, "delivering"), + or(isNull(decisionArchiveNotificationOutbox.lastAttemptAt), lte(decisionArchiveNotificationOutbox.lastAttemptAt, staleClaimAt)), + )); + const rows = await db.transaction(async (tx) => { + const pendingIds = await tx.select({ id: decisionArchiveNotificationOutbox.id }) + .from(decisionArchiveNotificationOutbox) + .where(eq(decisionArchiveNotificationOutbox.status, "pending")) + .orderBy(sql`${decisionArchiveNotificationOutbox.lastAttemptAt} asc nulls first`, + asc(decisionArchiveNotificationOutbox.createdAt), asc(decisionArchiveNotificationOutbox.id)) + .limit(limit) + .then((values) => values.map((value) => value.id)); + if (pendingIds.length === 0) return []; + return tx.update(decisionArchiveNotificationOutbox).set({ + status: "delivering", + lastAttemptAt: claimAt, + attemptCount: sql`${decisionArchiveNotificationOutbox.attemptCount} + 1`, + updatedAt: claimAt, + }).where(and( + inArray(decisionArchiveNotificationOutbox.id, pendingIds), + eq(decisionArchiveNotificationOutbox.status, "pending"), + )).returning(); + }); + const groups = new Map(); + for (const row of rows) { + const key = `${row.companyId}:${row.originAgentId}`; + const grouped = groups.get(key) ?? []; + grouped.push(row); + groups.set(key, grouped); + } + let notifiedAgents = 0; + let delivered = 0; + for (const grouped of groups.values()) { + const first = grouped[0]!; + try { + await options.notifyOriginAgent({ + companyId: first.companyId, + agentId: first.originAgentId, + items: grouped.map((row) => ({ + sourceKind: row.sourceKind as AttentionSourceKind, + sourceId: row.sourceId, + archiveVersion: row.archiveVersion, + issueId: row.originIssueId, + })), + }); + const deliveredAt = new Date(); + const settled = await db.update(decisionArchiveNotificationOutbox).set({ + status: "delivered", + deliveredAt, + lastAttemptAt: deliveredAt, + updatedAt: deliveredAt, + }).where(and( + inArray(decisionArchiveNotificationOutbox.id, grouped.map((row) => row.id)), + eq(decisionArchiveNotificationOutbox.status, "delivering"), + eq(decisionArchiveNotificationOutbox.lastAttemptAt, claimAt), + )).returning({ id: decisionArchiveNotificationOutbox.id }); + if (settled.length > 0) notifiedAgents += 1; + delivered += settled.length; + } catch { + const failedAt = new Date(); + await db.update(decisionArchiveNotificationOutbox).set({ + status: "pending", + lastAttemptAt: failedAt, + updatedAt: failedAt, + }).where(and( + inArray(decisionArchiveNotificationOutbox.id, grouped.map((row) => row.id)), + eq(decisionArchiveNotificationOutbox.status, "delivering"), + eq(decisionArchiveNotificationOutbox.lastAttemptAt, claimAt), + )); + } + } + return { notifiedAgents, delivered }; + } + + async function next() { + if (!options.notifyOriginAgent) return null; + const [pending] = await db.select({ id: decisionArchiveNotificationOutbox.id }) + .from(decisionArchiveNotificationOutbox).where(eq(decisionArchiveNotificationOutbox.status, "pending")).limit(1); + const [claimed] = await db.select({ attemptedAt: decisionArchiveNotificationOutbox.lastAttemptAt }) + .from(decisionArchiveNotificationOutbox).where(eq(decisionArchiveNotificationOutbox.status, "delivering")) + .orderBy(sql`${decisionArchiveNotificationOutbox.lastAttemptAt} asc nulls first`).limit(1); + const pendingAt = pending ? Date.now() + DECISION_ARCHIVE_RETRY_MS : null; + const claimAt = claimed ? (claimed.attemptedAt?.getTime() ?? Date.now() - CLAIM_MS) + CLAIM_MS : null; + return pendingAt === null ? claimAt : claimAt === null ? pendingAt : Math.min(pendingAt, claimAt); + } + + return { deliver: deliverNotifications, next }; +} diff --git a/server/src/services/decision-retention.ts b/server/src/services/decision-retention.ts index 18616e44d71..b65e6e3bb77 100644 --- a/server/src/services/decision-retention.ts +++ b/server/src/services/decision-retention.ts @@ -1,5 +1,7 @@ +import { decisionArchiveDeliveryService, type ArchiveNotificationBatch } from "./decision-archive-delivery.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { createHash } from "node:crypto"; -import { and, asc, eq, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm"; +import { and, eq, inArray, isNotNull, isNull, lte, ne, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { agents, @@ -31,16 +33,7 @@ export const DEFAULT_DECISION_ARCHIVE_DAYS = 90; const DAY_MS = 86_400_000; export type DecisionRetentionState = typeof decisionRetention.$inferSelect; -export type ArchiveNotificationBatch = { - companyId: string; - agentId: string; - items: Array<{ - sourceKind: AttentionSourceKind; - sourceId: string; - archiveVersion: number; - issueId: string; - }>; -}; +export type { ArchiveNotificationBatch } from "./decision-archive-delivery.js"; function sourceKey(sourceKind: string, sourceId: string) { return `${sourceKind}:${sourceId}`; @@ -371,6 +364,7 @@ export function decisionRetentionService( archived += 1; const origin = await resolveOrigin(txDb, input.companyId, item.sourceKind, item.subject.id); if (origin) { + await notifyDeliveryWork(txDb, DELIVERY_QUEUES.decisionArchiveDelivery); await txDb.insert(decisionArchiveNotificationOutbox).values({ companyId: input.companyId, sourceKind: item.sourceKind, @@ -394,80 +388,7 @@ export function decisionRetentionService( return archived; } - async function deliverNotifications(limit = 500) { - if (!options.notifyOriginAgent) return { notifiedAgents: 0, delivered: 0 }; - const claimAt = new Date(); - const staleClaimAt = new Date(claimAt.getTime() - 5 * 60_000); - await db.update(decisionArchiveNotificationOutbox).set({ status: "pending", updatedAt: claimAt }) - .where(and( - eq(decisionArchiveNotificationOutbox.status, "delivering"), - or(isNull(decisionArchiveNotificationOutbox.lastAttemptAt), lte(decisionArchiveNotificationOutbox.lastAttemptAt, staleClaimAt)), - )); - const rows = await db.transaction(async (tx) => { - const pendingIds = await tx.select({ id: decisionArchiveNotificationOutbox.id }) - .from(decisionArchiveNotificationOutbox) - .where(eq(decisionArchiveNotificationOutbox.status, "pending")) - .orderBy(asc(decisionArchiveNotificationOutbox.createdAt), asc(decisionArchiveNotificationOutbox.id)) - .limit(limit) - .then((values) => values.map((value) => value.id)); - if (pendingIds.length === 0) return []; - return tx.update(decisionArchiveNotificationOutbox).set({ - status: "delivering", - lastAttemptAt: claimAt, - attemptCount: sql`${decisionArchiveNotificationOutbox.attemptCount} + 1`, - updatedAt: claimAt, - }).where(and( - inArray(decisionArchiveNotificationOutbox.id, pendingIds), - eq(decisionArchiveNotificationOutbox.status, "pending"), - )).returning(); - }); - const groups = new Map(); - for (const row of rows) { - const key = `${row.companyId}:${row.originAgentId}`; - const grouped = groups.get(key) ?? []; - grouped.push(row); - groups.set(key, grouped); - } - let notifiedAgents = 0; - let delivered = 0; - for (const grouped of groups.values()) { - const first = grouped[0]!; - try { - await options.notifyOriginAgent({ - companyId: first.companyId, - agentId: first.originAgentId, - items: grouped.map((row) => ({ - sourceKind: row.sourceKind as AttentionSourceKind, - sourceId: row.sourceId, - archiveVersion: row.archiveVersion, - issueId: row.originIssueId, - })), - }); - const deliveredAt = new Date(); - await db.update(decisionArchiveNotificationOutbox).set({ - status: "delivered", - deliveredAt, - lastAttemptAt: deliveredAt, - updatedAt: deliveredAt, - }).where(and( - inArray(decisionArchiveNotificationOutbox.id, grouped.map((row) => row.id)), - eq(decisionArchiveNotificationOutbox.status, "delivering"), - )); - notifiedAgents += 1; - delivered += grouped.length; - } catch { - await db.update(decisionArchiveNotificationOutbox).set({ - status: "pending", - lastAttemptAt: new Date(), - updatedAt: new Date(), - }).where(and( - inArray(decisionArchiveNotificationOutbox.id, grouped.map((row) => row.id)), - eq(decisionArchiveNotificationOutbox.status, "delivering"), - )); - } - } - return { notifiedAgents, delivered }; - } + const archiveDelivery = decisionArchiveDeliveryService(db, options); return { syncItems, @@ -477,6 +398,7 @@ export function decisionRetentionService( revive: (input: Omit[0], "archived">) => changeArchived({ ...input, archived: false }), archiveReviewedManifest, autoArchive, - deliverNotifications, + deliverNotifications: archiveDelivery.deliver, + nextArchiveNotificationAt: archiveDelivery.next, }; } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 8af5f57babe..230be70dcf8 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,8 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + decisionArchiveDelivery: "decision-archive-delivery", + toolReviewProjection: "tool-review-projection", toolReviewInterruption: "tool-review-interruption", toolReviewRecovery: "tool-review-recovery", From a63d19816f77667570d91e07dd4b5fcb72a5b083 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:44:51 -0500 Subject: [PATCH 24/41] fix(server): wake decision continuations from committed changes (#17478) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A completed decision can owe a wake to the agent that proposed it. > - The decision row already stores the pending continuation. > - The main maintenance tick scans this debt even when no continuation is ready. > - Committed changes can activate recovery and leave settled history quiet. > - This pull request gives continuation delivery its own worker and shares foreground delivery ownership. > - Pending work stays recoverable without an unconditional continuation scan. ## Linked Issues or Issue Description Targets master after #17475 merged. This is slice 09 of the recurring server work migration. Each PR owns one reason for background work. The remaining inventory is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Decision continuation delivery and recovery in the server. **Current behavior** Foreground paths send a wake and clear a pending flag. The combined decision sweep scans pending continuations on every maintenance tick. It can overlap foreground delivery. **Proposed behavior** Terminal decision writes notify a dedicated worker through the existing transaction boundary. Foreground and recovery calls share one in-flight delivery per decision and DB owner. Recovery re-reads readiness, pages past failures, and retries only unresolved debt or errors. **Reason and benefit** Open decisions, running effects, and delivered history no longer cause an independent continuation scan. No queue table is added. Decision expiry, target invalidation, interrupted effects, and retention remain separate duties. ## What Changed - Extract continuation readiness, delivery, and paged recovery into the decision owner. Preserve the wake payload and at-least-once crash behavior. - Signal the ten decision update sites before SQL in their existing transaction scope. Keep effects and wake calls outside those transactions. - Explicitly create the continuation obligation on all three archive-proposal failure settlements, including recovered running rows without inherited pending metadata. - Serialize concurrent decision effect execution through final settlement. Coalesce foreground and worker wakes. Re-read the durable flag before sending. Fence effect completion to running state so competing completion cannot restore acknowledged debt. - Expose an opaque DB work-scope key. Child pools and nested transactions share that identity; independent roots remain separate. Existing signal consumers keep their behavior. - Bound each recovery pass to 100 candidates, retain a fair cursor across failures, and stop between deliveries on shutdown. Forward the existing AbortSignal through the shared deadline adapter. - Register enabled-heartbeat continuation recovery before decision housekeeping. Remove only its independent scan from the main tick. Preserve the public combined sweep and disabled-heartbeat behavior. - Add generated migration `0388_ambitious_micromax.sql`, an idempotent concurrent partial index for pending continuations, and keep five generated snapshots. - Add real producer, transaction, failure, concurrency, paging, quiet, and shutdown tests. Document ownership and remaining limits in `doc/idle-decision-continuations.md`. ## Verification Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally integrated with landed archive squash `5785090ff1`. All 31 independent incoming master files are preserved exactly, including controller-authority recovery. All 97 decision/startup/deadline cases plus server types/build pass after the required Runner vendor rebuild. The preceding full five-suite selection passed 293 cases; the continuation owner behavior is unchanged by master reconciliation. - **97 cases pass at this master integration** across decision, startup and scoped-deadline suites, plus server types/build. The preceding parent candidate passed the full 293-case decision/gateway/archive/startup/deadline selection; those results remain evidence for that revision. All selected cases executed with separate PostgreSQL fixtures where needed. - Tenant-driven global scan amplification reproduced six sweeps for six sequential real cancellations. Existing thirty-second notification coalescing now permits one recovery sweep per window while retaining pending work. The first green fixture wrongly expected the intent hold to clear before coalesced reconciliation; the hold remains and quiet is checked after settlement. Foreground delivery stays immediate. - The archive contention test uses a signaled dedicated single-connection read and two FIFO SQL round trips instead of a fixed delay. Removing effect coalescing makes it fail with two contender reads; restoring production code passes. No production test hook was added. - **Three new real PostgreSQL regressions failed before repair**: recovered archive proposals with invalid signatures, mismatched manifests or archive execution failures and no inherited pending flag. Each now persists failure, wakes once, acknowledges continuation delivery and remains quiet on replay. Running-state fencing is preserved. - The unmerged continuation index is regenerated as `0388_ambitious_micromax.sql` after parent migrations 386/387. All parent journal entries are unchanged, five generated snapshots remain, numbering/safety and snapshot/schema-drift checks pass. Fresh database migration apply and replay pass; the concurrent index is valid. - Existing lifecycle coverage includes foreground/worker coalescing, child pools, transaction commit/rollback, failed acknowledgement, restart, drain/resume, shutdown and fair 100-row recovery. Earlier bounded-pass and concurrent archive-result regressions and their repairs remain recorded in [the owning behavior audit](doc/idle-decision-continuations.md). - Failure accounting: the first combined command used an incorrect retention fixture variable and skipped 15 cases. The corrected separate external-PostgreSQL retention run passed all 15. The historical nine embedded-only work-signal cases were not executed by this selection. Extra upstream embedded-only suites attempted on the parent could not initialize locally; they remain CI verification, as recorded in the parent PR. Earlier reused-fixture, assertion and compile failures remain in the behavior audit and stack ledger; they are not erased by later passing checks. Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter @paperclipai/server build`, server Vitest for `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; `pnpm --filter @paperclipai/db check:migrations`, migration apply/replay, and the generated snapshot-drift test. Local targeted evidence does not establish a full local repository run, deployed sleep or fleet savings. Fresh exact-head CI and both automated reviews remain merge gates. ## Risks - Notification-driven recovery remains global but is coalesced to thirty seconds. This bounds request-driven scans; it does not introduce company-scoped selection. Foreground delivery remains immediate. - Delivery remains at least once. A crash or acknowledgement failure after accepting a wake can send it again. Local coalescing is not a cross-process claim. - Supported writers share the application DB signal owner. Independent SQL writers or multiple active server processes need a separate wake contract. Wake-producing service methods do not gain a new guarantee when called inside an uncommitted caller transaction. - The concurrent index needs top-level migration execution; it is registered with the existing migration owner and can be replayed safely. - No Cloud sleep blocker or external alarm coverage changes. Worker quiet does not prove deployed app or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-decision-continuations.md | 193 ++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 136 ++++++- packages/db/src/client.ts | 2 + packages/db/src/index.ts | 2 +- .../migrations/0388_ambitious_micromax.sql | 1 + ...{0383_snapshot.json => 0388_snapshot.json} | 107 +++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/decisions.ts | 2 + packages/db/src/work-signals.test.ts | 22 +- packages/db/src/work-signals.ts | 8 +- .../src/__tests__/decisions-service.test.ts | 340 +++++++++++++++++- .../server-startup-feedback-export.test.ts | 9 +- server/src/index.ts | 8 +- server/src/services/deadline-work.ts | 7 +- .../services/decision-continuation-work.ts | 20 ++ server/src/services/decision-continuations.ts | 66 ++++ .../services/decision-work-notifications.ts | 9 + server/src/services/decisions.ts | 120 +++---- .../services/delivery-work-notifications.ts | 2 + 19 files changed, 978 insertions(+), 83 deletions(-) create mode 100644 doc/idle-decision-continuations.md create mode 100644 packages/db/src/migrations/0388_ambitious_micromax.sql rename packages/db/src/migrations/meta/{0383_snapshot.json => 0388_snapshot.json} (99%) create mode 100644 server/src/services/decision-continuation-work.ts create mode 100644 server/src/services/decision-continuations.ts create mode 100644 server/src/services/decision-work-notifications.ts diff --git a/doc/idle-decision-continuations.md b/doc/idle-decision-continuations.md new file mode 100644 index 00000000000..c801ccf2e20 --- /dev/null +++ b/doc/idle-decision-continuations.md @@ -0,0 +1,193 @@ +# Decision continuation delivery while idle + +A terminal decision can owe an agent wake. Its existing row stores that debt: +`continuationPolicy = wake_origin_agent`, `metadata.continuationPending = true`, +and either expired/cancelled status or a decided status with succeeded/partial/ +failed execution. Open decisions and running effects are not ready to deliver. +This migration adds no queue table and changes no wake payload. + +## Ownership and producers + +`decisions.ts` owns decision creation, authorization, signatures, effects, and +state changes. Its ten update sites now use a narrow transaction-aware write +helper before the SQL statement. These cover the running claim; ordinary effect +completion; archive-proposal signature, manifest, and execution failures; archive +success; TTL rejection on use; cancel; direct dismissal; and background expiry. +The helper acquires intent in the existing outer transaction or its savepoint. +It does not include effect execution, provider calls, activity publication, or +agent wake callbacks in a new database transaction. + +Creation and bundle creation insert open decisions, so they do not create ready +continuation debt. Acknowledgement only clears debt and belongs to the delivery +owner; it does not signal itself. The production decision services are created +by server startup and routes with the application's DB handle. Independent SQL +writers and additional active processes require a separate notification and +ownership contract. This change does not make wake-producing service methods +safe to call inside an uncommitted caller transaction; only their SQL mutation +boundary has the existing outer-commit guarantee. + +`decision-continuations.ts` owns the readiness predicate, delivery, and recovery. +Each worker pass attempts at most 100 candidates. A saved in-memory cursor +advances past failed candidates between passes and wraps after the last page. +Remaining durable work retains another pass. This avoids starving later rows behind a failing first page. +A partial ID index contains ready pending continuations only. The predicate uses +fixed SQL literals so a prepared generic plan can still use that index. + +The shared DB package exposes an opaque work-scope identity. Raw DB object +identity is insufficient because dedicated and liveness child pools already +share their owner's signals. The key is shared by those children and nested +transactions; independent roots have different keys. No connection credentials, +private transaction state, or decision policy are exposed. Existing signal +consumers and transactions that do not register work keep their behavior. + +## Foreground, recovery, and failure + +Foreground delivery remains immediate. All service instances sharing the DB +owner coalesce a simultaneous wake for the same decision into one promise. +Delivery re-reads durable readiness after claiming that local flight, so a late +foreground call cannot resend a continuation already acknowledged by recovery. +The promise is removed on success or failure. Effect execution and its final +state write also share one local flight per decision. This prevents concurrent +archive callers from turning a successful archive into a stale-manifest failure. This is process-local coalescing, +not a new durable claim or an exactly-once guarantee. + +Effect completion requires the row to remain running. A competing completion +cannot restore pending metadata after another caller has completed and delivered +it. Effects retain their own existing durable claims and authorization checks. +The delivery worker never executes effects. + +A wake failure leaves pending metadata intact. A failed acknowledgement also +leaves debt, so a later attempt can send the wake again. A crash after accepting +the wake but before acknowledgement retains the existing at-least-once contract. +An uncertain producer transaction remains held until the shared work-signal +protocol establishes settlement; one empty scan cannot clear that uncertainty. + +## Lifecycle and scope + +Startup reconstructs pending delivery. Committed changes wake the worker through +the existing coordinator. Failures or remaining ready debt retain a 30-second +retry. Open/running decisions, no decisions, and delivered history have no +recurring continuation scan or deadline. Changes during a pass retain another +request. Drain/standby defer dispatch without consuming that request; shutdown +waits for the current admitted delivery and stops before starting another. +The shared deadline adapter forwards its existing AbortSignal to task execution; +existing callbacks that do not consume the signal keep their behavior. A future process-local retry is not a Cloud alarm. + +Enabled-heartbeat startup registers this worker before the combined decision +sweep. The main tick skips that sweep's independent continuation scan. Its public +API retains combined behavior by default. Expiry or effect recovery can still +deliver their newly produced continuations immediately. Disabled-heartbeat +startup keeps its existing behavior and does not dispatch this new worker. + +Decision expiry, invalidated-target scanning, interrupted effect recovery, and +retention remain separate migrations. Existing Cloud safety blockers are not +relaxed. This establishes worker quiet under the named conditions, not deployed +host/database suspension or measured fleet savings. + +## Verification + +The selected run passes 93 tests: 50 real-PostgreSQL decision cases, 39 startup +cases, and four database work-signal unit cases. Fifteen new continuation cases +cover real decide/cancel/dismiss/expiry producers, running-to-terminal completion, +foreground contention, dedicated child pools, stale caller re-read, restart, +failed acknowledgement, outer rollback/commit, first write during drain, failed +retry through drain, changes during delivery, shutdown, and paging past a failed +candidate. Quiet checks observe no SQL calls across an hour of controlled time. +Nine pre-existing embedded-PostgreSQL signal tests are skipped locally because +the embedded runtime is unavailable; CI must run that supported environment. +Workspace typecheck and build pass. The final literal-predicate adjustment also +passes all 49 decision tests before the additional archive-race regression. + +The original continuation index applied and replayed without error. After parent integration it is generated migration `0388_ambitious_micromax.sql`; the fresh apply and replay also pass. A rolled-back synthetic +40,000-row history with one pending continuation uses the partial index under +`force_generic_plan`, reads two shared buffers, and executes in 0.015 ms on the +local PostgreSQL server. This is query evidence, not a production latency claim. + +The first test selection had 90 passes and one failure: the injected PostgreSQL +acknowledgement error is wrapped by Drizzle, but the assertion expected its +message at the top level. The assertion now checks the original cause. No +production retry, hold, or timeout was weakened. A recursive unit-fixture type +initially prevented migration generation; an explicit fixture type fixed it. +Full CI and fresh review remain required before PR readiness. + + +Review found two real regressions. Before repair, a controlled concurrent archive +reported failed after archiving successfully, and shutdown sent a second wake +after stop was requested. Both regressions failed before the fix. Shared local +effect ownership fixes archive completion; bounded passes and abort propagation +fix backlog shutdown. The 93-case selection passes after repair. The paging test +also proves a failed first candidate cannot hide the 101st candidate across +bounded passes. A result-type alias initially made the known decision ID optional; +the alias now matches the existing outcome contract. + +The existing tool-review and archive-delivery consumers also pass 124 real-DB +tests after AbortSignal propagation. Together with the decision/startup/signal +selection, 217 selected tests pass; the same nine older embedded-only cases +remain locally skipped. Workspace typecheck passes after the outcome type fix. + + +## Explicit archive-failure delivery + +The three archive-proposal failure paths now explicitly set +`continuationPending: true` for `wake_origin_agent`, rather than assuming the +pre-read running row already carries that metadata. The existing running-state +fence remains, so stale settlement cannot re-arm an acknowledged result. +Invalid signatures, manifest mismatches and archive execution failures each +failed a real-PostgreSQL recovery regression before repair. Each now saves the +failed result, wakes once, acknowledges delivery and remains quiet on replay. + +Normal parent integration preserves the projection interaction-link index and +archive pending-order index. The unmerged continuation index was regenerated +as 0388 from the parent's generated metadata; every parent journal entry is +unchanged, five snapshots remain, the SQL uses the existing top-level concurrent +executor, and snapshot/schema drift passes. No generated snapshot was hand-edited. + +The final affected selection passes 111 cases: 53 decisions, 15 retention/archive, +42 startup and one scoped-deadline case, with all four suites exercised. The +first combined command used the wrong retention DB variable and skipped those +15 cases; the corrected external-PostgreSQL retention run passes all 15. Server +typecheck and build pass. The historical nine embedded-only work-signal skips +above are not reclassified as executed by this selection. Fresh CI and reviews +remain required. + + +## Bounded notification-driven recovery + +Security review identified that sequential decision writes could each start an +instance-wide continuation scan. A real-producer regression reproduced six +sweeps for six sequential cancellations. The worker now uses the existing +`notificationCoalesceMs: 30_000` option. The first wake after a quiet window still +reconciles promptly, and subsequent notifications in that window share its end +deadline. Foreground continuation delivery is unchanged and immediate. Recovery +can wait up to thirty seconds, matching the prior maintenance cadence. This +bounds global scan frequency; it does not make queries company-scoped or add a +cross-process notification contract. + +The regression verifies one scan during repeated real create/cancel writes, +retention and eventual delivery of a later pending row, and no recurring SQL +after settlement. Its first green attempt wrongly expected zero active work +while the coalesced request was pending. The existing intent hold correctly +remains until reconciliation; the test now checks that hold's actual terminal +boundary through the existing quiet assertion. No production hold was removed. +All 293 integrated cases across decisions, archive, gateway/projection, startup +and scoped-deadline suites pass without skips; server types/build pass. Fresh +security review, Greptile, Ponytail and exact-head CI remain merge gates. + + +The archive contention regression no longer waits a fixed 100 ms. A dedicated +single-connection contender signals its decision read; two FIFO SQL round trips +then establish that it reached shared execution while the first archive remains +held. The test requires one contender decision read and one archive execution. +Temporarily removing the effect-flight guard made this test fail (two reads); +restoring the production guard passes. The mutation was restored before commit. +This verifies actual cross-pool contention without a timing delay or a new +production-only test hook. + + +Archive #17475 is merged. Normal reconciliation with its exact squash +`5785090ff1` preserves all 31 independent incoming master files, including +Runner controller-authority recovery. All 97 decision/startup/deadline cases and +server types/build pass after that integration. The Runner vendor rebuilt from +its changed inputs. The preceding full five-suite selection passed 293 cases; +the continuation owner's production behavior is unchanged by reconciliation. +Fresh exact-head reviews and CI remain required on the master-target candidate. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 2345ab9bb90..6b1b248598f 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -15,10 +15,12 @@ Do not mark an item done from a timer removal or a green check alone. | 01 | Custom-image setup expiry | Session writes, existing expiry, startup/retry/drain | [#17443](https://github.com/paperclipai/paperclip/pull/17443), merged after CI and passing reviews | | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | -| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; notification delivery implemented in slice 08, PR pending | -| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Pending | +| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | +| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478), expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479), and effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) await fresh current-head gates; target invalidation review repair remains in progress | + | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | -| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Secret proposals: [#17446](https://github.com/paperclipai/paperclip/pull/17446), merged; gateway credentials: [#17450](https://github.com/paperclipai/paperclip/pull/17450), merged; pending tool-review expiry: [#17453](https://github.com/paperclipai/paperclip/pull/17453), merged; approved recovery: [#17454](https://github.com/paperclipai/paperclip/pull/17454), merged after current-head CI and both 5/5 reviews; interrupted claims: [#17458](https://github.com/paperclipai/paperclip/pull/17458), merged after current-head CI and both 5/5 reviews; projection: [#17468](https://github.com/paperclipai/paperclip/pull/17468), interaction-link index verified and awaiting fresh CI/review | +| 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Slices 02–07 merged: [secret proposals](https://github.com/paperclipai/paperclip/pull/17446), [credentials](https://github.com/paperclipai/paperclip/pull/17450), [review expiry](https://github.com/paperclipai/paperclip/pull/17453), [approved recovery](https://github.com/paperclipai/paperclip/pull/17454), [interrupted claims](https://github.com/paperclipai/paperclip/pull/17458), [projection](https://github.com/paperclipai/paperclip/pull/17468) | + | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | @@ -359,3 +361,131 @@ counter regression and corrected disabled-mode expectation are recorded in the behavior document. Full current-head CI and both reviews remain required. Retention eligibility/scanning, decision expiry, effect recovery, and decision continuation delivery remain independent unfinished reasons in the full ledger. + + +Slice 08 is [#17475](https://github.com/paperclipai/paperclip/pull/17475), head +`692e3ee02c`, based on #17468. The PR is attached and Greptile review was +requested. Fresh CI is active with no failures in the latest snapshot. +The first seven PRs are mergeable and have all checks passing at their recorded +heads. They remain open; no merge was requested for this stack. + +## Next owner audit: decision continuations + +The next branch is `codex/sleep-09-decision-continuations`, based on #17475. +No ninth implementation or readiness is claimed. `decisions.ts:sweepExpired` +currently mixes four distinct reasons: interrupted effect recovery, pending +continuation delivery, TTL expiry, and invalidated strict targets. Preserve +separate PRs for those duties. A deadline alone cannot replace target-change +notifications; target deletion and completion need an audited producer contract. + +For continuation delivery, the existing decision row already stores the durable +obligation: `continuationPolicy = wake_origin_agent`, +`metadata.continuationPending = true`, and a terminal outcome. Decided rows +are ready only after execution reaches succeeded, partial, or failed. Open +decisions and still-running effects must not create a delivery retry. + +All current decision-row writers are in `decisions.ts`. Production service +instances are created in server startup and decision routes with the server DB +handle. Audit creation/bundles, decision claims, regular and archive-proposal +effect settlement/failure, TTL expiry on use, background expiry, cancellation, +dismissal, and continuation acknowledgement. Reuse a narrow pre-write helper +inside existing transaction ownership; do not wrap provider/effect execution +or agent wake calls in a database transaction. The archive-proposal error +branches inherit the claim's pending metadata and must remain eligible after +terminal failure. + +The current foreground paths call `deliverContinuation` after saving the result. +It sends the wake then clears pending metadata. Simply adding an eager worker +would make foreground/worker overlap common. Preserve prompt foreground delivery +and the existing at-least-once crash contract, but coalesce simultaneous attempts +for the same decision and re-read durable readiness before sending. Scope any +in-memory coalescing to the actual database/instance owner; do not let unrelated +instances or differing transaction ownership share an accidental global key. +The DB signal owner also supports child pools, so a proposed coalescing mechanism +must explicitly reconcile that boundary rather than assuming object identity +is always sufficient. No new table is needed merely for this migration. + +After commit-aware producer coverage, add a paged continuation-only recovery +entry point and register it with the existing coordinator. Remove only its +independent pending scan from the periodic combined sweep. Keep normal +continuations produced by expiry/effect completion, and preserve the public +combined recovery entry point for current callers/tests. A failed wake or +acknowledgement retains its durable pending state and a retry. Matching delivered +history becomes quiet. + +Tests should exercise actual decide/cancel/dismiss/expiry producers, completion +of effects after a running claim, concurrent foreground and worker delivery, +wake/acknowledgement failures, restart, outer commit/rollback, first write and +failure during drain, pagination past failing rows, and delivered-history quiet. +The existing decisions suite has an embedded-only setup; use the repository's +validated external test database helper to exercise it on the disposable local +PostgreSQL server, without silently skipping the suite when embedded setup is +unavailable. + +Slice 08 review update: Greptile is 5/5 on `692e3ee02c` with no actionable +findings. The latest CI snapshot has 10 successful checks and 44 in progress, +with no failures. This is a live CI wait, not a readiness or completion claim. + + +## Slice 09 implementation: decision continuations + +The continuation owner and worker are implemented on `codex/sleep-09-decision-continuations`. +See [decision continuations](../idle-decision-continuations.md) for producer, +transaction, concurrency, failure, and test evidence. The existing row is durable +truth; the only DB API addition is an opaque shared work-scope identity for local +coalescing across child pools. No new queue table or external wake exemption is +added. Terminal effect settlement is fenced to running state so a competing +caller cannot re-arm an acknowledged continuation. A concurrent partial index +bounds pending selection independently of retained history. + +The selected run passes 92 tests; nine older embedded-Postgres cases are locally +skipped, with the new lifecycle tests running against real external PostgreSQL. +Workspace typecheck/build pass. Migration apply/replay and a 40,000-row prepared +query probe pass. Initial fixture type/error-shape failures are recorded in the +behavior document. PR creation and full CI/review are next. + +Parent #17475 at `692e3ee02c` now has passing checks and Greptile 5/5; the extra +Paperclip Review check was skipped. Fresh master is `fe85fd7ac4`, still at +migration 0377. The stack remains open and unmerged. The full inventory above +remains the completion scope. + + +Slice 09 is [#17478](https://github.com/paperclipai/paperclip/pull/17478), initially +`66bbef521e`. Review found a concurrent archive result race and unbounded backlog +shutdown. Both failed real-boundary regressions before repair. The local repair +shares effect execution through completion per DB owner/decision, bounds each +continuation pass to 100 with a fair cursor, and forwards cancellation through +the shared deadline adapter. The repaired selection passes 217 tests, with nine +older embedded-only signal tests skipped locally. Typecheck passes. Fresh build, +CI, and review remain required; the initial head has no other CI failure. + +The separate TTL branch `codex/sleep-10-decision-expiry` contains implementation +commit `18c11e3110`. It passed 101 selected tests, workspace typecheck and build. +Its fault fixture initially cleared its error collector before a previous +admitted failure finished; the assertion now verifies all errors after settled +recovery. No hold or retry was weakened. This branch must receive the parent +review repairs and adapt its own paged expiry worker to cancellation before PR +publication. No tenth PR or readiness is claimed. + + +Slice 09 review follow-up: every failed archive-proposal settlement explicitly +creates its origin-agent continuation obligation. Three recovered-row regressions +failed before repair and pass with exactly one wake and persisted acknowledgement. +The current parent-integrated selection passes 111 decision/archive/startup/deadline +cases, server types/build, fresh migration apply/replay and generated snapshot +drift. Own unmerged index is regenerated as `0388_ambitious_micromax.sql`; parent +386/387 migrations and journal entries are preserved. The initial skipped retention +command and corrected 15-case run are recorded in the behavior document. + +Slice 09 security follow-up: sequential real decision writes reproduced six +global continuation scans. Existing thirty-second notification coalescing now +bounds those scans while preserving immediate foreground wakes and durable +pending recovery. All 293 integrated decision/archive/gateway/startup/deadline +cases and server types/build pass. The incorrect expectation of zero active +work during the coalescing window is recorded in the behavior document; the +intent hold remains until reconciliation. Fresh security/CI/reviews required. + +Eight slices are merged after archive delivery #17475. Slice 09 now reconciles +with its exact squash `5785090ff1`; all 31 independent master files are preserved. +The 97 decision/startup/deadline cases and server types/build pass after the +Runner vendor rebuild. Fresh master-target CI and reviews remain required. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 1d98d929d7a..443005ca277 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -506,6 +506,8 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0385_open_madame_web.sql", "0386_steady_epoch.sql", "0387_boring_professor_monster.sql", + "0388_ambitious_micromax.sql", + "0384_funny_adam_warlock.sql", "0383_tool_review_company_expiry.sql", diff --git a/packages/db/src/index.ts b/packages/db/src/index.ts index d65d39527ac..0702f8c5336 100644 --- a/packages/db/src/index.ts +++ b/packages/db/src/index.ts @@ -45,4 +45,4 @@ export { issueRelations } from "./schema/issue_relations.js"; export { issueReferenceMentions } from "./schema/issue_reference_mentions.js"; export * from "./schema/index.js"; -export { signalDatabaseWork, subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "./work-signals.js"; +export { databaseWorkScopeKey, signalDatabaseWork, subscribeDatabaseWork, databaseWorkPending, reconcileDatabaseWork } from "./work-signals.js"; diff --git a/packages/db/src/migrations/0388_ambitious_micromax.sql b/packages/db/src/migrations/0388_ambitious_micromax.sql new file mode 100644 index 00000000000..ad828b44e2c --- /dev/null +++ b/packages/db/src/migrations/0388_ambitious_micromax.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "decisions_pending_continuation_idx" ON "decisions" USING btree ("id") WHERE "decisions"."continuation_policy" = 'wake_origin_agent' AND "decisions"."metadata" ->> 'continuationPending' = 'true' AND ("decisions"."status" IN ('expired', 'cancelled') OR ("decisions"."status" = 'decided' AND "decisions"."execution_status" IN ('succeeded', 'partial', 'failed'))); \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0383_snapshot.json b/packages/db/src/migrations/meta/0388_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0383_snapshot.json rename to packages/db/src/migrations/meta/0388_snapshot.json index 3336a6bb20e..c7cd58d2a5a 100644 --- a/packages/db/src/migrations/meta/0383_snapshot.json +++ b/packages/db/src/migrations/meta/0388_snapshot.json @@ -1,6 +1,6 @@ { - "id": "410457fc-266b-4bd0-8bc0-2b14b0634d07", - "prevId": "ecec138e-5312-4ad6-b051-bd4ee9ed4e34", + "id": "4f21a636-735f-4dce-9ab8-0583f3bc7b79", + "prevId": "de27fdf8-430a-443a-9502-725f92c48892", "version": "7", "dialect": "postgresql", "tables": { @@ -22110,6 +22110,34 @@ "concurrently": false, "method": "btree", "with": {} + }, + "decision_archive_notification_outbox_delivery_idx": { + "name": "decision_archive_notification_outbox_delivery_idx", + "columns": [ + { + "expression": "last_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "first" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decision_archive_notification_outbox\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -24030,6 +24058,22 @@ "method": "btree", "with": {} }, + "decisions_pending_continuation_idx": { + "name": "decisions_pending_continuation_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"continuation_policy\" = 'wake_origin_agent' AND \"decisions\".\"metadata\" ->> 'continuationPending' = 'true' AND (\"decisions\".\"status\" IN ('expired', 'cancelled') OR (\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" IN ('succeeded', 'partial', 'failed')))", + "concurrently": false, + "method": "btree", + "with": {} + }, "decisions_bundle_idx": { "name": "decisions_bundle_idx", "columns": [ @@ -52856,6 +52900,44 @@ "method": "btree", "with": {} }, + "tool_action_requests_executing_deadline_idx": { + "name": "tool_action_requests_executing_deadline_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'executing'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "tool_action_requests_approved_idx": { + "name": "tool_action_requests_approved_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'approved'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_invocation_idx": { "name": "tool_action_requests_invocation_idx", "columns": [ @@ -52871,6 +52953,27 @@ "method": "btree", "with": {} }, + "tool_action_requests_company_interaction_idx": { + "name": "tool_action_requests_company_interaction_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "interaction_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_issue_idx": { "name": "tool_action_requests_issue_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 4e9bf240449..68f12e89697 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2703,6 +2703,13 @@ "when": 1793136115698, "tag": "0387_boring_professor_monster", "breakpoints": true + }, + { + "idx": 388, + "version": "7", + "when": 1793137983484, + "tag": "0388_ambitious_micromax", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/decisions.ts b/packages/db/src/schema/decisions.ts index e2f668c40e6..18b18d0c704 100644 --- a/packages/db/src/schema/decisions.ts +++ b/packages/db/src/schema/decisions.ts @@ -69,6 +69,8 @@ export const decisions = pgTable( table.status, table.expiresAt, ), + pendingContinuationIdx: index("decisions_pending_continuation_idx").on(table.id) + .where(sql`${table.continuationPolicy} = 'wake_origin_agent' AND ${table.metadata} ->> 'continuationPending' = 'true' AND (${table.status} IN ('expired', 'cancelled') OR (${table.status} = 'decided' AND ${table.executionStatus} IN ('succeeded', 'partial', 'failed')))`), bundleIdx: index("decisions_bundle_idx").on(table.bundleId), originIssueIdx: index("decisions_origin_issue_idx").on(table.originIssueId), companyIdempotencyUq: uniqueIndex("decisions_company_idempotency_uq") diff --git a/packages/db/src/work-signals.test.ts b/packages/db/src/work-signals.test.ts index 13bd4b8552a..b14740bd38e 100644 --- a/packages/db/src/work-signals.test.ts +++ b/packages/db/src/work-signals.test.ts @@ -3,7 +3,7 @@ import { drizzle } from "drizzle-orm/postgres-js"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { createDb, withDedicatedDbConnection } from "./client.js"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./test-embedded-postgres.js"; -import { databaseWorkPending, installDatabaseWorkSignals, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js"; +import { databaseWorkScopeKey, databaseWorkPending, installDatabaseWorkSignals, reconcileDatabaseWork, signalDatabaseWork, subscribeDatabaseWork } from "./work-signals.js"; const support = await getEmbeddedPostgresTestSupport(); (support.supported ? describe : describe.skip)("database work signals", () => { @@ -187,3 +187,23 @@ describe("work signal query boundaries", () => { await expect(signalDatabaseWork(connection, "bad")).rejects.toThrow("inside a transaction"); }); }); + + +describe("database work scope identity", () => { + type TestConnection = { execute: () => Promise; transaction: (callback: (tx: TestConnection) => unknown) => Promise }; + function connection(): TestConnection { + return { execute: vi.fn(async () => []), transaction: async callback => callback(connection()) }; + } + it("shares owner identity through child pools and nested transactions, but isolates independent owners", async () => { + const first = installDatabaseWorkSignals(connection()); + const second = installDatabaseWorkSignals(connection()); + const child = installDatabaseWorkSignals(connection(), first); + const key = databaseWorkScopeKey(first); + expect(databaseWorkScopeKey(second)).not.toBe(key); + expect(databaseWorkScopeKey(child)).toBe(key); + await child.transaction(async tx => { + expect(databaseWorkScopeKey(tx)).toBe(key); + await tx.transaction(async nested => { expect(databaseWorkScopeKey(nested)).toBe(key); }); + }); + }); +}); diff --git a/packages/db/src/work-signals.ts b/packages/db/src/work-signals.ts index 4abb2570a5e..cf914398e5d 100644 --- a/packages/db/src/work-signals.ts +++ b/packages/db/src/work-signals.ts @@ -5,7 +5,7 @@ import { sql } from "drizzle-orm"; // an XID, so unrelated transactions keep their existing SQL behavior. type Listener = (event: "intent" | "settled", hints?: ReadonlySet) => void; type Transaction = { topics: Set; hints: Map>; xid?: Promise; uncertain: boolean }; -type Scope = { listeners: Map>; transactions: Set; root: Connection }; +type Scope = { key: object; listeners: Map>; transactions: Set; root: Connection }; type Context = { scope: Scope; transaction?: Transaction }; type Connection = { transaction: (...args: any[]) => Promise; execute: (...args: any[]) => Promise }; const contexts = new WeakMap(); @@ -19,7 +19,7 @@ function publish(scope: Scope, topic: string, event: "intent" | "settled" = "set /** Installed once by createDb; dedicated child pools share their owner's scope. */ export function installDatabaseWorkSignals(db: T, owner?: object): T { - const scope = (owner && contexts.get(owner)?.scope) || { listeners: new Map(), transactions: new Set(), root: db }; + const scope = (owner && contexts.get(owner)?.scope) || { key: {}, listeners: new Map(), transactions: new Set(), root: db }; install(db, { scope }); return db; } @@ -112,3 +112,7 @@ export function subscribeDatabaseWork(owner: object, topic: string, listener: Li } return () => { listeners.delete(listener); }; } + +export function databaseWorkScopeKey(connection: object): object { + return contexts.get(connection)?.scope.key ?? connection; +} diff --git a/server/src/__tests__/decisions-service.test.ts b/server/src/__tests__/decisions-service.test.ts index 4b418e355f1..63719ba1ea7 100644 --- a/server/src/__tests__/decisions-service.test.ts +++ b/server/src/__tests__/decisions-service.test.ts @@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto"; import { mkdtempSync, rmSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; -import { eq, inArray } from "drizzle-orm"; +import { eq, inArray, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { activityLog, @@ -11,6 +11,7 @@ import { companies, companyMemberships, createDb, + withDedicatedDbConnection, decisionEffectExecutions, decisionRetention, decisions, @@ -23,10 +24,19 @@ import { import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { attentionService } from "../services/attention.js"; import { decisionService } from "../services/decisions.js"; +import * as retentionModule from "../services/decision-retention.js"; import { hashAttentionArchiveManifest } from "../services/decision-retention.js"; import type { AttentionArchiveManifestEntry, AttentionArchiveTargetSnapshot } from "@paperclipai/shared"; -const support = await getEmbeddedPostgresTestSupport(); +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { decisionContinuationService } from "../services/decision-continuations.js"; +import { registerDecisionContinuationWork } from "../services/decision-continuation-work.js"; +import { writeDecisionChange } from "../services/decision-work-notifications.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_DECISIONS_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describePg = support.supported ? describe : describe.skip; describePg("decisionService", () => { @@ -41,9 +51,11 @@ describePg("decisionService", () => { let decidedByUserId: string; let wakes: Array>; + let stopWorker: (() => Promise) | undefined; + beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-decisions-"); - db = createDb(tempDb.connectionString); + if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-decisions-"); + db = createDb(externalUrl ?? tempDb!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); }, 20_000); beforeEach(async () => { @@ -69,13 +81,14 @@ describePg("decisionService", () => { }); afterEach(async () => { + await stopWorker?.(); stopWorker = undefined; delete process.env.PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE; delete process.env.PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS; delete process.env.PAPERCLIP_AGENT_JWT_SECRET; await db.delete(decisionEffectExecutions); await db.delete(decisionTargetIssues); await db.delete(decisions); await db.delete(decisionRetention); await db.delete(activityLog); await db.delete(issueComments); await db.delete(issueRelations); await db.delete(heartbeatRuns); await db.delete(issues); await db.delete(agents); await db.delete(companyMemberships); await db.delete(authUsers); await db.delete(companies); }); - afterAll(async () => tempDb?.cleanup()); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); const agentActor = () => ({ type: "agent" as const, companyId, agentId, runId, source: "agent_jwt" as const, onBehalfOfUserId: originResponsibleUserId, onBehalfOfMemberships: [{ companyId, membershipRole: "member", status: "active" }] }); @@ -194,6 +207,87 @@ describePg("decisionService", () => { expect(states.find((row) => row.sourceId === unreviewedIssueId)?.archivedAt).toBeNull(); }); + it.each(["invalid_signature", "manifest_mismatch", "archive_failure"] as const)("delivers a recovered archive failure without inherited pending metadata: %s", async failure => { + const activityAt = new Date("2026-04-01T00:00:00.000Z"); + const manifest: AttentionArchiveManifestEntry[] = [{ companyId, sourceKind: "review", sourceId: targetIssueId, + expectedVersion: 1, activityAt: activityAt.toISOString(), reason: "Archive" }]; + const row = await service().create({ companyId, actor: agentActor(), agentId, runId, title: "Archive?", body: "Exact set", + continuationPolicy: "wake_origin_agent", options: [{ id: "archive", label: "Archive", effects: [] }], + metadata: { kind: "attention_archive_proposal", manifestHash: failure === "manifest_mismatch" ? "incorrect" : hashAttentionArchiveManifest(manifest) }, + additionalTargetSnapshots: { [`attention:review:${targetIssueId}`]: { status: "attention", assigneeAgentId: null, + assigneeUserId: null, updatedAt: activityAt.toISOString(), attentionArchive: manifest[0] } } }); + await db.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: "archive", + decidedByUserId, decidedAt: new Date(), updatedAt: new Date(Date.now() - 120_000), + ...(failure === "invalid_signature" ? { signedSpec: "invalid" } : {}), + metadata: { ...row.metadata, continuationPending: false } }).where(eq(decisions.id, row.id)); + const original = retentionModule.decisionRetentionService; + const factory = failure === "archive_failure" ? vi.spyOn(retentionModule, "decisionRetentionService").mockImplementation((source, options) => ({ + ...original(source, options), archiveReviewedManifest: async () => { throw new Error("archive_failure"); }, + })) : undefined; + try { + expect(await service().sweepExpired()).toMatchObject({ resumed: 1 }); + const saved = await service().get(row.id); + expect(saved).toMatchObject({ executionStatus: "failed", metadata: { archiveProposalError: failure, continuationPending: false } }); + expect(saved!.metadata.continuationDeliveredAt).toEqual(expect.any(String)); + expect(wakes).toHaveLength(1); + await service().sweepExpired(); + expect(wakes).toHaveLength(1); + } finally { factory?.mockRestore(); } + }); + + it("serializes concurrent archive completion before reporting its outcome", async () => { + await db.update(issues).set({ status: "in_review", assigneeUserId: decidedByUserId }).where(eq(issues.id, targetIssueId)); + const activityAt = new Date("2026-04-01T00:00:00.000Z"); + await db.insert(decisionRetention).values({ companyId, sourceKind: "review", sourceId: targetIssueId, sourceActivityAt: activityAt }); + const manifest: AttentionArchiveManifestEntry[] = [{ companyId, sourceKind: "review", sourceId: targetIssueId, + expectedVersion: 1, activityAt: activityAt.toISOString(), reason: "Archive" }]; + const row = await service().create({ companyId, actor: agentActor(), agentId, runId, title: "Archive?", body: "Exact set", + continuationPolicy: "wake_origin_agent", options: [{ id: "archive", label: "Archive", effects: [] }], + metadata: { kind: "attention_archive_proposal", manifestHash: hashAttentionArchiveManifest(manifest) }, + additionalTargetSnapshots: { [`attention:review:${targetIssueId}`]: { status: "attention", assigneeAgentId: null, + assigneeUserId: null, updatedAt: activityAt.toISOString(), attentionArchive: manifest[0] } } }); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const original = retentionModule.decisionRetentionService; + let archived = false, calls = 0; + const factory = vi.spyOn(retentionModule, "decisionRetentionService").mockImplementation((source, options) => { + const actual = original(source, options); + return { ...actual, archiveReviewedManifest: async input => { + calls++; + const result = await actual.archiveReviewedManifest(input); + archived = true; await gate; return result; + } }; + }); + const decide = () => service().decide({ id: row.id, optionId: "archive", decidedByUserId, userActor: boardActor() }); + const first = decide(); + let second: ReturnType | undefined; + try { + await vi.waitFor(() => expect(archived).toBe(true)); + await withDedicatedDbConnection(db, async contender => { + let observed!: () => void; + const readStarted = new Promise(resolve => { observed = resolve; }); + let decisionReads = 0; + contender.$client.options.debug = (_id, query) => { + if (/^select/i.test(query) && query.includes('from "decisions"')) { decisionReads++; observed(); } + }; + try { + second = decisionService(contender, { wakeOriginAgent: async input => { wakes.push(input); } }) + .decide({ id: row.id, optionId: "archive", decidedByUserId, userActor: boardActor() }); + await readStarted; + await contender.execute(sql`select 1`); + await contender.execute(sql`select 1`); + expect(decisionReads).toBe(1); + release(); + const results = await Promise.all([first, second]); + expect(results.map(result => result.executionStatus)).toEqual(["succeeded", "succeeded"]); + expect(calls).toBe(1); expect(wakes).toHaveLength(1); + const [state] = await db.select().from(decisionRetention); + expect(state.archiveVersion).toBe(1); expect(state.archivedAt).not.toBeNull(); + } finally { release(); await Promise.allSettled([first, ...(second ? [second] : [])]); } + }); + } finally { release(); await Promise.allSettled([first, ...(second ? [second] : [])]); factory.mockRestore(); } + }); + it("skips strict stale targets and fails closed on intersection denial", async () => { const stale = await createCommentDecision("strict"); await db.update(issues).set({ updatedAt: new Date(Date.now() + 1_000) }).where(eq(issues.id, targetIssueId)); @@ -656,4 +750,240 @@ describePg("decisionService", () => { expect(result).toMatchObject({ status: "decided", chosenOptionId: "dismissed" }); expect(wakes).toEqual([{ companyId, agentId, issueId: originIssueId, decisionId: created.id, outcome: "decided" }]); }); + describe("continuation worker", () => { + let coordinator: ReturnType | undefined; + let draining: boolean; + const errors: unknown[] = []; + const wake = vi.fn(async (_input: Record) => {}); + const workerService = () => decisionService(db, { wakeOriginAgent: wake }); + + beforeEach(async () => { + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + draining = false; errors.length = 0; wake.mockReset(); + stopWorker = async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.execute(sql`drop trigger if exists decision_ack_fail on decisions`); + await db.execute(sql`drop function if exists decision_ack_fail()`); + expect(errors).toEqual([]); + }; + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + async function start(svc = workerService()) { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerDecisionContinuationWork(coordinator, svc, () => !draining).ready; + } + async function quiet() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(idleWorkSnapshot().active).toBe(0); + expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"), executes = vi.spyOn(db, "execute"); + await vi.advanceTimersByTimeAsync(60 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); executes.mockRestore(); + } + async function pending(id: string) { + expect((await service().get(id))?.metadata.continuationPending).toBe(true); + } + async function delivered(id: string) { + expect((await service().get(id))?.metadata.continuationPending).toBe(false); + } + const markReady = (id: string, source = db) => writeDecisionChange(source, async tx => tx.update(decisions) + .set({ status: "cancelled", metadata: { continuationPending: true } }).where(eq(decisions.id, id))); + + it("coalesces sequential tenant writes without dropping pending recovery", async () => { + const row = await createCommentDecision(); + const svc = workerService(), sweep = vi.spyOn(svc, "sweepContinuations"); + await start(svc); await quiet(); sweep.mockClear(); + await svc.cancel(row.id, { actorType: "agent", actorId: agentId }); + await vi.advanceTimersByTimeAsync(1000); + await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0)); + expect(sweep).toHaveBeenCalledTimes(1); + for (let i = 0; i < 5; i++) { + const next = await createCommentDecision(); + await svc.cancel(next.id, { actorType: "agent", actorId: agentId }); + await vi.advanceTimersByTimeAsync(1000); + } + expect(sweep).toHaveBeenCalledTimes(1); + const recovery = await createCommentDecision(); + await markReady(recovery.id); + await vi.advanceTimersByTimeAsync(1000); + await pending(recovery.id); + expect(sweep).toHaveBeenCalledTimes(1); + await quiet(); await delivered(recovery.id); + expect(sweep).toHaveBeenCalledTimes(2); + expect(wake.mock.calls.filter(([input]) => input.decisionId === recovery.id)).toHaveLength(1); + }); + + it.each(["decide", "cancel", "dismiss", "expire"] as const)("reactivates from real %s and leaves delivered history quiet", async action => { + const row = await createCommentDecision(); + await start(); await quiet(); + const svc = workerService(); + if (action === "decide") await svc.decide({ id: row.id, optionId: "yes", decidedByUserId, userActor: boardActor() }); + if (action === "cancel") await svc.cancel(row.id, { actorType: "agent", actorId: agentId }); + if (action === "dismiss") await svc.dismiss(row.id, decidedByUserId, boardActor()); + if (action === "expire") { await expireDecisionNow(row.id); await svc.sweepExpired(new Date(), { includeContinuations: false }); } + await quiet(); await delivered(row.id); + expect(wake).toHaveBeenCalledTimes(1); + expect(wake.mock.calls[0][0]).toMatchObject({ decisionId: row.id, companyId, agentId, issueId: originIssueId }); + }); + + it("stays quiet while open or running, then wakes when effect completion commits", async () => { + const row = await createCommentDecision(); + await db.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: "yes", + decidedByUserId, metadata: { continuationPending: true } }).where(eq(decisions.id, row.id)); + await start(); await quiet(); expect(wake).not.toHaveBeenCalled(); + await workerService().decide({ id: row.id, optionId: "yes", decidedByUserId, userActor: boardActor() }); + await quiet(); await delivered(row.id); expect(wake).toHaveBeenCalledTimes(1); + expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1); + }); + + it("coalesces simultaneous foreground replay and worker delivery without replaying effects", async () => { + const row = await createCommentDecision(); + await start(); + await Promise.all(Array.from({ length: 3 }, () => workerService().decide({ + id: row.id, optionId: "yes", decidedByUserId, userActor: boardActor(), + }).catch(error => { if (error.message !== "decision_already_resolved") throw error; }))); + await quiet(); await delivered(row.id); expect(wake).toHaveBeenCalledTimes(1); + expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1); + }); + + it("shares an in-flight wake with a dedicated child pool and rereads after completion", async () => { + const row = await createCommentDecision(); await markReady(row.id); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + wake.mockImplementationOnce(async () => gate); + const first = decisionContinuationService(db, wake).deliver(row.id); + const childWake = vi.fn(async () => {}); + let child: Promise; + try { + await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(1)); + child = withDedicatedDbConnection(db, async connection => { + const attempt = decisionContinuationService(connection, childWake).deliver(row.id); + expect(attempt).toBe(first); + await attempt; + }); + await vi.advanceTimersByTimeAsync(100); + } finally { release(); } + await first; await child!; + await decisionContinuationService(db, childWake).deliver(row.id); + expect(childWake).not.toHaveBeenCalled(); await delivered(row.id); + }); + + it("reconstructs pending delivery at restart without replaying decision effects", async () => { + const row = await createCommentDecision(); + wake.mockRejectedValueOnce(new Error("wake unavailable")); + await expect(workerService().decide({ id: row.id, optionId: "yes", decidedByUserId, userActor: boardActor() })).rejects.toThrow("wake unavailable"); + await pending(row.id); await start(); await quiet(); await delivered(row.id); + expect(wake).toHaveBeenCalledTimes(2); + expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1); + }); + + it("retries a failed acknowledgement with the existing at-least-once wake contract", async () => { + const row = await createCommentDecision(); await start(); + await db.execute(sql`create function decision_ack_fail() returns trigger language plpgsql as $$ begin + if NEW.metadata ->> 'continuationPending' = 'false' then raise exception 'ack unavailable'; end if; + return NEW; end $$`); + await db.execute(sql`create trigger decision_ack_fail before update on decisions for each row execute function decision_ack_fail()`); + await expect(workerService().cancel(row.id, { actorType: "agent", actorId: agentId })).rejects.toMatchObject({ cause: { message: "ack unavailable" } }); + await pending(row.id); + await vi.advanceTimersByTimeAsync(60_000); + await db.execute(sql`drop trigger decision_ack_fail on decisions`); + await quiet(); await delivered(row.id); + expect(wake.mock.calls.length).toBeGreaterThan(1); + for (const error of errors) expect(error).toMatchObject({ cause: { message: "ack unavailable" } }); + errors.length = 0; + }); + + it("waits for an outer transaction, ignores rollback, and recovers the committed write", async () => { + const row = await createCommentDecision(); await start(); await quiet(); + await expect(db.transaction(async tx => { + await markReady(row.id, tx as unknown as typeof db); + await vi.advanceTimersByTimeAsync(60_000); + expect(wake).not.toHaveBeenCalled(); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect((await service().get(row.id))?.status).toBe("open"); + await db.transaction(async tx => { + await markReady(row.id, tx as unknown as typeof db); + await vi.advanceTimersByTimeAsync(60_000); expect(wake).not.toHaveBeenCalled(); + }); + await quiet(); await delivered(row.id); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("retains a first write during drain without scanning and delivers after resume", async () => { + const row = await createCommentDecision(); await start(); await quiet(); draining = true; + await markReady(row.id); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); expect(wake).not.toHaveBeenCalled(); + draining = false; await quiet(); await delivered(row.id); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("retains a failed wake through drain and resumes its retry", async () => { + const row = await createCommentDecision(); await start(); + wake.mockImplementationOnce(async () => { draining = true; throw new Error("wake unavailable"); }); + await markReady(row.id); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(errors).toHaveLength(1); }); + await pending(row.id); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; errors.length = 0; await quiet(); await delivered(row.id); expect(wake).toHaveBeenCalledTimes(2); + }); + + it("preserves changes arriving during delivery and stops after shutdown", async () => { + const firstRow = await createCommentDecision(), secondRow = await createCommentDecision(); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + wake.mockImplementationOnce(async () => { await markReady(secondRow.id); await gate; }); + await markReady(firstRow.id); + const starting = start(); + try { await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(1)); } + finally { release(); } + await starting; await quiet(); await delivered(firstRow.id); await delivered(secondRow.id); + expect(wake).toHaveBeenCalledTimes(2); + await coordinator!.stop(); coordinator = undefined; + await vi.advanceTimersByTimeAsync(60_000); expect(wake).toHaveBeenCalledTimes(2); + }); + + it("awaits an admitted wake before completing shutdown", async () => { + const row = await createCommentDecision(), other = await createCommentDecision(); + await markReady(row.id); await markReady(other.id); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + wake.mockImplementationOnce(async () => gate); + const starting = start(); + let stopped = false; + let stopping: Promise; + try { + await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(1)); + stopping = coordinator!.stop().then(() => { stopped = true; }); + await vi.advanceTimersByTimeAsync(60_000); + expect(stopped).toBe(false); + } finally { release(); } + await starting; await stopping!; coordinator = undefined; + expect(wake).toHaveBeenCalledTimes(1); + const sentId = wake.mock.calls[0][0].decisionId as string; + await delivered(sentId); await pending(sentId === row.id ? other.id : row.id); expect(stopped).toBe(true); + }); + + it("pages past a failed first candidate and preserves its retry", async () => { + const seed = await createCommentDecision(); + await db.delete(decisionTargetIssues).where(eq(decisionTargetIssues.decisionId, seed.id)); + await db.delete(decisions).where(eq(decisions.id, seed.id)); + const ids = Array.from({ length: 101 }, (_, i) => `10000000-0000-4000-8000-${String(i).padStart(12, "0")}`); + await db.insert(decisions).values(ids.map(id => ({ ...seed, id, status: "cancelled", metadata: { continuationPending: true } }))); + wake.mockImplementation(async input => { if (input.decisionId === ids[0]) throw new Error("first unavailable"); }); + const recovery = workerService(); + await expect(recovery.sweepContinuations()).rejects.toThrow("first unavailable"); + expect(wake).toHaveBeenCalledTimes(100); await pending(ids.at(-1)!); + expect(await recovery.sweepContinuations()).toEqual({ retry: true }); + expect(wake).toHaveBeenCalledTimes(101); await delivered(ids.at(-1)!); await pending(ids[0]); + wake.mockImplementation(async () => {}); + await start(); await quiet(); await delivered(ids[0]); expect(wake).toHaveBeenCalledTimes(102); + }); + }); + }); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 2f454b226e2..8eee51072d4 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -6,7 +6,8 @@ import { readIdleSleepSafety } from "../services/idle-sleep-safety.js"; import * as idleLocalWork from "../services/idle-local-work.js"; import { idleBackupWakeMarker, markIdleBackupWakeRequired } from "../services/idle-database-backup.js"; import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; -import { decisionRetentionService, reconcileBuiltInAgentsOnStartup, reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup } from "../services/index.js"; +import { decisionService, decisionRetentionService, reconcileBuiltInAgentsOnStartup, reconcileCodexLocalManagedHomesOnStartup, reconcilePersistedRuntimeServicesOnStartup } from "../services/index.js"; + import { companies, runDatabaseBackup, type Db } from "@paperclipai/db"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import { chmodSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; @@ -348,6 +349,7 @@ vi.mock("../services/index.js", () => ({ })), decisionService: vi.fn(() => ({ sweepExpired: vi.fn(async () => ({ expired: 0 })), + sweepContinuations: vi.fn(async () => ({ retry: false })), })), decisionRetentionService: vi.fn(() => ({ autoArchive: vi.fn(async () => 0), @@ -913,6 +915,10 @@ describe("startServer feedback export wiring", () => { expect(routineServiceMock.tickScheduledTriggers).toHaveBeenCalledTimes(1); expect(environmentCustomImagesServiceMock.cleanupExpiredSetupSessions).toHaveBeenCalledTimes(1); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.customImageExpiry)).toBe(true); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionContinuation)).toBe(true); + const decisions = vi.mocked(decisionService).mock.results.at(-1)!.value; + expect(decisions.sweepContinuations).toHaveBeenCalledTimes(1); + expect(decisions.sweepExpired).toHaveBeenCalledWith(expect.any(Date), { includeContinuations: false }); } finally { setIntervalSpy.mockRestore(); } @@ -949,6 +955,7 @@ describe("startServer feedback export wiring", () => { expect(routineServiceMock.tickScheduledTriggers).not.toHaveBeenCalled(); expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(environmentCustomImagesServiceMock.cleanupExpiredSetupSessions).not.toHaveBeenCalled(); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionContinuation)).toBe(false); } finally { setIntervalSpy.mockRestore(); } diff --git a/server/src/index.ts b/server/src/index.ts index 37661c37c0e..afb2b803ac9 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,5 @@ +import { registerDecisionContinuationWork } from "./services/decision-continuation-work.js"; + import { registerDecisionArchiveDeliveryWork } from "./services/decision-archive-delivery-work.js"; import { registerToolReviewProjectionWork } from "./services/tool-review-projection-work.js"; @@ -1835,7 +1837,9 @@ async function startServerWithDatabaseTeardown( logger.warn({ ...toolHealthSweep }, "startup tool connection health sweep found failing connections"); } }); - await step("decision_expiry", () => decisionExecutor.sweepExpired()); + await step("decision_continuation_worker", () => registerDecisionContinuationWork(deliveryWork, decisionExecutor, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("decision_expiry", () => decisionExecutor.sweepExpired(new Date(), { includeContinuations: false })); const loginSchedule = loginCleanupSchedule(db, config.heartbeatSchedulerIntervalMs); await step("login_cleanup_workers", () => registerLoginCleanupWork(deliveryWork, [ { queue: DELIVERY_QUEUES.adapterLoginCleanup, run: () => adapterLoginReaper.sweep().then(logAdapterLoginReaperResult), next: loginSchedule.adapter }, @@ -1856,7 +1860,7 @@ async function startServerWithDatabaseTeardown( trackHeartbeatSchedulerWork((async () => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; scheduleRoutineStatusReconcile(); - trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired().catch((err: unknown) => { + trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired(new Date(), { includeContinuations: false }).catch((err: unknown) => { logger.error({ err }, "decision expiry sweep failed"); })); trackHeartbeatSchedulerWork(runRetentionSweep().catch((err: unknown) => { diff --git a/server/src/services/deadline-work.ts b/server/src/services/deadline-work.ts index 044c3cbfce4..d6ff2841037 100644 --- a/server/src/services/deadline-work.ts +++ b/server/src/services/deadline-work.ts @@ -5,7 +5,7 @@ export function registerDeadlineWork( coordinator: ReturnType, queue: DeliveryQueue, task: { - run: () => Promise; + run: (signal: AbortSignal) => Promise; next: () => Promise; canRun: () => boolean; retryMs: number; @@ -21,7 +21,7 @@ export function registerDeadlineWork( retryMs: task.retryMs, pausePassiveMaintenance: true, onSettled: () => { requested++; }, - run: async () => { + run: async signal => { if (!task.canRun()) { requested++; return; } if (Date.now() < earliest) { next = earliest; return; } if (Date.now() < coalesceUntil && (next === null || next > Date.now())) { @@ -30,7 +30,8 @@ export function registerDeadlineWork( } const generation = ++requested; earliest = Date.now() + task.retryMs; - await task.run(); + await task.run(signal); + if (signal.aborted) return; const deadline = await task.next(); // Back off failed or still-overdue work, not a successfully read future deadline. if (deadline === null || deadline > Date.now()) earliest = 0; diff --git a/server/src/services/decision-continuation-work.ts b/server/src/services/decision-continuation-work.ts new file mode 100644 index 00000000000..c9c5b0fbcad --- /dev/null +++ b/server/src/services/decision-continuation-work.ts @@ -0,0 +1,20 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { decisionService } from "./decisions.js"; + +export function registerDecisionContinuationWork( + coordinator: ReturnType, + service: Pick, "sweepContinuations">, + canRun: () => boolean, +) { + let next: number | null = null; + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.decisionContinuation, { + retryMs: 30_000, notificationCoalesceMs: 30_000, canRun, + run: async signal => { + const result = await service.sweepContinuations(100, signal); + next = result.retry ? Date.now() + 30_000 : null; + }, + next: async () => next, + }); +} diff --git a/server/src/services/decision-continuations.ts b/server/src/services/decision-continuations.ts new file mode 100644 index 00000000000..79c70b43969 --- /dev/null +++ b/server/src/services/decision-continuations.ts @@ -0,0 +1,66 @@ +import { and, asc, eq, gt, sql } from "drizzle-orm"; +import { databaseWorkScopeKey, decisions, type Db } from "@paperclipai/db"; + +export type DecisionContinuationWake = (input: { + companyId: string; agentId: string; issueId: string; decisionId: string; + outcome: "decided" | "expired" | "cancelled"; +}) => Promise; + +const flights = new WeakMap>>(); +const ready = sql`${decisions.continuationPolicy} = 'wake_origin_agent' + AND ${decisions.metadata} ->> 'continuationPending' = 'true' + AND (${decisions.status} IN ('expired', 'cancelled') OR + (${decisions.status} = 'decided' AND ${decisions.executionStatus} IN ('succeeded', 'partial', 'failed')))`; + +export function decisionContinuationService(db: Db, wake: DecisionContinuationWake) { + const scope = databaseWorkScopeKey(db); + let pending = flights.get(scope); + if (!pending) flights.set(scope, pending = new Map()); + const inFlight = pending; + let recoveryCursor: string | undefined; + + function deliver(id: string): Promise { + const existing = inFlight.get(id); + if (existing) return existing; + const attempt = (async () => { + const [decision] = await db.select().from(decisions).where(and(eq(decisions.id, id), ready)); + if (!decision) return; + await wake({ companyId: decision.companyId, agentId: decision.originAgentId, + issueId: decision.originIssueId, decisionId: id, + outcome: decision.status === "expired" ? "expired" : decision.status === "cancelled" ? "cancelled" : "decided" }); + await db.update(decisions).set({ + metadata: sql`coalesce(${decisions.metadata}, '{}'::jsonb) || jsonb_build_object( + 'continuationPending', false, 'continuationDeliveredAt', ${new Date().toISOString()}::text)`, + updatedAt: new Date(), + }).where(and(eq(decisions.id, id), ready)); + })().finally(() => { inFlight.delete(id); }); + inFlight.set(id, attempt); + return attempt; + } + + async function sweep(maxRows = 100, signal?: AbortSignal) { + let attempted = 0; + const errors: unknown[] = []; + while (attempted < maxRows) { + if (signal?.aborted) return { retry: true }; + const limit = Math.min(100, maxRows - attempted); + const rows = await db.select({ id: decisions.id }).from(decisions) + .where(and(ready, recoveryCursor ? gt(decisions.id, recoveryCursor) : undefined)) + .orderBy(asc(decisions.id)).limit(limit); + for (const row of rows) { + if (signal?.aborted) return { retry: true }; + try { await deliver(row.id); } catch (error) { errors.push(error); } + recoveryCursor = row.id; + } + attempted += rows.length; + if (rows.length < limit) { recoveryCursor = undefined; break; } + } + if (errors.length === 1) throw errors[0]; + if (errors.length) throw new AggregateError(errors, "Decision continuation delivery failed"); + if (signal?.aborted) return { retry: true }; + const remaining = await db.select({ id: decisions.id }).from(decisions).where(ready).limit(1); + return { retry: remaining.length > 0 }; + } + + return { deliver, sweep }; +} diff --git a/server/src/services/decision-work-notifications.ts b/server/src/services/decision-work-notifications.ts new file mode 100644 index 00000000000..af1ac08487f --- /dev/null +++ b/server/src/services/decision-work-notifications.ts @@ -0,0 +1,9 @@ +import type { Db } from "@paperclipai/db"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export function writeDecisionChange(db: Db, write: (transaction: Db) => Promise): Promise { + return db.transaction(async transaction => { + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionContinuation); + return write(transaction as unknown as Db); + }); +} diff --git a/server/src/services/decisions.ts b/server/src/services/decisions.ts index c9a57063548..e37d1393b76 100644 --- a/server/src/services/decisions.ts +++ b/server/src/services/decisions.ts @@ -1,7 +1,7 @@ import { randomUUID } from "node:crypto"; -import { and, asc, count, desc, eq, gt, gte, inArray, lte, or, sql } from "drizzle-orm"; +import { and, asc, count, desc, eq, gt, gte, inArray, lte, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; -import { companyMemberships, decisionBundles, decisionEffectExecutions, decisionRetention, decisions, decisionTargetIssues, heartbeatRuns, issueRelations, issues } from "@paperclipai/db"; +import { databaseWorkScopeKey, companyMemberships, decisionBundles, decisionEffectExecutions, decisionRetention, decisions, decisionTargetIssues, heartbeatRuns, issueRelations, issues } from "@paperclipai/db"; import { ATTENTION_SOURCE_KINDS, decisionEffectTargetIssueIds } from "@paperclipai/shared"; import type { AttentionArchiveManifestEntry, DecisionEffect, DecisionInput, DecisionOption, DecisionStatsCounts, DecisionStatsResponse } from "@paperclipai/shared"; import { conflict, forbidden, notFound, tooManyRequests, unprocessable } from "../errors.js"; @@ -14,11 +14,16 @@ import { } from "./issues.js"; import { decisionRetentionService, hashAttentionArchiveManifest } from "./decision-retention.js"; +import { decisionContinuationService, type DecisionContinuationWake } from "./decision-continuations.js"; +import { writeDecisionChange } from "./decision-work-notifications.js"; + type Snapshot = { status: string; assigneeAgentId: string | null; assigneeUserId: string | null; updatedAt: string; descendantCount?: number; descendantIds?: string[]; childCount?: number; attentionArchive?: unknown }; -type Wake = (input: { companyId: string; agentId: string; issueId: string; decisionId: string; outcome: "decided" | "expired" | "cancelled" }) => Promise; -export type DecisionServiceOptions = { wakeOriginAgent: Wake }; + +export type DecisionServiceOptions = { wakeOriginAgent: DecisionContinuationWake }; const DAY = 86_400_000; +type DecisionOutcome = typeof decisions.$inferSelect & { executions: Array }; +const effectFlights = new WeakMap>>(); function targetIds(options: DecisionOption[]) { const result = new Set(); @@ -118,6 +123,11 @@ function boardCanActDirectly(actor: AuthorizationActor, companyId: string) { } export function decisionService(db: Db, options: DecisionServiceOptions) { + const continuations = decisionContinuationService(db, options.wakeOriginAgent); + const scope = databaseWorkScopeKey(db); + let flights = effectFlights.get(scope); + if (!flights) effectFlights.set(scope, flights = new Map()); + const executing = flights; const authz = authorizationService(db); let targetSweepCursor: string | null = null; type CreateInput = { companyId: string; actor: AuthorizationActor; agentId: string; runId: string; bundleId?: string | null; @@ -544,7 +554,19 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { } } - async function runEffects(decision: typeof decisions.$inferSelect, userActor: AuthorizationActor) { + function runEffects(decision: typeof decisions.$inferSelect, userActor: AuthorizationActor): Promise { + const existing = executing.get(decision.id); + if (existing) return existing; + const attempt = (async () => { + const current = await get(decision.id); + if (!current) throw notFound("Decision not found"); + return current.executionStatus === "running" ? executeDecisionEffects(current, userActor) : outcome(current.id); + })().finally(() => { executing.delete(decision.id); }); + executing.set(decision.id, attempt); + return attempt; + } + + async function executeDecisionEffects(decision: typeof decisions.$inferSelect, userActor: AuthorizationActor) { const option = decision.options.find((item) => item.id === decision.chosenOptionId); if (!option || !decision.decidedByUserId) throw unprocessable("Stored decision outcome is invalid"); const run = await db.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(eq(heartbeatRuns.id, decision.originRunId)).then((rows) => rows[0] ?? null); @@ -555,8 +577,8 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { options: decision.options, targetSnapshots: decision.targetSnapshots as Record, }), decision.signedSpec)) { - await db.update(decisions).set({ executionStatus: "failed", updatedAt: new Date(), metadata: { ...metadata, archiveProposalError: "invalid_signature" } }) - .where(eq(decisions.id, decision.id)); + await writeDecisionChange(db, async tx => tx.update(decisions).set({ executionStatus: "failed", updatedAt: new Date(), metadata: { ...metadata, archiveProposalError: "invalid_signature", ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }) + .where(and(eq(decisions.id, decision.id), eq(decisions.executionStatus, "running")))); return outcome(decision.id); } if (option.id === "archive") { @@ -566,8 +588,8 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { ); const manifestHash = manifest ? hashAttentionArchiveManifest(manifest) : null; if (!manifest || manifestHash !== metadata.manifestHash) { - await db.update(decisions).set({ executionStatus: "failed", updatedAt: new Date(), metadata: { ...metadata, archiveProposalError: "manifest_mismatch" } }) - .where(eq(decisions.id, decision.id)); + await writeDecisionChange(db, async tx => tx.update(decisions).set({ executionStatus: "failed", updatedAt: new Date(), metadata: { ...metadata, archiveProposalError: "manifest_mismatch", ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }) + .where(and(eq(decisions.id, decision.id), eq(decisions.executionStatus, "running")))); return outcome(decision.id); } const originActor: AuthorizationActor = { @@ -588,50 +610,30 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { }); } catch (error) { const message = error instanceof Error ? error.message : "archive_proposal_failed"; - await db.update(decisions).set({ executionStatus: "failed", updatedAt: new Date(), metadata: { ...metadata, archiveProposalError: message } }) - .where(eq(decisions.id, decision.id)); + await writeDecisionChange(db, async tx => tx.update(decisions).set({ executionStatus: "failed", updatedAt: new Date(), metadata: { ...metadata, archiveProposalError: message, ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }) + .where(and(eq(decisions.id, decision.id), eq(decisions.executionStatus, "running")))); return outcome(decision.id); } } - await db.update(decisions).set({ + await writeDecisionChange(db, async tx => tx.update(decisions).set({ executionStatus: "succeeded", updatedAt: new Date(), metadata: { ...metadata, ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) }, - }).where(eq(decisions.id, decision.id)); + }).where(and(eq(decisions.id, decision.id), eq(decisions.executionStatus, "running")))); return outcome(decision.id); } for (let index = 0; index < option.effects.length; index += 1) await executeEffect(decision, option.effects[index]!, index, userActor, decision.decidedByUserId, run?.responsibleUserId ?? null); const rows = await db.select().from(decisionEffectExecutions).where(eq(decisionEffectExecutions.decisionId, decision.id)); const successful = rows.filter((row) => row.status === "executed").length; const status = rows.every((row) => row.status === "executed") ? "succeeded" : successful ? "partial" : "failed"; - await db.update(decisions).set({ executionStatus: status, updatedAt: new Date(), metadata: { ...decision.metadata, - ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(eq(decisions.id, decision.id)); + await writeDecisionChange(db, async tx => tx.update(decisions).set({ executionStatus: status, updatedAt: new Date(), metadata: { ...decision.metadata, + ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, decision.id), eq(decisions.executionStatus, "running")))); return outcome(decision.id); } - async function deliverContinuation( - decision: typeof decisions.$inferSelect, - outcome: "decided" | "expired" | "cancelled", - ) { - const metadata = decision.metadata as Record; - if (decision.continuationPolicy !== "wake_origin_agent" || metadata.continuationPending !== true) return; - // Delivery is intentionally at-least-once: a crash after wakeup but before - // this acknowledgement can retry, while a crash before wakeup cannot lose - // the continuation. Heartbeat wakeups coalesce concurrent queued work. - await options.wakeOriginAgent({ companyId: decision.companyId, agentId: decision.originAgentId, - issueId: decision.originIssueId, decisionId: decision.id, outcome }); - await db.update(decisions).set({ - metadata: sql`coalesce(${decisions.metadata}, '{}'::jsonb) || jsonb_build_object( - 'continuationPending', false, - 'continuationDeliveredAt', ${new Date().toISOString()}::text - )`, - updatedAt: new Date(), - }).where(and(eq(decisions.id, decision.id), sql`${decisions.metadata} ->> 'continuationPending' = 'true'`)); - } - async function resumeDecision(decision: typeof decisions.$inferSelect, userActor: AuthorizationActor) { const result = decision.executionStatus === "running" ? await runEffects(decision, userActor) : await outcome(decision.id); - await deliverContinuation(result, "decided"); + await continuations.deliver(result.id); return outcome(decision.id); } @@ -651,25 +653,25 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { } if (current.status !== "open") throw conflict("decision_already_resolved", { code: "decision_already_resolved" }); if (current.expiresAt <= new Date()) { - const [expired] = await db.update(decisions).set({ status: "expired", updatedAt: new Date(), metadata: { ...metadata, expiredReason: "ttl", + const [expired] = await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "expired", updatedAt: new Date(), metadata: { ...metadata, expiredReason: "ttl", ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }) - .where(and(eq(decisions.id, current.id), eq(decisions.status, "open"), lte(decisions.expiresAt, new Date()))).returning(); + .where(and(eq(decisions.id, current.id), eq(decisions.status, "open"), lte(decisions.expiresAt, new Date()))).returning()); if (expired) { await logActivity(db, { companyId: expired.companyId, actorType: "system", actorId: "decision-expiry-sweeper", agentId: expired.originAgentId, runId: expired.originRunId, action: "decision.expired", entityType: "decision", entityId: expired.id, details: { expiredReason: "ttl" } }); - await deliverContinuation(expired, "expired"); + await continuations.deliver(expired.id); } throw conflict("decision_expired", { code: "decision_expired" }); } if (!current.options.some((option) => option.id === input.optionId)) throw unprocessable("Unknown optionId"); const values = input.inputValues ?? {}; for (const field of current.inputs ?? []) { const value = values[field.id] ?? ""; if (field.required && !value.trim()) throw unprocessable(`Input ${field.id} is required`); if (field.maxLength && value.length > field.maxLength) throw unprocessable(`Input ${field.id} is too long`); } - const [claimed] = await db.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: input.optionId, inputValues: values, + const [claimed] = await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: input.optionId, inputValues: values, decidedByUserId: input.decidedByUserId, decidedAt: new Date(), updatedAt: new Date(), metadata: { ...metadata, decideIdempotencyKey: input.idempotencyKey ?? null, ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}), ...(input.dismissed ? { dismissed: true, dismissReason: input.dismissReason ?? null } : {}) } }) - .where(and(eq(decisions.id, current.id), eq(decisions.status, "open"))).returning(); + .where(and(eq(decisions.id, current.id), eq(decisions.status, "open"))).returning()); if (!claimed) throw conflict("decision_already_resolved", { code: "decision_already_resolved" }); const run = await db.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(eq(heartbeatRuns.id, claimed.originRunId)).then((rows) => rows[0] ?? null); await logActivity(db, { companyId: claimed.companyId, actorType: "system", actorId: "decision-executor", agentId: claimed.originAgentId, runId: claimed.originRunId, @@ -683,14 +685,14 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { const current = await get(id); if (!current) throw notFound("Decision not found"); if (actor.actorType === "agent" && actor.actorId !== current.originAgentId) throw forbidden("Only the origin agent may cancel"); if (current.status === "cancelled") { - await deliverContinuation(current, "cancelled"); + await continuations.deliver(current.id); return (await get(id))!; } - const [updated] = await db.update(decisions).set({ status: "cancelled", updatedAt: new Date(), metadata: { ...current.metadata, - ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, id), eq(decisions.status, "open"))).returning(); + const [updated] = await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "cancelled", updatedAt: new Date(), metadata: { ...current.metadata, + ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, id), eq(decisions.status, "open"))).returning()); if (!updated) throw conflict("decision_already_resolved", { code: "decision_already_resolved" }); await logActivity(db, { companyId: updated.companyId, actorType: actor.actorType, actorId: actor.actorId, runId: actor.runId, action: "decision.cancelled", entityType: "decision", entityId: id }); - await deliverContinuation(updated, "cancelled"); + await continuations.deliver(updated.id); return (await get(id))!; } @@ -699,14 +701,14 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { if (!verifyDecisionSpec(spec({ id: current.id, options: current.options, targetSnapshots: current.targetSnapshots as Record }), current.signedSpec)) throw forbidden("Decision signature verification failed"); const empty = current.options.find((option) => option.effects.length === 0); if (empty) return decide({ id, optionId: empty.id, decidedByUserId: userId, userActor, dismissed: true, dismissReason: reason }); - const [updated] = await db.update(decisions).set({ status: "decided", executionStatus: "succeeded", chosenOptionId: "dismissed", decidedByUserId: userId, + const [updated] = await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "decided", executionStatus: "succeeded", chosenOptionId: "dismissed", decidedByUserId: userId, decidedAt: new Date(), updatedAt: new Date(), metadata: { ...current.metadata, dismissed: true, dismissReason: reason ?? null, - ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, id), eq(decisions.status, "open"))).returning(); + ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, id), eq(decisions.status, "open"))).returning()); if (!updated) throw conflict("decision_already_resolved", { code: "decision_already_resolved" }); await logActivity(db, { companyId: updated.companyId, actorType: "system", actorId: "decision-executor", agentId: updated.originAgentId, runId: updated.originRunId, responsibleUserIdOverride: userId, action: "decision.dismissed", entityType: "decision", entityId: updated.id, details: { chosenOptionId: "dismissed", decidedByUserId: userId, dismissed: true } }); - await deliverContinuation(updated, "decided"); + await continuations.deliver(updated.id); return outcome(id); } @@ -718,7 +720,7 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { return { ...bundle, decisions: created }; }); } - async function sweepExpired(now = new Date()) { + async function sweepExpired(now = new Date(), options: { includeContinuations?: boolean } = {}) { const configuredBatchSize = Number(process.env.PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE ?? 100); const batchSize = Number.isFinite(configuredBatchSize) ? Math.max(1, Math.trunc(configuredBatchSize)) @@ -737,17 +739,9 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { const recovered = await runEffects(decision, await recoveryActor(decision.companyId, decision.decidedByUserId)); if (recovered.executionStatus === "running") continue; resumed += 1; - await deliverContinuation(recovered, "decided"); - } - const pendingContinuations = await db.select().from(decisions) - .where(and(eq(decisions.continuationPolicy, "wake_origin_agent"), - sql`${decisions.metadata} ->> 'continuationPending' = 'true'`, - or(inArray(decisions.status, ["expired", "cancelled"]), and(eq(decisions.status, "decided"), - inArray(decisions.executionStatus, ["succeeded", "partial", "failed"]))))) - .orderBy(asc(decisions.updatedAt)).limit(batchSize); - for (const decision of pendingContinuations) { - await deliverContinuation(decision, decision.status === "expired" ? "expired" : decision.status === "cancelled" ? "cancelled" : "decided"); + await continuations.deliver(recovered.id); } + if (options.includeContinuations !== false) await continuations.sweep(batchSize); const ttlRows = await db.select().from(decisions) .where(and(eq(decisions.status, "open"), lte(decisions.expiresAt, now))) .orderBy(asc(decisions.expiresAt)).limit(batchSize); @@ -774,13 +768,13 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { targets.some((target) => snapshots[target.id]?.status !== "done"); if (!targetGone && !targetsCompleted && decision.expiresAt >= now) continue; const reason = targetGone ? "target_gone" : targetsCompleted ? "target_completed" : "ttl"; - const [updated] = await db.update(decisions).set({ status: "expired", updatedAt: now, metadata: { ...decision.metadata, expiredReason: reason, - ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, decision.id), eq(decisions.status, "open"))).returning(); + const [updated] = await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "expired", updatedAt: now, metadata: { ...decision.metadata, expiredReason: reason, + ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, decision.id), eq(decisions.status, "open"))).returning()); if (!updated) continue; expired += 1; await logActivity(db, { companyId: updated.companyId, actorType: "system", actorId: "decision-expiry-sweeper", agentId: updated.originAgentId, runId: updated.originRunId, action: "decision.expired", entityType: "decision", entityId: updated.id, details: { expiredReason: reason } }); - await deliverContinuation(updated, "expired"); } + await continuations.deliver(updated.id); } return { expired, resumed }; } - return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepExpired }; + return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepExpired, sweepContinuations: continuations.sweep }; } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 230be70dcf8..31ece3af9fb 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,8 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + decisionContinuation: "decision-continuation", + decisionArchiveDelivery: "decision-archive-delivery", toolReviewProjection: "tool-review-projection", From 64a42d2cf04ff839f56659a99b5ef26751aac375 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:52:46 -0500 Subject: [PATCH 25/41] fix(evals): honor the Daytona setup attempt deadline (#17947) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://entire.com/gh/paperclipai/-/trails/110/changes/paperclip/110 ## Thinking Path > - Budgeted Product E2E cases must attribute asynchronous hello probes to setup. > - Daytona can take longer than local startup to provision an isolated environment. > - The setup helper imposed a 45-second local limit on both environments. > - A qualified remote marker stopped while the agent was still verifying. > - Remote readiness should use its existing finite attempt deadline. ## Linked Issues or Issue Description Refs #17936. **What happened?** The bounded Daytona marker campaign stopped at the setup helper after 45 seconds while the agent was still verifying. The case permits a 15-minute attempt and the environment allows up to five minutes to provision. It recorded zero model requests; retained cleanup passed and a provider-side company-label query confirmed no sandbox remained. **Expected behavior** Daytona readiness can finish within the existing attempt budget. Local setup keeps the current short limit; observed errors, holds and identity changes still stop either environment immediately. **Steps to reproduce** Run a budgeted native marker against a cold Daytona environment whose readiness takes longer than 45 seconds. ## What Changed - Pass the execution environment to the budgeted setup helper. - Use the existing finite attempt deadline for Daytona; retain the 45-second local limit. - Set a five-minute saved-account hello-probe allowance only for explicit budgeted Daytona fixtures, matching the environment provisioning allowance. A later qualification exposed the inner probe’s separate 90-second timeout. - Keep the gateway in the setup phase until the same agent is ready. Model limits, attempt limits, cleanup and automatic retries are unchanged. - Update the operations guide with the two timing windows. ## Verification - Focused setup and operations regression suites: 48 passed. - Eval TypeScript check passed. - A fixture regression verifies the five-minute probe allowance is remote-only and preserves gateway credential routing. The timing regression advances the observed clock past 45 seconds: Daytona reaches ready while local setup still stops. Existing tests retain identity/error/hold and attempt-deadline guards. - Fresh real Daytona qualification follows this correction with one attempt and the existing $20 stage cap. ## Risks A failed remote setup can now consume more of its existing attempt window. It cannot extend that deadline, increase model allowance, or automatically retry. This change does not claim the prior failed campaign passed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/OPERATIONS.md | 5 ++++- tests/runner-e2e/budgeted-setup.test.ts | 30 +++++++++++++++++++++---- tests/runner-e2e/budgeted-setup.ts | 4 +++- tests/runner-e2e/live-fixtures.ts | 1 + tests/runner-e2e/operations.test.ts | 16 +++++++++---- tests/runner-e2e/runner.spec.ts | 2 +- 6 files changed, 47 insertions(+), 11 deletions(-) diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index be61693fb07..53dfa5b570f 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -440,8 +440,11 @@ attempt. Installing the ACP package alone does not satisfy this setup path. Bounded cases keep the gateway in the setup phase until the created agent's asynchronous lifecycle verification reaches ready. This wait uses the remaining -attempt deadline, allows at most 45 seconds, and stops on an observed setup error +attempt deadline, allows at most 45 seconds locally and the bounded attempt window +for Daytona provisioning, and stops on an observed setup error or hold. The private `budgeted-setup.json` snapshot retains state transitions. +Budgeted Daytona fixtures give the saved-account hello probe five minutes inside +that attempt window; its former 90-second default could expire during cold startup. Only then does the case begin agent-phase accounting and create its task. Older evidence that advanced after fixture creation may label hello probes as agent cost. Its campaign totals still include those requests; retain that phase diff --git a/tests/runner-e2e/budgeted-setup.test.ts b/tests/runner-e2e/budgeted-setup.test.ts index f7ddaf2c5b0..c1c7e4dacb7 100644 --- a/tests/runner-e2e/budgeted-setup.test.ts +++ b/tests/runner-e2e/budgeted-setup.test.ts @@ -21,7 +21,7 @@ it("keeps asynchronous hello probes in setup until the same agent is ready", asy const get = vi.fn().mockResolvedValueOnce({ ...ready, status: "paused", lifecycleState: "verifying", adapterConfig: { private: true } }).mockResolvedValue(ready); let saved: EvalSetupObservation[] = []; try { - await finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company", + await finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company", deadlineAt: Date.now() + 5_000, api: { get } as unknown as RunnerApi, observe: async observations => { expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup"); @@ -43,7 +43,7 @@ it.each([ ] as const)("preserves setup attribution and the failed state when readiness is rejected: %s", async (state, reason) => { const f = await fixture(), observe = vi.fn(async () => {}), get = vi.fn().mockResolvedValue(state); try { - await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company", + await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company", deadlineAt: Date.now() + 5_000, api: { get } as unknown as RunnerApi, observe }, f.env)).rejects.toThrow(reason); expect(get).toHaveBeenCalledOnce(); expect(observe).toHaveBeenCalledOnce(); @@ -54,7 +54,7 @@ it.each([ it("does not start a readiness request after the attempt deadline", async () => { const f = await fixture(), get = vi.fn(); try { - await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company", + await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company", deadlineAt: Date.now() - 1, api: { get } as unknown as RunnerApi, observe: async () => {} }, f.env)).rejects.toThrow("Timed out"); expect(get).not.toHaveBeenCalled(); expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup"); @@ -66,10 +66,32 @@ it("does not advance when saving readiness crosses the deadline", async () => { const deadlineAt = Date.now() + 5_000; const now = vi.spyOn(Date, "now"); try { - await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, agentId: "agent", companyId: "company", + await expect(finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId: "local", agentId: "agent", companyId: "company", deadlineAt, api: { get } as unknown as RunnerApi, observe: async () => { now.mockReturnValue(deadlineAt); } }, f.env)).rejects.toThrow("Timed out saving"); expect(get).toHaveBeenCalledOnce(); expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup"); } finally { now.mockRestore(); await rm(f.directory, { recursive: true, force: true }); } }); + +it.each(["local", "daytona"] as const)("uses the bounded attempt window for remote setup while keeping the local limit: %s", async environmentId => { + const f = await fixture(); + let time = Date.now(); + const now = vi.spyOn(Date, "now").mockImplementation(() => time); + const get = vi.fn().mockResolvedValueOnce({ ...ready, status: "paused", lifecycleState: "verifying" }).mockResolvedValue(ready); + let observations = 0; + try { + const result = finishBudgetedEvalSetup({ executionId: identity.executionId, environmentId, agentId: "agent", companyId: "company", + deadlineAt: time + 15 * 60_000, api: { get } as unknown as RunnerApi, + observe: async () => { if (++observations === 1) time += 46_000; } }, f.env); + if (environmentId === "daytona") { + await result; + expect(get).toHaveBeenCalledTimes(2); + expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("agent"); + } else { + await expect(result).rejects.toThrow("Timed out"); + expect(get).toHaveBeenCalledOnce(); + expect((await readEvalAttemptPhase(f.file, identity)).phase).toBe("setup"); + } + } finally { now.mockRestore(); await rm(f.directory, { recursive: true, force: true }); } +}); diff --git a/tests/runner-e2e/budgeted-setup.ts b/tests/runner-e2e/budgeted-setup.ts index 42dc216ea50..ee69773da95 100644 --- a/tests/runner-e2e/budgeted-setup.ts +++ b/tests/runner-e2e/budgeted-setup.ts @@ -1,3 +1,4 @@ +import type { RunnerEnvironmentId } from "./types.js"; import type { Agent } from "../../packages/shared/src/types/agent.js"; import { pollUntil, type RunnerApi } from "./api.js"; import { beginEvalAgentPhase } from "./attempt-phase.js"; @@ -7,6 +8,7 @@ export type EvalSetupObservation = SetupState & { observedAt: string }; export async function finishBudgetedEvalSetup(input: { executionId: string; + environmentId: RunnerEnvironmentId; agentId: string; companyId: string; deadlineAt: number; @@ -14,7 +16,7 @@ export async function finishBudgetedEvalSetup(input: { observe: (observations: EvalSetupObservation[]) => Promise; }, env: NodeJS.ProcessEnv = process.env) { if (env.PAPERCLIP_E2E_GATEWAY_BASE_URL) { - const deadlineAt = Math.min(input.deadlineAt, Date.now() + 45_000); + const deadlineAt = input.environmentId === "daytona" ? input.deadlineAt : Math.min(input.deadlineAt, Date.now() + 45_000); const observations: EvalSetupObservation[] = []; let previous: string | undefined; const scoped = (agent: SetupState) => agent.id === input.agentId && agent.companyId === input.companyId; diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index 517b25b3d90..29cdb4ff214 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -385,6 +385,7 @@ export async function setupLiveFixtures(input: { if (managedHiring || savedCopilot || budgetedGateway) { const account = value(resolved, "ai-connection"); const config = agent.adapterConfig as Record; + if (budgetedGateway && execution.environment.id === "daytona") config.helloProbeTimeoutSec = 300; delete config.env; agent.runtimeConfig = { ...(agent.runtimeConfig as Record), diff --git a/tests/runner-e2e/operations.test.ts b/tests/runner-e2e/operations.test.ts index 8e03148dfdc..74cff67c76f 100644 --- a/tests/runner-e2e/operations.test.ts +++ b/tests/runner-e2e/operations.test.ts @@ -164,16 +164,21 @@ it.each([ cell.id, "feedback-persistence.runner-codex.local.complaint-and-suggestion", "feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion", + "core-compatibility.runner-acpx-claude.daytona.message-marker", ])("routes %s through a shared connection without ambient provider bindings", async executionId => { const selected = runnerMatrix.find(execution => execution.id === executionId)!; const anthropic = selected.profile.credential === "ANTHROPIC_API_KEY"; - const admitted = manifest(); admitted.executionIds = [selected.id]; + const remote = selected.environment.id === "daytona"; + const origin = remote ? "https://fixture.trycloudflare.com" : "http://127.0.0.1:43123"; + const admitted = manifest(); + if (remote) admitted.remoteGateway = "cloudflare"; admitted.executionIds = [selected.id]; admitted.models[0]!.provider = anthropic ? "anthropic" : "openai"; admitted.models[0]!.price.provider = admitted.models[0]!.provider; admitted.models[0]!.price.model = selected.profile.model; if (anthropic) Object.assign(admitted.models[0]!.price, { cacheWrite5mUsdPerMillion: "2", cacheWrite1hUsdPerMillion: "2" }); expect(validateEvalOperationsManifest(admitted, [selected], 0)).toBe(admitted); - vi.stubEnv("PAPERCLIP_E2E_GATEWAY_BASE_URL", "http://127.0.0.1:43123"); + vi.stubEnv("PAPERCLIP_E2E_GATEWAY_BASE_URL", origin); + if (remote) vi.stubEnv("PAPERCLIP_E2E_REMOTE_GATEWAY_ORIGIN", origin); vi.stubEnv("PAPERCLIP_E2E_GATEWAY_TOKEN", "ephemeral-fixture-token"); const id = "11111111-1111-4111-8111-111111111111", grantId = "22222222-2222-4222-8222-222222222222"; let connection: unknown, agent: Record | undefined, company: Record | undefined; @@ -181,12 +186,15 @@ it.each([ async get() { return [{ id, driver: "local" }]; }, async postSensitive(url: string, body: unknown) { if (url.endsWith("/ai-connections")) { connection = body; return { connectionId: id, grantId }; } return { id }; }, async post(url: string, body: Record) { if (url.endsWith("/agents")) agent = body; if (url === "/api/companies") company = body; return { id, name: "fixture" }; }, + async delete() {}, } as unknown as RunnerApi; - const fixtures = await setupLiveFixtures({ api, execution: selected, executionNonce: "fixture", workspacePath: "/tmp/fixture", credentials: { [selected.profile.credential]: "ephemeral-fixture-token" } }); + const fixtures = await setupLiveFixtures({ api, execution: selected, executionNonce: "fixture", workspacePath: "/tmp/fixture", credentials: { [selected.profile.credential]: "ephemeral-fixture-token", ...(remote ? { DAYTONA_API_KEY: "fixture-daytona" } : {}) }, + ...(remote ? { daytonaImage: `ghcr.io/paperclip/image@sha256:${"a".repeat(64)}` } : {}) }); await fixtures.teardown(); - expect(createAiConnectionSchema.parse(connection)).toMatchObject({ ownership: "shared", apiKey: "ephemeral-fixture-token", routing: { kind: "gateway", protocol: anthropic ? "messages" : "responses", baseUrl: `http://127.0.0.1:43123${anthropic ? "" : "/v1"}`, auth: "bearer" } }); + expect(createAiConnectionSchema.parse(connection)).toMatchObject({ ownership: "shared", apiKey: "ephemeral-fixture-token", routing: { kind: "gateway", protocol: anthropic ? "messages" : "responses", baseUrl: `${origin}${anthropic ? "" : "/v1"}`, auth: "bearer" } }); expect(aiConnectionBindingSchema.parse(agent!.runtimeConfig.aiConnection)).toEqual({ provider: anthropic ? "anthropic" : "openai", method: "api_key", mode: "shared", connectionId: id, grantId }); expect(agent!.adapterConfig.env).toBeUndefined(); + expect(agent!.adapterConfig.helloProbeTimeoutSec).toBe(remote ? 300 : undefined); if (selected.suite.id === "feedback-persistence") { expect(company!.budgetMonthlyCents).toBe(1_000); expect(agent!.budgetMonthlyCents).toBe(1_000); diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 8836e27dfe8..1584725be40 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -957,7 +957,7 @@ for (const execution of executions) { daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, onCompanyCreated, }); - await finishBudgetedEvalSetup({ executionId: execution.id, agentId: fixtures.agent.id, + await finishBudgetedEvalSetup({ executionId: execution.id, environmentId: execution.environment.id, agentId: fixtures.agent.id, companyId: fixtures.company.id, deadlineAt: startedAtMs + deadlineMs, api, observe: observations => writeSanitizedJson(snapshotsDir, "budgeted-setup.json", { observations }, secrets) }); await workerTiming.enter("execution"); From 7a8df3095365da84b6871933fc20a29316fa8834 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 17:53:58 -0500 Subject: [PATCH 26/41] fix(runner): admit Grok effort through native session metadata (#17942) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip lets people manage AI agents for work. > - Paperclip Runner must preserve each agent's model and thinking effort. > - The Grok migration in #17922 passes explicit effort to native Runner. > - The pinned Grok 1.0.13 CLI reports native session metadata instead of standard ACP config options. > - The standard effort guard rejects that valid session before the first prompt. > - This pull request binds model and effort on Grok session open and verifies the native reply. > - Restored sessions keep the selected effort instead of reverting to Grok's default. ## Linked Issues or Issue Description Related: https://github.com/paperclipai/paperclip/pull/17922 **What happened?** A migrated Grok 4.7 agent with High effort fails at native session.open before it can run a tool. Grok 1.0.13 has no standard configOptions response and rejects session/set_config_option. Its native session metadata supports the requested model and effort. A cold load also resets effort to High unless the request supplies it again. The upgrade banner also calls Grok “Codex”. **Expected behavior** The native session must acknowledge the selected model and effort before a prompt. A cold restore must preserve both values. The upgrade banner must describe the selected harness accurately. **Steps to reproduce** 1. Upgrade a Grok agent to Paperclip Runner with explicit High effort. 2. Assign a bounded shell task. The pre-fix native session fails during admission. 3. For the restore case, create a Low native session, close the process, and load it without effort metadata. The pinned CLI reports High. **Paperclip version or commit** Observed at f60ac29cf116015deb64eb8a1f24d02f6f35fe56 with Grok 1.0.13. **Deployment mode** Cloud, with the board operator interface. ## What Changed - Project the bound model and effort into native Grok session/new, session/load, and session/resume metadata before transport writes and observation. - Require a correlated native acknowledgement with the exact model, effort, and session identity. Reject conflicting metadata, selection drift, stale connections, and prompts before admission. - Connect lazy restored handles through read-only native session inspection. Keep the existing process ownership and launch snapshot checks. - Replace Grok's unsupported standard config setter. Keep the existing Claude effort path. - Add behavioral guard cases and a real ACPX client fixture for fresh and cold Low/High sessions. The fixture resets effort on restore if metadata is absent, as the pinned CLI does. - Publish Copilot profile v69 because its source identity includes the shared runtime adapter. Preserve published v66/v67/v68 bytes and the reviewed Grok v67 candidate as history. Keep old profiles closed for launch. - Explain the native protocol in the Grok Runner guide and use provider-independent upgrade banner text. ## Verification - Three native create/load/resume cases failed before the repair. The pre-fix guard rejected the native metadata reply. - Seven affected Runner test files: 234 tests passed after the master merge. These include native Grok admission, the real ACPX transport fixture, runtime ownership, Claude effort, and profile history. - The review follow-up extends the same Low/High fixture through a complete first turn after cold restore. It checks the repeated native load, original session id, actual response event, and completed turn. All 127 tests in the three affected Grok/adapter files passed again; Runner typecheck and source identity checks also passed. - All 167 existing agent form tests passed. Token gates and diff checks passed. - Workspace typecheck passed. Final Runner typecheck and generated profile checks passed. - Production `pnpm build` passed, including after the master conflict resolution. Runner typecheck, profile history checks, and the 234 affected tests passed on the merged source. All 55 Rust ACPX library tests also passed on the merged source. - Real pinned Grok 1.0.13 metadata probes reproduced the unsupported standard setter and native create/load/resume behavior. No model prompt was sent. An earlier actual ACPX runtime probe passed fresh and cold Low/High metadata checks. A repeat with an explicitly isolated HOME failed authentication before session creation, so it does not count as final live acceptance. The deterministic transport fixture is green; a deployed shell task remains required. - `pnpm test:run` did not complete locally. Embedded Postgres failed during bootstrap after five attempts. This is the same local fixture limitation recorded for #17922. Fresh PR CI must pass before merge. - After deployment, retry the original bounded shell task with the saved Grok 4.7/High configuration. Verify actual tool output, a successful run, and Done state after reload. - Final commit `810d84fd19728e7d798989582f26b05f69166db2`: all CI gates passed. Greptile and the formal Paperclip review both reached 5/5. No unresolved review threads remain. ## Risks - This depends on the pinned Grok 1.0.13 native metadata format. Missing or conflicting replies stop admission instead of silently changing effort. - Cold restores must make one read-only native inspection call before admission. The existing handshake timeout and process ownership checks still apply. - The shared adapter changes Copilot's source identity. The new current profile includes the final source hashes; the prior profile remains decodable but cannot launch. - This does not change the Grok CLI version, account selection, search policy, dependency patch, or lockfile. ## Model Used OpenAI Codex, GPT-6. The exact serving model ID and context window are not exposed in this session. The agent used reasoning, code editing, shell tools, and browser tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/grok-native-runner.md | 12 +- packages/paperclip-runner/acpx-profiles.json | 4 +- .../src/generated_acpx_profiles.rs | 2 +- .../scripts/copilot-profile-identity.mjs | 4 +- .../acpx/codex-runtime-adapter.test.ts | 36 +++-- .../src/drivers/acpx/codex-runtime-adapter.ts | 12 +- .../drivers/acpx/copilot-installation.test.ts | 22 ++- .../src/drivers/acpx/generated-profiles.ts | 4 +- .../src/drivers/acpx/grok-effort.test.ts | 133 ++++++++++++------ .../src/drivers/acpx/grok-effort.ts | 86 ++++++++++- .../acpx/grok-native-session-options.test.ts | 89 ++++++++++++ .../src/drivers/acpx/profile-compatibility.ts | 4 +- ...-effort-candidate-historical-identity.json | 85 +++++++++++ .../copilot-profile-v69-identity.json | 85 +++++++++++ ui/src/components/AgentConfigForm.tsx | 2 +- 15 files changed, 499 insertions(+), 81 deletions(-) create mode 100644 packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v67-grok-native-effort-candidate-historical-identity.json create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v69-identity.json diff --git a/doc/grok-native-runner.md b/doc/grok-native-runner.md index e9a820d1be4..851c60d23c4 100644 --- a/doc/grok-native-runner.md +++ b/doc/grok-native-runner.md @@ -17,9 +17,15 @@ Grok Build's qualified harness does not require the `enableNativeRunner` flag. Explicit thinking effort uses `reasoningEffort` (`minimal`, `low`, `medium`, `high`, or `xhigh`). The immutable v10 execution input, provider descriptor, -and recovery identity bind this value as `grokEffort`. Before a prompt, the -Runner applies ACP `reasoning_effort` and requires Grok to acknowledge the exact -value. Missing configuration, a rejected value, or effort drift stops admission. +and recovery identity bind this value as `grokEffort`. The Grok provider boundary +adds native `_meta.modelId` and `_meta.reasoningEffort` to session create, load, +and resume requests before transport writes and observation. Grok 1.0.13 reports +the selected effort in `_meta["x.ai/sessionConfig"].options`; it does not support +the standard `session/set_config_option` effort API. The Runner checks that native +acknowledgement and the effective model before a prompt. A lazy restored handle +must connect through read-only native session inspection first. Reapplying the +bound metadata on restore prevents Grok from resetting effort to its default. +Missing configuration, a rejected value, or effort drift stops admission. Omitting effort retains Grok's provider default. This setting remains outside the model-visible task prompt. Historical execution inputs retain their schemas. diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index 0e5443132cc..c96818e7445 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 68, + "agentProfileVersion": 69, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63", + "commandDigest": "sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index 5e9a354c50c..f99ff39b7a0 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63", + "sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index c7d9b1143b7..21737c1085a 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v68-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v69-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 68 }; +const declaration = { ...historical.declaration, agentProfileVersion: 69 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts index 1f77dc18833..76f0d7c914e 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.test.ts @@ -69,36 +69,44 @@ describe("Codex ACPX runtime adapter", () => { await port.close({ reason: "test complete" }); }); it.each([false, true])("acknowledges configured Grok effort before admission (restoring=%s)", async restoring => { - const runtime = fakeRuntime(); + const runtime = Object.assign(fakeRuntime(), { requestExtension: vi.fn(async () => ({})) }); const child = fakeChild(); const options = openOptions({ spawn: () => child, close: async () => undefined }); - options.profile = { ...options.profile, agent: "grok" }; + options.profile = resolveQualifiedAcpxProfile("grok", "grok-4.7"); options.grokEffort = "high"; options.restoringSession = restoring; options.refreshConsumedCommand = vi.fn(async () => undefined); let created!: AcpRuntimeOptions; const spawnProvider = () => created.spawnAgent!({ command: "ignored", args: [], options: {} }); - vi.mocked(runtime.ensureSession).mockImplementation(async () => { - if (!restoring) spawnProvider(); - return HANDLE; - }); - vi.mocked(runtime.setConfigOption).mockImplementation(async input => { - if (restoring) spawnProvider(); + const acknowledge = () => { const guard = created.protocolGuardFactory!(); guard("outbound", { id: 0, method: "session/load", params: { sessionId: "backend-1" } }); - guard("inbound", { id: 0, result: { configOptions: [{ id: "reasoning_effort", currentValue: "default" }] } }); - guard("outbound", { id: 1, method: "session/set_config_option", params: { sessionId: "backend-1", configId: input.key, value: input.value } }); - guard("inbound", { id: 1, result: { configOptions: [{ id: "reasoning_effort", currentValue: input.value }] } }); + guard("inbound", { id: 0, result: { models: { currentModelId: "grok-4.7" }, + _meta: { "x.ai/sessionConfig": { options: [{ id: "high", category: "mode", selected: true }] } } } }); + }; + vi.mocked(runtime.ensureSession).mockImplementation(async input => { + expect(input.sessionOptions).toMatchObject({ model: "grok-4.7" }); + if (!restoring) { spawnProvider(); acknowledge(); } + return HANDLE; + }); + runtime.requestExtension.mockImplementation(async () => { + spawnProvider(); acknowledge(); return {}; }); const port = await openCodexAcpxRuntime(options, { createRegistry: () => registry(), createStore: () => store(), createRuntime: value => { created = value; return runtime; }, awaitProviderOwnership: providerOwnershipEstablished, awaitProviderExit: providerOwnershipEstablished, }); - expect(runtime.setConfigOption).toHaveBeenCalledExactlyOnceWith({ handle: HANDLE, key: "reasoning_effort", value: "high" }); + expect(runtime.setConfigOption).not.toHaveBeenCalled(); expect(await port.identity()).toMatchObject({ grokEffort: "high" }); - if (restoring) expect(options.refreshConsumedCommand).toHaveBeenCalledOnce(); - else expect(options.refreshConsumedCommand).not.toHaveBeenCalled(); + if (restoring) { + expect(runtime.requestExtension).toHaveBeenCalledExactlyOnceWith({ handle: HANDLE, method: "_x.ai/session/info", + params: { sessionId: "backend-1" }, sessionMode: "persistent" }); + expect(options.refreshConsumedCommand).toHaveBeenCalledOnce(); + } else { + expect(runtime.requestExtension).not.toHaveBeenCalled(); + expect(options.refreshConsumedCommand).not.toHaveBeenCalled(); + } expect(runtime.startTurn).not.toHaveBeenCalled(); await port.close({ reason: "test complete" }); }); diff --git a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts index b8d804d7ee1..ac4fb23f740 100644 --- a/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts +++ b/packages/paperclip-runner/src/drivers/acpx/codex-runtime-adapter.ts @@ -323,7 +323,7 @@ export async function openQualifiedAcpxRuntime( const selectedGrokEffort = resolveGrokEffort(options.profile.agent, options.grokEffort); const grokDisableWebSearch = resolveGrokDisableWebSearch(options.profile.agent, options.grokDisableWebSearch); const effortAdmission = selectedClaudeEffort ? createClaudeEffortAdmission(selectedClaudeEffort) - : selectedGrokEffort ? createGrokEffortAdmission(selectedGrokEffort) : undefined; + : selectedGrokEffort ? createGrokEffortAdmission(selectedGrokEffort, options.profile.reportedModelId) : undefined; const cursorInstructions = options.profile.agent === "cursor" ? createCursorInstructionAdmission(options.systemInstructions) : null; @@ -565,9 +565,15 @@ export async function openQualifiedAcpxRuntime( handle = ensuredHandle; options.signal?.throwIfAborted(); if (!effortAdmission.isReady()) { - if (!runtime.setConfigOption) throw new Error("Effort admission requires native configuration"); const spawnsBeforeControl = commandLaunches.count; - await runtime.setConfigOption({ handle: ensuredHandle, key: selectedGrokEffort ? "reasoning_effort" : "effort", value: (selectedGrokEffort ?? selectedClaudeEffort)! }); + if (selectedGrokEffort) { + if (!runtime.requestExtension) throw new Error("Grok admission requires native session inspection"); + await runtime.requestExtension({ handle: ensuredHandle, method: "_x.ai/session/info", + params: { sessionId: requireIdentity(ensuredHandle).backendSessionId }, sessionMode: "persistent" }); + } else { + if (!runtime.setConfigOption) throw new Error("Effort admission requires native configuration"); + await runtime.setConfigOption({ handle: ensuredHandle, key: "effort", value: selectedClaudeEffort! }); + } effortAdmission.assertReady(); await children.verifyLifetimeOwnership(); options.signal?.throwIfAborted(); diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 35d3225b5f5..4436f17831c 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,27 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(68); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(69); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719"); + }); + + it("preserves published and candidate history while binding native Grok admission in v69", async () => { + const published = readFileSync(new URL("../../../test/fixtures/copilot-profile-v68-identity.json", import.meta.url)); + const candidate = readFileSync(new URL("../../../test/fixtures/copilot-profile-v67-grok-native-effort-candidate-historical-identity.json", import.meta.url)); + expect(createHash("sha256").update(published).digest("hex")).toBe("9692e8de4e75682c74f8e635e26b8de1f03f1d3e41966dafbb7314d35b75c3cf"); + expect(createHash("sha256").update(candidate).digest("hex")).toBe("0a20ffe33c92d86f58013582b19c2371f96d93e7e4e0e8096fbf0e7236bd02f4"); + const prior = JSON.parse(published.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v69-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "agentProfileVersion", "profileCompatibilitySourceSha256", "runtimeAdapterSourceSha256", + ]); + expect(Object.keys(current.declaration).sort()).toEqual(Object.keys(prior.declaration).sort()); + for (const history of [prior, JSON.parse(candidate.toString("utf8"))]) { + expect(isSupportedAcpxProfileVersion("copilot", history.declaration.agentProfileVersion)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: history.declaration.agentProfileVersion, commandDigest: history.commandDigest })).rejects.toThrow("exact pinned"); + } }); it("preserves both lineages while binding the merged implementation in v68", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index 7b94907809f..9fcf232a819 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 68, + "agentProfileVersion": 69, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:1ce8bb0e00df81fa9eb71cff5063ce769637f5db7b3aa0ada9ade651db089c63", + "commandDigest": "sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts b/packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts index 98308f3c6e1..f51bd127bd7 100644 --- a/packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/grok-effort.test.ts @@ -1,19 +1,37 @@ import { describe, expect, it } from "vitest"; import { admittedGrokEffort, createGrokEffortAdmission, resolveGrokEffort } from "./grok-effort.js"; -const response = (value: string, sessionId?: string) => ({ ...(sessionId ? { sessionId } : {}), configOptions: [{ id: "reasoning_effort", currentValue: value }] }); -function loaded(method = "session/new") { - const admission = createGrokEffortAdmission("high"); +function response(value: string, sessionId?: string) { + return { ...(sessionId ? { sessionId } : {}), models: { currentModelId: "grok-4.7" }, + _meta: { "x.ai/sessionConfig": { options: [ + { id: "grok-4.7", category: "model", selected: true }, + { id: value, category: "mode", selected: true }, + ] } } }; +} +function loaded(method = "session/new", expected: "low" | "high" = "high") { + const admission = createGrokEffortAdmission(expected, "grok-4.7"); const guard = admission.createGuard(); - guard("outbound", { id: 1, method, params: method === "session/new" ? {} : { sessionId: "s" } }); - guard("inbound", { id: 1, result: response("medium", method === "session/new" ? "s" : undefined) }); + guard("outbound", { id: 1, method, params: { sessionId: "s" } }); + guard("inbound", { id: 1, result: response(expected, method === "session/new" ? "s" : undefined) }); return { admission, guard }; } -function select(guard: ReturnType["createGuard"]>, acknowledged = "high") { - guard("outbound", { id: 2, method: "session/set_config_option", params: { sessionId: "s", configId: "reasoning_effort", value: "high" } }); - guard("inbound", { id: 2, result: response(acknowledged) }); -} + describe("Grok effort admission", () => { + it.each(["session/new", "session/load", "session/resume"])("admits the pinned Grok native metadata after %s", method => { + const { admission, guard } = loaded(method); + expect(admission.isReady()).toBe(true); + expect(() => guard("outbound", { id: 2, method: "session/prompt", params: { sessionId: "s" } })).not.toThrow(); + }); + it.each(["session/new", "session/load", "session/resume"])("binds native model and effort without replacing instructions on %s", method => { + const admission = createGrokEffortAdmission("low", "grok-4.7"), guard = admission.createGuard(); + const request = { id: 1, method, params: { sessionId: "s", _meta: { rules: "Bound task instructions" } } }; + guard("outbound", request); + expect(request.params._meta).toEqual({ rules: "Bound task instructions", modelId: "grok-4.7", reasoningEffort: "low" }); + }); + it.each([{ modelId: "other-model" }, { reasoningEffort: "high" }])("rejects conflicting native request configuration", meta => { + const admission = createGrokEffortAdmission("low", "grok-4.7"), guard = admission.createGuard(); + expect(() => guard("outbound", { id: 1, method: "session/new", params: { _meta: meta } })).toThrow("request conflicts"); + }); it.each(["minimal", "low", "medium", "high", "xhigh"] as const)("accepts exact %s without aliases", value => { expect(resolveGrokEffort("grok", value)).toBe(value); expect(admittedGrokEffort("grok", value, { grokEffort: value }, { grokEffort: value })).toBe(value); @@ -23,45 +41,72 @@ describe("Grok effort admission", () => { expect(() => resolveGrokEffort("pi", "high")).toThrow("only supported"); expect(() => admittedGrokEffort("grok", "high", { grokEffort: "high" }, { grokEffort: "medium" })).toThrow("differs"); }); - it.each(["session/new", "session/load", "session/resume"])("requires acknowledged effort before prompting after %s", method => { - const { admission, guard } = loaded(method); - expect(admission.isReady()).toBe(false); - select(guard); - expect(admission.isReady()).toBe(true); - expect(() => guard("outbound", { id: 3, method: "session/prompt", params: { sessionId: "s" } })).not.toThrow(); - }); - - it("rejects a resumed session with missing effort configuration", () => { - const admission = createGrokEffortAdmission("high"); - const guard = admission.createGuard(); - guard("outbound", { id: 1, method: "session/resume", params: { sessionId: "s" } }); - expect(() => guard("inbound", { id: 1, result: {} })).toThrow("missing or ambiguous"); - }); - it("blocks a prompt based only on a restored default", () => { - const { guard } = loaded("session/load"); - expect(() => guard("outbound", { id: 3, method: "session/prompt", params: { sessionId: "s" } })).toThrow("does not acknowledge"); - }); - - it("requires the bound effort acknowledgement on each cold connection", () => { + it("requires the native acknowledgement on each cold connection", () => { const { admission, guard } = loaded(); - select(guard); + const cold = admission.createGuard(); + expect(() => guard("inbound", { id: 1, result: response("high", "s") })).toThrow("authority was replaced"); + expect(() => cold("outbound", { id: 2, method: "session/prompt", params: { sessionId: "s" } })).toThrow("does not acknowledge"); const resumed = admission.createGuard(); - resumed("outbound", { id: 4, method: "session/load", params: { sessionId: "s" } }); - expect(() => resumed("outbound", { id: 5, method: "session/prompt", params: { sessionId: "s" } })).toThrow("does not acknowledge"); - const fresh = loaded(); select(fresh.guard); - const acknowledged = fresh.admission.createGuard(); - acknowledged("outbound", { id: 4, method: "session/load", params: { sessionId: "s" } }); - acknowledged("inbound", { id: 4, result: response("high") }); - expect(() => acknowledged("outbound", { id: 5, method: "session/prompt", params: { sessionId: "s" } })).not.toThrow(); + resumed("outbound", { id: 3, method: "session/load", params: { sessionId: "s" } }); + resumed("inbound", { id: 3, result: response("high") }); + expect(() => resumed("outbound", { id: 4, method: "session/prompt", params: { sessionId: "s" } })).not.toThrow(); }); - it("rejects a provider that ignores the requested effort", () => { + it("rejects a restored provider default that differs from the bound effort", () => { + const admission = createGrokEffortAdmission("low", "grok-4.7"); + const guard = admission.createGuard(); + guard("outbound", { id: 1, method: "session/load", params: { sessionId: "s" } }); + expect(() => guard("inbound", { id: 1, result: response("high") })).toThrow("differs"); + expect(() => admission.assertReady()).toThrow("differs"); + }); + it.each([{}, { configOptions: [{ id: "reasoning_effort", currentValue: "high" }] }, + { _meta: { "x.ai/sessionConfig": { options: [{ id: "high", category: "mode", selected: false }] } } }])("rejects missing native effort acknowledgement", result => { + const admission = createGrokEffortAdmission("high", "grok-4.7"), guard = admission.createGuard(); + guard("outbound", { id: 1, method: "session/load", params: { sessionId: "s" } }); + expect(() => guard("inbound", { id: 1, result })).toThrow("missing or ambiguous"); + }); + it("rejects ambiguous native effort selections", () => { + const admission = createGrokEffortAdmission("high", "grok-4.7"), guard = admission.createGuard(); + const result = response("high", "s"); + result._meta["x.ai/sessionConfig"].options.push({ id: "low", category: "mode", selected: true }); + guard("outbound", { id: 1, method: "session/new", params: {} }); + expect(() => guard("inbound", { id: 1, result })).toThrow("missing or ambiguous"); + }); + it("rejects an effort acknowledgement for a different effective model", () => { + const admission = createGrokEffortAdmission("high", "grok-4.7"), guard = admission.createGuard(); + const result = response("high", "s"); result.models.currentModelId = "grok-4.6"; + guard("outbound", { id: 1, method: "session/new", params: {} }); + expect(() => guard("inbound", { id: 1, result })).toThrow("model acknowledgement conflicts"); + }); + it("does not admit an uncorrelated or failed response", () => { + const admission = createGrokEffortAdmission("high", "grok-4.7"), guard = admission.createGuard(); + guard("outbound", { id: 1, method: "session/new", params: {} }); + guard("inbound", { id: 2, result: response("high", "s") }); + expect(admission.isReady()).toBe(false); + guard("inbound", { id: 1, error: { code: -32603 } }); + expect(admission.isReady()).toBe(false); + }); + it.each(["response", "detail"])("rejects a conflicting native %s session identity", field => { + const admission = createGrokEffortAdmission("high", "grok-4.7"), guard = admission.createGuard(); + const result = { ...response("high"), ...(field === "response" ? { sessionId: "other" } : {}), + _meta: { ...response("high")._meta, ...(field === "detail" ? { "x.ai/sessionDetail": { sessionId: "other" } } : {}) } }; + guard("outbound", { id: 1, method: "session/load", params: { sessionId: "s" } }); + expect(() => guard("inbound", { id: 1, result })).toThrow("different session"); + }); + it.each(["session/set_config_option", "session/set_model"])("rejects unsupported late %s controls", method => { const { guard } = loaded(); - expect(() => select(guard, "medium")).toThrow("did not apply"); + expect(() => guard("outbound", { id: 2, method, params: { sessionId: "s" } })).toThrow("when the native session opens"); }); - it("rejects drift and wrong-session prompts after admission", () => { - const { guard } = loaded(); select(guard); - expect(() => guard("inbound", { method: "session/update", params: { sessionId: "s", update: { sessionUpdate: "config_option_update", ...response("low") } } })).toThrow("drifted"); - const fresh = loaded(); select(fresh.guard); - expect(() => fresh.guard("outbound", { id: 3, method: "session/prompt", params: { sessionId: "foreign" } })).toThrow("different session"); + it("rejects a prompt for another session", () => { + const { guard } = loaded(); + expect(() => guard("outbound", { id: 2, method: "session/prompt", params: { sessionId: "other" } })).toThrow("different session"); + }); + it.each([ + { sessionId: "s", model_id: "grok-4.7", reasoning_effort: "low" }, + { sessionId: "other", model_id: "grok-4.7", reasoning_effort: "high" }, + { sessionId: "s", model_id: "grok-4.6", reasoning_effort: "high" }, + ])("rejects native model or effort drift", update => { + const { guard } = loaded(); + expect(() => guard("inbound", { method: "_x.ai/session_notification", params: { sessionId: update.sessionId, + update: { sessionUpdate: "model_changed", model_id: update.model_id, reasoning_effort: update.reasoning_effort } } })).toThrow("drifted"); }); }); diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts b/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts index 19118a0472a..8dd25d3f123 100644 --- a/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts +++ b/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts @@ -1,5 +1,3 @@ -import { createAcpxConfigAdmission } from "./config-admission.js"; - export type GrokEffort = "minimal" | "low" | "medium" | "high" | "xhigh"; export function resolveGrokEffort(agent: string, value: unknown): GrokEffort | undefined { if (value === undefined || value === "") return undefined; @@ -7,9 +5,83 @@ export function resolveGrokEffort(agent: string, value: unknown): GrokEffort | u if (value !== "minimal" && value !== "low" && value !== "medium" && value !== "high" && value !== "xhigh") throw new Error("Grok effort must be minimal, low, medium, high, or xhigh"); return value; } -export function createGrokEffortAdmission(expected: GrokEffort) { - return createAcpxConfigAdmission(expected, { label: "Grok effort", configId: "reasoning_effort", - supportedValues: ["default", "minimal", "low", "medium", "high", "xhigh"] }); +export function createGrokEffortAdmission(expected: GrokEffort, expectedModel: string) { + type State = { sessionId: string | null; ready: boolean; error: Error | null }; + let current: State | null = null; + const failure = (detail: string) => new Error(`Grok effort admission failed: ${detail}`); + const assertReady = () => { + if (current?.error) throw current.error; + if (!current?.sessionId || !current.ready) throw failure("native session does not acknowledge the selected effort"); + }; + const readAcknowledgement = (result: Record) => { + const config = object(object(result._meta)["x.ai/sessionConfig"]); + const options = Array.isArray(config.options) ? config.options.map(object) : []; + const selected = options.filter(option => option.category === "mode" && option.selected === true); + if (selected.length !== 1) throw failure("native effort selection is missing or ambiguous"); + if (selected[0]!.id !== expected) throw failure("native effort differs from the selected effort"); + const model = object(result.models).currentModelId; + const selectedModels = options.filter(option => option.category === "model" && option.selected === true); + if (typeof model !== "string" || !model.trim() || model !== expectedModel + || selectedModels.length > 1 || selectedModels.length === 1 && selectedModels[0]!.id !== model) { + throw failure("native model acknowledgement conflicts with the selected model"); + } + }; + return { + assertReady, + isReady() { if (current?.error) throw current.error; return current?.ready === true; }, + createGuard() { + const state: State = { sessionId: null, ready: false, error: null }; + current = state; + let pending: { id: string | number; method: string; sessionId: unknown } | null = null; + return (direction: "inbound" | "outbound", value: unknown): void => { + if (current !== state) throw failure("connection authority was replaced"); + if (state.error) throw state.error; + const message = object(value), params = object(message.params); + try { + if (direction === "outbound") { + if (message.method === "session/prompt") { + assertReady(); + if (params.sessionId !== state.sessionId) throw failure("prompt belongs to a different session"); + } else if (message.method === "session/new" || message.method === "session/load" || message.method === "session/resume") { + if (typeof message.id !== "string" && typeof message.id !== "number" || pending) throw failure("uncorrelated or overlapping session admission"); + const meta = object(params._meta); + if (meta.reasoningEffort !== undefined && meta.reasoningEffort !== expected + || meta.modelId !== undefined && meta.modelId !== expectedModel) { + throw failure("native session request conflicts with the bound configuration"); + } + params._meta = { ...meta, modelId: expectedModel, reasoningEffort: expected }; + state.ready = false; state.sessionId = null; + pending = { id: message.id, method: message.method, sessionId: params.sessionId }; + } else if (message.method === "session/set_config_option" || message.method === "session/set_model") { + throw failure("model and effort must be selected when the native session opens"); + } + return; + } + if (message.method === "_x.ai/session_notification" && object(params.update).sessionUpdate === "model_changed") { + if (!state.ready) return; + const update = object(params.update); + if (params.sessionId !== state.sessionId || update.reasoning_effort !== expected + || update.model_id !== expectedModel) throw failure("native model or effort drifted from the selected configuration"); + return; + } + if (message.method !== undefined || !pending || message.id !== pending.id) return; + const request = pending; pending = null; + if (message.error !== undefined) return; + const result = object(message.result); + const sessionId = request.method === "session/new" ? result.sessionId : request.sessionId; + if (typeof sessionId !== "string" || !sessionId.trim()) throw failure("native acknowledgement has no session identity"); + const detail = object(object(result._meta)["x.ai/sessionDetail"]); + if (result.sessionId !== undefined && result.sessionId !== sessionId + || detail.sessionId !== undefined && detail.sessionId !== sessionId) throw failure("native acknowledgement belongs to a different session"); + readAcknowledgement(result); + state.sessionId = sessionId; state.ready = true; + } catch (error) { + state.error = error instanceof Error ? error : failure("invalid native acknowledgement"); + throw state.error; + } + }; + }, + }; } export function admittedGrokEffort(agent: string, requested: unknown, descriptor: Record, identity: Record): GrokEffort | undefined { const expected = resolveGrokEffort(agent, requested); @@ -17,3 +89,7 @@ export function admittedGrokEffort(agent: string, requested: unknown, descriptor if (effective !== expected || descriptor.grokEffort !== expected) throw new Error("Grok effort differs from the admitted configuration"); return effective; } + +function object(value: unknown): Record { + return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +} diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts b/packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts new file mode 100644 index 00000000000..6770890cf53 --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts @@ -0,0 +1,89 @@ +import { spawn } from "node:child_process"; +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { createAcpRuntime, createAgentRegistry, createRuntimeStore, type AcpRuntimeHandle } from "acpx/runtime"; +import { expect, it } from "vitest"; +import { createGrokEffortAdmission } from "./grok-effort.js"; + +it.each(["low", "high"] as const)("transports and restores native Grok %s effort through the real ACPX client", async effort => { + const root = await mkdtemp(join(tmpdir(), "grok-native-session-options-")); + const fixture = join(root, "grok"), wirePath = join(root, "wire.jsonl"), statePath = join(root, "native.json"); + await writeFile(fixture, String.raw` + const fs = require("node:fs"), readline = require("node:readline"); + const statePath = ${JSON.stringify(statePath)}, wirePath = ${JSON.stringify(wirePath)}; + const reply = (id, result) => process.stdout.write(JSON.stringify({jsonrpc:"2.0",id,result})+"\n"); + readline.createInterface({input:process.stdin}).on("line", line => { + const request = JSON.parse(line), params = request.params ?? {}; + fs.appendFileSync(wirePath, JSON.stringify({method:request.method,sessionId:params.sessionId,meta:params._meta})+"\n"); + if (request.method === "initialize") return reply(request.id, {protocolVersion:1,agentCapabilities:{loadSession:true},authMethods:[]}); + if (["session/new","session/load","session/resume"].includes(request.method)) { + const previous = fs.existsSync(statePath) ? JSON.parse(fs.readFileSync(statePath,"utf8")) : {}; + const model = params._meta?.modelId ?? previous.model; + const effort = params._meta?.reasoningEffort ?? "high"; + fs.writeFileSync(statePath,JSON.stringify({model,effort})); + return reply(request.id, {...(request.method === "session/new" ? {sessionId:"grok-native-session"} : {}), + models:{currentModelId:model,availableModels:[{modelId:model,name:model}]}, + _meta:{"x.ai/sessionConfig":{options:[{id:model,category:"model",selected:true},{id:effort,category:"mode",selected:true}]}}}); + } + if (request.method === "_x.ai/session/info") return reply(request.id, {result:{sessionId:params.sessionId}}); + if (request.method === "session/prompt") { + process.stdout.write(JSON.stringify({jsonrpc:"2.0",method:"session/update",params:{sessionId:params.sessionId, + update:{sessionUpdate:"agent_message_chunk",content:{type:"text",text:"GROK_RESTORED_TURN_OK"}}}})+"\n"); + return reply(request.id,{stopReason:"end_turn"}); + } + + process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:request.id,error:{code:-32601,message:"Unsupported native method"}})+"\n"); + }); + `); + const store = createRuntimeStore({ stateDir: join(root, "store") }); + const model = "grok-exact-model"; + let admission = createGrokEffortAdmission(effort, model); + const options = { + cwd: root, + sessionStore: store, + agentRegistry: createAgentRegistry({ overrides: { grok: [fixture, "agent", "stdio"] } }), + permissionMode: "approve-reads" as const, + timeoutMs: 5_000, + protocolGuardFactory: () => admission.createGuard(), + spawnAgent: () => spawn(process.execPath, [fixture], { cwd: root, env: {}, stdio: "pipe" }), + }; + const input = { sessionKey: "grok-native-options", agent: "grok", mode: "persistent" as const, + sessionOptions: { model } }; + let runtime = createAcpRuntime(options); + let handle: AcpRuntimeHandle | undefined; + try { + handle = await runtime.ensureSession(input); + admission.assertReady(); + const originalSession = handle.backendSessionId; + await runtime.close({ handle, reason: "cold native restore", discardPersistentState: false }); + admission = createGrokEffortAdmission(effort, model); + runtime = createAcpRuntime(options); + handle = await runtime.ensureSession(input); + expect(handle.backendSessionId).toBe(originalSession); + expect(admission.isReady()).toBe(false); + await runtime.requestExtension({ handle, method: "_x.ai/session/info", params: { sessionId: handle.backendSessionId }, sessionMode: "persistent" }); + admission.assertReady(); + const turn = runtime.startTurn({ handle, text: "Verify the restored fixture", mode: "prompt", requestId: "grok-cold-turn", timeoutMs: 5_000 }); + const output: string[] = []; + const drain = (async () => { + for await (const event of turn.events) if (event.type === "text_delta") output.push(event.text); + })(); + const result = await turn.result; + await drain; + expect(result.status).toBe("completed"); + expect(output.join("")).toBe("GROK_RESTORED_TURN_OK"); + admission.assertReady(); + const wire = (await readFile(wirePath, "utf8")).trim().split("\n").map(line => JSON.parse(line)); + const opens = wire.filter(request => ["session/new", "session/load", "session/resume"].includes(request.method)); + expect(opens.map(request => request.method)).toEqual(["session/new", "session/load", "session/load"]); + for (const request of opens) expect(request.meta).toMatchObject({ modelId: model, reasoningEffort: effort }); + expect(wire.filter(request => request.method === "session/prompt")).toEqual([ + { method: "session/prompt", sessionId: originalSession }, + ]); + expect(wire.some(request => request.method === "session/set_config_option")).toBe(false); + } finally { + if (handle) await runtime.close({ handle, reason: "native metadata test complete", discardPersistentState: false }); + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 25804b55c31..6f9990c550f 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v67-grok-native-effort-candidate-historical-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v67-grok-native-effort-candidate-historical-identity.json new file mode 100644 index 00000000000..7805e82d794 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v67-grok-native-effort-candidate-historical-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:2c0854674e3d00fd00cb0659b5ddb9788f47fb2b11f771901f338cb384926318", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 67, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62", + "semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "4cae579aac836762fdbf92c5447425d8b756e87a5766bc0c7a1885a90541c8d6", + "semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b", + "providerLaunchSourceSha256": "4c1abd77bbed9723e669009a78dab6fe024837718ced9f22c159f9e22ce1f26c", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "9584d9801b87f67e60b0007beafa909fc5b05dc6adac71409db1c7f80a2f7916", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297", + "profileCompatibilitySourceSha256": "4b0f14e8e2a34e1b6795a5be63265451e36c7583d99a246c5471e01f26f6e170", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "976fef21ec9f3ae9dd5eabb133a55ae578f1c0729c657c30e42562a80db237d8", + "runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2", + "runtimeSandboxSourceSha256": "981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v69-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v69-identity.json new file mode 100644 index 00000000000..1b84327a954 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v69-identity.json @@ -0,0 +1,85 @@ +{ + "commandDigest": "sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 69, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62", + "semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "4cae579aac836762fdbf92c5447425d8b756e87a5766bc0c7a1885a90541c8d6", + "semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b", + "providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "93a631b8439df1590b0cf442cf84a9b05f004f12bfee90291651ccfc0861effe", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297", + "profileCompatibilitySourceSha256": "3513d4b0c57a17b125f7d83e3a844be4b88de7475577dac99132aef00cbdca63", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "sidecarErrorSourceSha256": "b09af4577f8bf42eee7036b2ad9842202cfc763e72b53c3287f82c6090cdad80", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "976fef21ec9f3ae9dd5eabb133a55ae578f1c0729c657c30e42562a80db237d8", + "runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2", + "runtimeSandboxSourceSha256": "981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} diff --git a/ui/src/components/AgentConfigForm.tsx b/ui/src/components/AgentConfigForm.tsx index 4966526432c..56b223ab862 100644 --- a/ui/src/components/AgentConfigForm.tsx +++ b/ui/src/components/AgentConfigForm.tsx @@ -1592,7 +1592,7 @@ export function AgentConfigForm(props: AgentConfigFormProps) {

Upgrade to Paperclip Runner

- Keep {harnessType === "claude_local" ? "Claude" : "Codex"}, your model, account, and compatible settings. + Keep your harness, model, account, and compatible settings. New sessions start fresh; past runs stay available.

{canQualifyRunnerUpgrade && runnerAvailability?.defaultRunner !== "paperclip" ? ( From 3421366903543b7c39517a0078907f71252f1734 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 18:10:36 -0500 Subject: [PATCH 27/41] fix(server): schedule decision TTL expiry from saved deadlines (#17479) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Open decisions have a saved expiry time. > - The main maintenance tick scans those times even when none is due. > - Committed creation and resolution can maintain the actual next deadline. > - The existing scheduler can run expiry only when that deadline arrives. > - This pull request gives decision TTL expiry its own worker. > - Terminal history becomes quiet while expiry and continuation delivery remain recoverable. ## Linked Issues or Issue Description Targets master after #17478 merged. This is slice 10 of the recurring server work migration. It owns decision TTL expiry. Target invalidation, interrupted effects, and retention remain separate migrations. The full ledger is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`. **What existing behavior does this improve?** Time-based decision expiry in the server. **Current behavior** The combined decision maintenance sweep queries expired open decisions on each tick. It mixes that query with target invalidation, effect recovery, and continuation delivery. **Proposed behavior** Creation, bundle creation, and existing decision updates notify the shared scheduler. A dedicated worker runs at the first saved open expiry. Resolving an open decision removes its deadline. The combined tick opts out of TTL selection while its other duties remain intact. **Reason and benefit** A future decision no longer requires repeated TTL queries before its deadline. Terminal history and an empty decision table leave no recurring TTL work. No queue table is added. ## What Changed - Add an expiry owner for bounded due selection, next-deadline lookup, and expiry settlement. Reuse it for expiry-on-use and the existing target expiration transition. - Use the central expiry-only creation notification before insert, including bundles and caller-owned transactions. Update and terminal transitions continue to notify expiry and continuation. - Register the enabled-heartbeat TTL worker before decision housekeeping. Preserve disabled-heartbeat behavior and the public combined sweep's default batch limit. - Honor `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100). Keep a fair expiry/ID cursor across failures. Stop between candidates on shutdown and reconstruct saved work after restart. - Recheck open status and due TTL during settlement. Merge metadata and persist activity atomically with expiry. Publish activity and deliver continuation after commit. Handle exact timestamp equality. - Continue successful overdue batches after one second through the existing deadline adapter; preserve thirty-second failure retry and all default callers. - Add generated migration `0389_brave_hulk.sql`, an idempotent concurrent partial index on open expiry and ID. Preserve five generated snapshots. - Add 20 real-PostgreSQL lifecycle cases and startup ownership assertions. Document contracts and limitations in `doc/idle-decision-expiry.md`. ## Verification Current candidate: `aa03d2c4c25af4e6c2acf49d3160bbaf6ada030b`. This PR targets master after #17478 merged. Creation now signals only its real expiry obligation; terminal updates preserve continuation delivery. - **117 current cases pass** across all 20 expiry, 54 decision/continuation, 42 startup and one scoped-deadline cases. Server typecheck/build pass. Separate migrated PostgreSQL fixtures exercise real transaction and notification behavior. - Single and bundle creation each failed a regression before the expiry-only signal repair: open rows unnecessarily notified continuation recovery. Both now arm their saved expiry without a continuation signal. Cancelling each row afterward still signals and delivers continuation normally. The distinction is in the existing central notification owner; there are no per-route flags or counters. - **Three new real PostgreSQL regressions failed before repair:** a configured batch of two was ignored, a successful 101-row backlog waited for the thirty-second error retry, and an activity insert failure left the row expired without its audit. All three pass. The audit failure rolls back expiry; retry records one activity, preserves unrelated metadata and wakes once. - The shared deadline adapter has optional `continuationMs`. Only this owner opts into one-second successful overdue continuation. Other callers keep the previous default, and errors still retry after thirty seconds. The archive consumer passes with the option absent. - Migration `0389_brave_hulk.sql` is regenerated from the parent's metadata. Parent SQL/journal entries are preserved, five generated snapshots remain, schema drift and migration numbering/safety pass. Fresh migration apply and replay pass. - Lifecycle coverage includes first producer activation, future-deadline quiet, terminal history, exact equality, commit/rollback, restart, drain/resume, failed expiry and wake delivery, fair paging through failed rows at PostgreSQL timestamp precision, shutdown and recovery. - Failure accounting: the first full repair selection passed 113 cases and failed a startup assertion that observed a pending follow-up notification before reconciliation. The assertion now waits for the exact saved future deadline, then still proves no early expiration. Earlier fault-fixture and timestamp-cursor failures remain recorded in [the behavior audit](doc/idle-decision-expiry.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. Commands: server typecheck/build and Vitest for `decision-expiry-work`, `decisions-service`, `decision-retention-service`, `server-startup-feedback-export` and `scoped-deadline-work`; DB migration checks, snapshot drift and actual migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail remain required before merge. ## Risks - Expiry uses `expiresAt <= now`; the old combined loop skipped exact equality. The deadline boundary is now covered explicitly. - TTL expiry records the TTL reason. Target-driven expiry preserves its own reason. When both causes apply, the first terminal transition wins. - Supported producers share the application DB signal owner. Independent SQL writers or multiple active processes need a separate notification contract. New deadline-edit APIs must use the existing helper. - The index requires top-level concurrent migration execution. It is registered with the migration owner and replayed in verification. - Future in-process deadlines are not external Cloud alarms. Existing sleep guards remain conservative. This PR does not claim host/database suspension or measured savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-decision-expiry.md | 177 ++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 40 ++- packages/db/src/client.ts | 2 + .../db/src/migrations/0389_brave_hulk.sql | 1 + ...{0384_snapshot.json => 0389_snapshot.json} | 113 +++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/decisions.ts | 2 + .../__tests__/decision-expiry-work.test.ts | 330 ++++++++++++++++++ .../server-startup-feedback-export.test.ts | 7 +- server/src/index.ts | 7 +- server/src/services/deadline-work.ts | 7 +- server/src/services/decision-expiry-work.ts | 16 + server/src/services/decision-expiry.ts | 65 ++++ .../services/decision-work-notifications.ts | 11 +- server/src/services/decisions.ts | 32 +- .../services/delivery-work-notifications.ts | 1 + 16 files changed, 789 insertions(+), 29 deletions(-) create mode 100644 doc/idle-decision-expiry.md create mode 100644 packages/db/src/migrations/0389_brave_hulk.sql rename packages/db/src/migrations/meta/{0384_snapshot.json => 0389_snapshot.json} (99%) create mode 100644 server/src/__tests__/decision-expiry-work.test.ts create mode 100644 server/src/services/decision-expiry-work.ts create mode 100644 server/src/services/decision-expiry.ts diff --git a/doc/idle-decision-expiry.md b/doc/idle-decision-expiry.md new file mode 100644 index 00000000000..6cb6b3579a7 --- /dev/null +++ b/doc/idle-decision-expiry.md @@ -0,0 +1,177 @@ +# Decision TTL expiry while idle + +Open decisions already contain an `expiresAt` deadline. The expiry worker now +uses that timestamp instead of scanning TTL candidates on each maintenance tick. +This slice owns time-based expiry only. Target invalidation, interrupted effect +recovery, and retention remain separate duties. + +## Producers and transaction ownership + +Creation and bundle creation call `notifyDecisionCreated` before the decision +insert in their existing transaction. It signals expiry only: an open row cannot +owe continuation delivery yet. Existing decision updates share `notifyDecisionChange`, which now +signals expiry as well as continuation delivery. Cancellation, dismissal, and +execution claims therefore remove obsolete expiry deadlines. All production +inserts and updates are in the decision owner; acknowledgement only changes +terminal delivery metadata and cannot change an open expiry. No supported API +currently edits an existing decision's expiry timestamp. A future such writer +must use this same notification boundary. + +The helper adds no recovery wrappers to routes. Outer transactions and nested +savepoints retain the shared commit/rollback and uncertain-outcome protocol. +Creation without a committed row cannot publish a business deadline. Startup +reconstructs saved deadlines, including already-overdue work. Independent SQL +writers and another active server process need their own wake contract. + +`decision-expiry.ts` owns due selection, the next deadline, and expiry settlement. +`decisions.ts` delegates expiry-on-use and the existing target-driven expiration +transition to that owner. Target selection remains outside the TTL worker. +Startup registers TTL work before the combined decision sweep. The periodic +combined sweep opts out of TTL selection and cannot expire a row merely because +its TTL elapsed. Its public default still includes TTL and preserves the +configured combined-sweep batch limit. + +## Deadline, settlement, and recovery + +The next deadline is the earliest open `expiresAt`. A partial index on +`(expires_at, id)` covers open decisions. The query selects one indexed row; +retained terminal history does not enlarge the deadline lookup. Expiry is due +when `expiresAt <= now`, including equality. The old combined path skipped the +exact equality despite selecting that row; this migration fixes that boundary. + +Each worker pass honors `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100), +ordered by expiry and ID. +Its in-memory cursor advances across failures and wraps after the last page. +Remaining overdue work retains a one-second continuation through the saved +earliest deadline after a successful pass. +Failures retain the existing 30-second retry throttle. New earlier creation or +resolution of the earliest decision causes deadline reconciliation. When only a +future expiry remains, the worker does not query again until that deadline or a +committed change. No open decisions means no recurring expiry work. + +Settlement requires the row to remain open. TTL settlement also rechecks that +the saved deadline is due. Metadata is merged into the current row. The existing +expiry activity is persisted in the same transaction as the terminal transition. +Publication and origin continuation follow commit. If the audit insert fails, +the expiry and continuation obligation both roll back. A deadline-driven expiry +records `ttl`; target-driven expiration retains its own reason. When both causes +are possible, the first successful terminal transition determines the reason. +Continuation failures belong to the continuation worker after the decision is +expired. Neither owner replays decision effects. + +Drain and standby defer dispatch while retaining the request. The worker checks +the coordinator's AbortSignal between candidates. Shutdown finishes the admitted +expiry and leaves untouched candidates for restart. It does not process the +whole backlog before stopping. There is no new queue table or permanent DB +listener, and no provider call is held inside a new transaction. + +## Limits + +The disabled-heartbeat branch keeps its existing behavior and does not register +this worker. Expiry-on-use remains enforced independently of background work. +The in-process deadline is not an external Cloud alarm. Existing conservative +sleep blockers remain; this migration does not permit an instance to sleep with +unsupported future obligations. Worker quiet is not proof of host or database +suspension. + +## Verification + +The original combined selection passed 103 tests: 14 initial real-PostgreSQL expiry worker +cases, 50 decision service/continuation cases, and 39 startup cases. It covered +empty and terminal-history quiet, actual create and bundle producers, exact +expiry, moving the next deadline earlier, cancellation, restart, outer rollback +and commit, first creation during drain, failed retry through drain, failed wake +delivery, separate aggregate/TTL ownership, 101 candidates with a failed first +row, creation during delivery, and shutdown/restart with outstanding candidates. +The parent archive race and bounded continuation shutdown regressions also pass. + +The first selection had 97 passes and one fixture failure. The fault test cleared +its error collector before an already-admitted failed query had finished. It now +checks all collected causes after recovery settles. The next selection passed +101 cases; after the parent review repairs and bounded expiry/shutdown coverage, +all 103 pass. No production hold, retry, or timeout was weakened. + +The original index applied and replayed successfully; after normal parent +integration it is regenerated as `0389_brave_hulk.sql`. Fresh apply also passes. With 40,000 synthetic decisions +and one open row, the next-deadline query uses the partial index, reads two shared +buffers, and executes in 0.022 ms locally. The probe transaction rolls back. This +is query evidence, not a production latency or savings claim. Workspace +typecheck and build pass on the combined branch. Full PR CI and fresh review +remain required before readiness. + +A follow-up paging regression uses 101 overdue rows sharing a PostgreSQL +microsecond timestamp, with the first 100 writes failing. The original Date-based +cursor rounded the timestamp to milliseconds and repeatedly selected that failed +page, starving the last valid row. The cursor now carries PostgreSQL timestamp +text into its tuple comparison; business deadline calculations still use Date. +The regression failed before the fix and verifies progress past the failed page, +then recovery of all failed rows after the fault is removed. +The repaired expiry suite passes all 15 cases, and server typechecking passes. + + +## Current review repairs and verification + +Three review findings failed real PostgreSQL regressions before repair: the +standalone worker ignored the configured batch limit, a successful overdue +backlog waited thirty seconds between batches, and a failed audit write left the +row expired without durable activity. The worker now honors its existing batch +configuration and continues successful overdue work after one second. The +shared deadline adapter accepts an optional `continuationMs`; omitted values +preserve every existing consumer's retry interval. Errors still retain the +existing thirty-second retry. Expiry and durable activity commit atomically, +then publish activity and deliver the origin continuation. Provider calls stay +outside that transaction. + +All 129 selected cases pass: 18 expiry, 53 decision/continuation, 15 archive, 42 +startup and one scoped-deadline case. The archive selection exercises an existing +shared-deadline consumer with the new option absent. Server typecheck/build and +generated snapshot/schema drift pass. Parent journal entries remain unchanged +and five generated snapshots remain. No snapshot was hand-edited. + +The first complete selection passed 113 cases and failed the future-startup +assertion because it inspected the scheduler before an additional committed +notification reconciliation finished. Coordinator `ready` awaits the first +admitted pass, while a notification arriving during that pass can schedule +another immediate pass. The assertion now waits for reconciliation to expose +the exact saved future deadline, then still proves the decision does not expire +early. The three production regressions and this fixture failure remain +recorded; no error retry or business deadline was relaxed to pass the tests. +Fresh current-head CI and reviews remain merge gates. + + +After normal integration of continuation recovery's final coalescing and +deterministic contention tests and pinned master `8161f2a813`, all 130 selected +cases pass: 76 expiry/archive/startup/deadline cases plus all 54 decision cases. +Server types/build pass. Existing notification coalescing remains thirty +seconds for continuation recovery; expiry's successful backlog continuation is +one second, and failure retry remains thirty seconds. Migration replay reports +no pending migrations and the parent journal remains an exact prefix. Extra +upstream embedded-only tests attempted during parent integration could not +initialize locally; fresh CI must cover that environment, as recorded in the +archive parent PR. + + +Normal integration with master-target continuation parent `18f682863c` preserves +expiry, deadline and producer implementations exactly. All 61 expiry/startup/ +scoped-deadline cases and server types/build pass with the rebuilt Runner vendor. +The previous full 130-case selection remains evidence at its stated revision. +No migration SQL changed, and five generated snapshots remain. Fresh head CI +and reviews are required after this parent integration. + + +## Creation notification review repair + +Single and bundle creation previously notified continuation recovery even though +the inserted decision was open. Two real producer regressions failed before +repair because continuation received intent and settlement. Creation now uses the +existing central notification file's expiry-only `notifyDecisionCreated` helper. +Update and terminal transitions keep `notifyDecisionChange`, including the normal +continuation signal. No optional flags, per-route bookkeeping or new table is +needed. Both regressions verify the exact future expiry, no early wake, and a +subsequent cancellation that still signals and delivers continuation normally. + +All 117 current expiry/decision/startup/deadline cases pass, including 20 expiry +cases. Server types/build pass. The preceding f7 candidate's identical-tree +merge proof remains history; this subsequent creation-signal repair changes +runtime behavior and has its own acceptance above. Fresh current-head CI and +both reviews are required. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 6b1b248598f..0ff600d0569 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -16,7 +16,7 @@ Do not mark an item done from a timer removal or a green check alone. | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | -| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478), expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479), and effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) await fresh current-head gates; target invalidation review repair remains in progress | +| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479), effects [#17482](https://github.com/paperclipai/paperclip/pull/17482), and targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates | | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | | 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Slices 02–07 merged: [secret proposals](https://github.com/paperclipai/paperclip/pull/17446), [credentials](https://github.com/paperclipai/paperclip/pull/17450), [review expiry](https://github.com/paperclipai/paperclip/pull/17453), [approved recovery](https://github.com/paperclipai/paperclip/pull/17454), [interrupted claims](https://github.com/paperclipai/paperclip/pull/17458), [projection](https://github.com/paperclipai/paperclip/pull/17468) | @@ -489,3 +489,41 @@ Eight slices are merged after archive delivery #17475. Slice 09 now reconciles with its exact squash `5785090ff1`; all 31 independent master files are preserved. The 97 decision/startup/deadline cases and server types/build pass after the Runner vendor rebuild. Fresh master-target CI and reviews remain required. + + + + +## Slice 10: decision TTL expiry + +The TTL implementation now includes parent continuation repairs from `5f96892a58`. +The merge preserves both the expiry owner and per-decision effect ownership. +See [decision TTL expiry](../idle-decision-expiry.md). Creation and bundle writes +join the same notification helper; an indexed minimum open expiry drives work. +The combined tick opts out of TTL selection without removing target invalidation +or effect recovery. Expiry-on-use remains enforced, including exact equality. +Bounded passes retain a fair expiry/ID cursor and stop between candidates. + +All 103 selected tests pass after the merge, including fourteen TTL worker cases. +Migration replay and a 40,000-row history query probe pass (two buffers). +Final workspace typecheck/build and fresh PR CI/review remain required. +The parent continuation PR is awaiting fresh review/CI on `5f96892a58`, with both +initial findings repaired and resolved. The full inventory remains in scope. + +Slice 10 review repair: configured batch bounds, one-second successful overdue +continuations and atomic expiration/activity settlement pass all three previously +failing real PostgreSQL regressions. All 129 affected expiry/continuation/archive/ +startup/deadline cases, server types/build and snapshot drift pass. Own concurrent +index was regenerated as `0389_brave_hulk.sql`, preserving parent history. The +initial startup-reconciliation assertion failure and corrected boundary are +recorded in the owning behavior audit. Latest-head CI/reviews remain required. + +Final slice 10 parent integration passes 130 expiry/decision/archive/startup/ +deadline cases, server types/build and migration replay. The parent continuation +coalescing and deterministic cross-pool contention regression remain intact. +The archive parent now targets master after projection #17468 merged. Fresh +current-head reviews and CI remain required in dependency order. + +Slice 10 creation signal follow-up: single and bundle creation now notify expiry +only through the central owner. Terminal updates still signal continuation. Both +real regressions failed before repair; all 117 current expiry/decision/startup/ +deadline cases, server types and build pass. Fresh CI/reviews remain required. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 443005ca277..fa192adb0d1 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -507,6 +507,8 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0386_steady_epoch.sql", "0387_boring_professor_monster.sql", "0388_ambitious_micromax.sql", + "0389_brave_hulk.sql", + "0384_funny_adam_warlock.sql", diff --git a/packages/db/src/migrations/0389_brave_hulk.sql b/packages/db/src/migrations/0389_brave_hulk.sql new file mode 100644 index 00000000000..db1e9f48622 --- /dev/null +++ b/packages/db/src/migrations/0389_brave_hulk.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "decisions_open_expiry_idx" ON "decisions" USING btree ("expires_at","id") WHERE "decisions"."status" = 'open'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0384_snapshot.json b/packages/db/src/migrations/meta/0389_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0384_snapshot.json rename to packages/db/src/migrations/meta/0389_snapshot.json index bb2ed3c9595..1c3273c9da4 100644 --- a/packages/db/src/migrations/meta/0384_snapshot.json +++ b/packages/db/src/migrations/meta/0389_snapshot.json @@ -1,6 +1,6 @@ { - "id": "4f96f70a-fd19-4d35-ad8a-8d5559b83cbd", - "prevId": "410457fc-266b-4bd0-8bc0-2b14b0634d07", + "id": "394e85a6-7fc2-4ab1-80a3-33bb4c7003aa", + "prevId": "4f21a636-735f-4dce-9ab8-0583f3bc7b79", "version": "7", "dialect": "postgresql", "tables": { @@ -22110,6 +22110,34 @@ "concurrently": false, "method": "btree", "with": {} + }, + "decision_archive_notification_outbox_delivery_idx": { + "name": "decision_archive_notification_outbox_delivery_idx", + "columns": [ + { + "expression": "last_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "first" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decision_archive_notification_outbox\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -24030,6 +24058,44 @@ "method": "btree", "with": {} }, + "decisions_open_expiry_idx": { + "name": "decisions_open_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'open'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_pending_continuation_idx": { + "name": "decisions_pending_continuation_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"continuation_policy\" = 'wake_origin_agent' AND \"decisions\".\"metadata\" ->> 'continuationPending' = 'true' AND (\"decisions\".\"status\" IN ('expired', 'cancelled') OR (\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" IN ('succeeded', 'partial', 'failed')))", + "concurrently": false, + "method": "btree", + "with": {} + }, "decisions_bundle_idx": { "name": "decisions_bundle_idx", "columns": [ @@ -52856,6 +52922,28 @@ "method": "btree", "with": {} }, + "tool_action_requests_executing_deadline_idx": { + "name": "tool_action_requests_executing_deadline_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"tool_action_requests\".\"status\" = 'executing'", + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_approved_idx": { "name": "tool_action_requests_approved_idx", "columns": [ @@ -52887,6 +52975,27 @@ "method": "btree", "with": {} }, + "tool_action_requests_company_interaction_idx": { + "name": "tool_action_requests_company_interaction_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "interaction_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_issue_idx": { "name": "tool_action_requests_issue_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 68f12e89697..14f1a4ad042 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2710,6 +2710,13 @@ "when": 1793137983484, "tag": "0388_ambitious_micromax", "breakpoints": true + }, + { + "idx": 389, + "version": "7", + "when": 1793138457589, + "tag": "0389_brave_hulk", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/decisions.ts b/packages/db/src/schema/decisions.ts index 18b18d0c704..5fe03d8f090 100644 --- a/packages/db/src/schema/decisions.ts +++ b/packages/db/src/schema/decisions.ts @@ -69,6 +69,8 @@ export const decisions = pgTable( table.status, table.expiresAt, ), + openExpiryIdx: index("decisions_open_expiry_idx").on(table.expiresAt, table.id) + .where(sql`${table.status} = 'open'`), pendingContinuationIdx: index("decisions_pending_continuation_idx").on(table.id) .where(sql`${table.continuationPolicy} = 'wake_origin_agent' AND ${table.metadata} ->> 'continuationPending' = 'true' AND (${table.status} IN ('expired', 'cancelled') OR (${table.status} = 'decided' AND ${table.executionStatus} IN ('succeeded', 'partial', 'failed')))`), bundleIdx: index("decisions_bundle_idx").on(table.bundleId), diff --git a/server/src/__tests__/decision-expiry-work.test.ts b/server/src/__tests__/decision-expiry-work.test.ts new file mode 100644 index 00000000000..51c5c7c9673 --- /dev/null +++ b/server/src/__tests__/decision-expiry-work.test.ts @@ -0,0 +1,330 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, agents, companies, createDb, subscribeDatabaseWork, decisionBundles, decisionEffectExecutions, decisions, + decisionTargetIssues, heartbeatRuns, issues } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { decisionService } from "../services/decisions.js"; +import { registerDecisionExpiryWork } from "../services/decision-expiry-work.js"; +import { registerDecisionContinuationWork } from "../services/decision-continuation-work.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_DECISION_EXPIRY_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("decision expiry worker", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let companyId: string, agentId: string, issueId: string, runId: string; + let coordinator: ReturnType | undefined; + let draining: boolean; + const errors: unknown[] = []; + const wake = vi.fn(async () => {}); + const service = () => decisionService(db, { wakeOriginAgent: wake }); + + beforeAll(async () => { + if (!externalUrl) temporary = await startEmbeddedPostgresTestDatabase("decision-expiry-"); + db = createDb(externalUrl ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 30_000); + beforeEach(async () => { + process.env.PAPERCLIP_DECISION_SIGNING_SECRET = "0123456789abcdef0123456789abcdef"; + companyId = randomUUID(); agentId = randomUUID(); issueId = randomUUID(); runId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Expiry", issuePrefix: `E${companyId.slice(0, 6)}`, requireBoardApprovalForNewAgents: false }); + await db.insert(agents).values({ id: agentId, companyId, name: "Proposer", role: "engineer", status: "active", adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} }); + await db.insert(issues).values({ id: issueId, companyId, title: "Origin", status: "in_progress", priority: "medium", assigneeAgentId: agentId }); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", contextSnapshot: { issueId } }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + draining = false; errors.length = 0; wake.mockReset(); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs(); + await db.execute(sql`drop trigger if exists decision_expiry_fail on decisions`); + await db.execute(sql`drop function if exists decision_expiry_fail()`); + await db.execute(sql`drop trigger if exists decision_expiry_audit_fail on activity_log`); + await db.execute(sql`drop function if exists decision_expiry_audit_fail()`); + await db.delete(decisionEffectExecutions); await db.delete(decisionTargetIssues); await db.delete(decisions); + await db.delete(decisionBundles); await db.delete(activityLog); await db.delete(heartbeatRuns); + await db.delete(issues); await db.delete(agents); await db.delete(companies); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + const actor = () => ({ type: "agent" as const, companyId, agentId, runId, source: "agent_jwt" as const }); + function input(expiresAt: Date) { + return { companyId, agentId, runId, actor: actor(), title: "Decision", body: "Body", expiresAt, + continuationPolicy: "wake_origin_agent" as const, options: [{ id: "yes", label: "Yes", effects: [] }] }; + } + const create = (at = new Date(Date.now() + 120_000)) => service().create(input(at)); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, + canReconcile: () => true, onError: error => errors.push(error) }); + await registerDecisionContinuationWork(coordinator, service(), () => !draining).ready; + await registerDecisionExpiryWork(coordinator, service(), () => !draining).ready; + } + async function quiet() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"), executes = vi.spyOn(db, "execute"); + await vi.advanceTimersByTimeAsync(60 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); executes.mockRestore(); + } + async function expired(id: string) { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(await service().get(id)).toMatchObject({ status: "expired", metadata: { expiredReason: "ttl", continuationPending: false } }); + }, { timeout: 5000 }); + } + + it("is quiet when empty, arms the real producer deadline, and leaves expired history quiet", async () => { + await start(); await quiet(); + const deadline = new Date(Date.now() + 120_000); + const row = await create(deadline); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(coordinator!.nextWakeAt()).toBe(deadline.getTime()); + }); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(Math.max(0, deadline.getTime() - Date.now() - 1000)); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); expect(wake).not.toHaveBeenCalled(); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision.expired"))).toHaveLength(1); + }); + + it.each(["single", "bundle"] as const)("arms expiry without waking continuation recovery for %s creation", async mode => { + await start(); await quiet(); + const events: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionContinuation, event => { events.push(event); }); + try { + const deadline = new Date(Date.now() + 120_000); + const row = mode === "single" ? await create(deadline) : (await service().createBundle({ + companyId, actor: actor(), agentId, runId, title: "Bundle", summary: "Summary", decisions: [input(deadline)], + })).decisions[0]; + expect(events).toEqual([]); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1); + expect(coordinator!.nextWakeAt()).toBe(deadline.getTime()); + }); + expect(wake).not.toHaveBeenCalled(); + await service().cancel(row.id, { actorType: "agent", actorId: agentId }); + await quiet(); + expect(events).toEqual(["intent", "settled"]); + expect(wake).toHaveBeenCalledTimes(1); + } finally { unsubscribe(); } + }); + + it("honors the configured expiry batch size", async () => { + vi.stubEnv("PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE", "2"); + const seed = await create(); await db.delete(decisions).where(eq(decisions.id, seed.id)); + await db.insert(decisions).values(Array.from({ length: 5 }, () => ({ ...seed, id: randomUUID(), expiresAt: new Date(Date.now() - 1000) }))); + const recovery = service(); + expect(await recovery.sweepTtlExpiry()).toEqual({ expired: 2 }); + expect(await recovery.sweepTtlExpiry()).toEqual({ expired: 2 }); + expect(await recovery.sweepTtlExpiry()).toEqual({ expired: 1 }); + expect(wake).toHaveBeenCalledTimes(5); + }); + + it("continues a successful overdue backlog within one second instead of waiting for error retry", async () => { + const seed = await create(); await db.delete(decisions).where(eq(decisions.id, seed.id)); + await db.insert(decisions).values(Array.from({ length: 101 }, () => ({ ...seed, id: randomUUID(), expiresAt: new Date(Date.now() - 1000) }))); + await start(); + expect(wake).toHaveBeenCalledTimes(100); + const next = coordinator!.nextWakeAt(); + expect(next).not.toBeNull(); + expect(next! - Date.now()).toBeLessThanOrEqual(1000); + await vi.advanceTimersByTimeAsync(2000); + await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(101)); + await quiet(); + }); + + it("rolls expiry back if its durable activity cannot be recorded", async () => { + const row = await create(); + await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1000), metadata: { ...row.metadata, retained: "unchanged" } }).where(eq(decisions.id, row.id)); + await db.execute(sql`create function decision_expiry_audit_fail() returns trigger language plpgsql as $$ begin + raise exception 'expiry audit unavailable'; end $$`); + await db.execute(sql`create trigger decision_expiry_audit_fail before insert on activity_log + for each row when (NEW.action = 'decision.expired') execute function decision_expiry_audit_fail()`); + await expect(service().sweepTtlExpiry()).rejects.toMatchObject({ cause: { message: "expiry audit unavailable" } }); + expect(await service().get(row.id)).toMatchObject({ status: "open", metadata: { retained: "unchanged" } }); + expect(wake).not.toHaveBeenCalled(); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision.expired"))).toEqual([]); + await db.execute(sql`drop trigger decision_expiry_audit_fail on activity_log`); + expect(await service().sweepTtlExpiry()).toEqual({ expired: 1 }); + expect(await service().get(row.id)).toMatchObject({ status: "expired", metadata: { retained: "unchanged" } }); + expect(wake).toHaveBeenCalledTimes(1); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision.expired"))).toHaveLength(1); + }); + + it("reconstructs a future expiry on restart and does not expire it early", async () => { + const row = await create(); await start(); + await vi.waitFor(() => expect(coordinator!.nextWakeAt()).toBe(row.expiresAt.getTime())); + await vi.advanceTimersByTimeAsync(60_000); + expect((await service().get(row.id))?.status).toBe("open"); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("expires at the exact saved timestamp", async () => { + const row = await create(); + expect(await service().sweepTtlExpiry(new Date(row.expiresAt.getTime() - 1))).toEqual({ expired: 0 }); + expect(await service().sweepTtlExpiry(row.expiresAt)).toEqual({ expired: 1 }); + expect((await service().get(row.id))?.status).toBe("expired"); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("replaces a later deadline when another decision is created and removes it after cancellation", async () => { + const later = await create(new Date(Date.now() + 600_000)); await start(); + const earlier = await create(); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(coordinator!.nextWakeAt()).toBe(earlier.expiresAt.getTime()); }); + await service().cancel(earlier.id, { actorType: "agent", actorId: agentId }); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(coordinator!.nextWakeAt()).toBe(later.expiresAt.getTime()); }); + await service().cancel(later.id, { actorType: "agent", actorId: agentId }); + await quiet(); expect(wake).toHaveBeenCalledTimes(2); + }); + + it("registers bundle creation inside its existing transaction", async () => { + await start(); await quiet(); const at = new Date(Date.now() + 120_000); + const bundle = await service().createBundle({ companyId, actor: actor(), agentId, runId, title: "Bundle", summary: "Summary", + decisions: [input(at), input(at)] }); + await expired(bundle.decisions[0].id); await expired(bundle.decisions[1].id); await quiet(); + expect(wake).toHaveBeenCalledTimes(2); + }); + + it("does not publish an outer rollback and learns an outer commit", async () => { + await start(); await quiet(); + await expect(db.transaction(async tx => { + await service().create(input(new Date(Date.now() + 30_000)), tx as unknown as typeof db); + await vi.advanceTimersByTimeAsync(60_000); expect(wake).not.toHaveBeenCalled(); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect(await db.select().from(decisions)).toHaveLength(0); + let id = ""; + await db.transaction(async tx => { + id = (await service().create(input(new Date(Date.now() + 30_000)), tx as unknown as typeof db)).id; + await vi.advanceTimersByTimeAsync(60_000); expect(wake).not.toHaveBeenCalled(); + }); + await expired(id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("retains first creation during drain without SQL until resume", async () => { + await start(); await quiet(); draining = true; + const row = await create(new Date(Date.now() + 30_000)); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); expect(wake).not.toHaveBeenCalled(); + draining = false; await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("keeps a failed expiry retry through drain and preserves pending state", async () => { + const row = await create(new Date(Date.now() + 30_000)); await start(); + await db.execute(sql`create function decision_expiry_fail() returns trigger language plpgsql as $$ begin + if NEW.status = 'expired' then raise exception 'expiry unavailable'; end if; return NEW; end $$`); + await db.execute(sql`create trigger decision_expiry_fail before update on decisions for each row execute function decision_expiry_fail()`); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(errors.length).toBeGreaterThan(0); }); + draining = true; expect((await service().get(row.id))?.status).toBe("open"); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(60_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await db.execute(sql`drop trigger decision_expiry_fail on decisions`); + draining = false; await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + for (const error of errors) expect(error).toMatchObject({ cause: { message: "expiry unavailable" } }); errors.length = 0; + }); + + it("leaves failed continuation delivery to its separate retry owner", async () => { + const row = await create(new Date(Date.now() + 30_000)); await start(); + wake.mockRejectedValueOnce(new Error("wake unavailable")); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(2); + expect(errors.every(error => String(error).includes("wake unavailable"))).toBe(true); errors.length = 0; + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision.expired"))).toHaveLength(1); + }); + + it("does not deliver terminal continuation debt from the TTL-only pass", async () => { + const row = await create(); + await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, issueId)); + await db.update(decisions).set({ status: "cancelled", metadata: { continuationPending: true } }).where(eq(decisions.id, row.id)); + expect(await service().sweepTtlExpiry()).toEqual({ expired: 0 }); + expect(wake).not.toHaveBeenCalled(); expect(await service().nextExpiryAt()).toBeNull(); + }); + + it("keeps TTL expiry out of the aggregate sweep when its worker owns the duty", async () => { + const row = await create(); + await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(decisions.id, row.id)); + expect(await service().sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false })).toEqual({ expired: 0, resumed: 0 }); + expect((await service().get(row.id))?.status).toBe("open"); + expect(await service().sweepTtlExpiry()).toEqual({ expired: 1 }); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("pages past a failed candidate and retries it without repeating completed expirations", async () => { + const seed = await create(); await db.delete(decisions).where(eq(decisions.id, seed.id)); + const ids = Array.from({ length: 101 }, (_, i) => `10000000-0000-4000-8000-${String(i).padStart(12, "0")}`); + await db.insert(decisions).values(ids.map(id => ({ ...seed, id, expiresAt: new Date(Date.now() - 1000) }))); + await db.execute(sql`create function decision_expiry_fail() returns trigger language plpgsql as $$ begin + if NEW.id = '10000000-0000-4000-8000-000000000000' and NEW.status = 'expired' then raise exception 'expiry unavailable'; end if; + return NEW; end $$`); + await db.execute(sql`create trigger decision_expiry_fail before update on decisions for each row execute function decision_expiry_fail()`); + const recovery = service(); + await expect(recovery.sweepTtlExpiry()).rejects.toMatchObject({ cause: { message: "expiry unavailable" } }); + expect(wake).toHaveBeenCalledTimes(99); + expect((await service().get(ids.at(-1)!))?.status).toBe("open"); + expect(await recovery.sweepTtlExpiry()).toEqual({ expired: 1 }); + expect((await service().get(ids.at(-1)!))?.status).toBe("expired"); + expect((await service().get(ids[0]))?.status).toBe("open"); expect(wake).toHaveBeenCalledTimes(100); + await db.execute(sql`drop trigger decision_expiry_fail on decisions`); + await start(); await quiet(); expect((await service().get(ids[0]))?.status).toBe("expired"); + expect(wake).toHaveBeenCalledTimes(101); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision.expired"))).toHaveLength(101); + }); + + it("learns another expiry created during delivery", async () => { + const first = await create(new Date(Date.now() + 30_000)); + let secondId = ""; + wake.mockImplementationOnce(async () => { secondId = (await create(new Date(Date.now() + 90_000))).id; }); + await start(); await expired(first.id); + await vi.waitFor(() => expect(secondId).not.toBe("")); + await expired(secondId); await quiet(); expect(wake).toHaveBeenCalledTimes(2); + }); + + + it("finishes an admitted expiry but leaves the next candidate for restart during shutdown", async () => { + const first = await create(), second = await create(); + await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1000) }); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + wake.mockImplementationOnce(async () => gate); + const starting = start(); + let stopped = false; + let stopping: Promise; + try { + await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(1)); + stopping = coordinator!.stop().then(() => { stopped = true; }); + await vi.advanceTimersByTimeAsync(60_000); expect(stopped).toBe(false); + } finally { release(); } + await starting; await stopping!; coordinator = undefined; + expect(wake).toHaveBeenCalledTimes(1); expect(stopped).toBe(true); + const states = await Promise.all([service().get(first.id), service().get(second.id)]); + expect(states.map(row => row!.status).sort()).toEqual(["expired", "open"]); + await start(); await quiet(); expect(wake).toHaveBeenCalledTimes(2); + }); + + + it("pages past a full failed batch using database timestamp precision", async () => { + const seed = await create(); await db.delete(decisions).where(eq(decisions.id, seed.id)); + const ids = Array.from({ length: 101 }, (_, i) => `20000000-0000-4000-8000-${String(i).padStart(12, "0")}`); + await db.insert(decisions).values(ids.map(id => ({ ...seed, id }))); + await db.execute(sql`update decisions set expires_at = date_trunc('milliseconds', now() - interval '1 minute') + interval '456 microseconds'`); + await db.execute(sql`create function decision_expiry_fail() returns trigger language plpgsql as $$ begin + if NEW.id <> '20000000-0000-4000-8000-000000000100' and NEW.status = 'expired' then raise exception 'expiry unavailable'; end if; + return NEW; end $$`); + await db.execute(sql`create trigger decision_expiry_fail before update on decisions for each row execute function decision_expiry_fail()`); + const recovery = service(); + await expect(recovery.sweepTtlExpiry()).rejects.toThrow("Decision expiry failed"); + expect(await recovery.sweepTtlExpiry()).toEqual({ expired: 1 }); + expect((await service().get(ids.at(-1)!))?.status).toBe("expired"); + await db.execute(sql`drop trigger decision_expiry_fail on decisions`); + expect(await recovery.sweepTtlExpiry()).toEqual({ expired: 100 }); + expect(await recovery.sweepTtlExpiry()).toEqual({ expired: 0 }); + expect(wake).toHaveBeenCalledTimes(101); + }); + +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 8eee51072d4..c2ee1e0caa4 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -349,6 +349,8 @@ vi.mock("../services/index.js", () => ({ })), decisionService: vi.fn(() => ({ sweepExpired: vi.fn(async () => ({ expired: 0 })), + sweepTtlExpiry: vi.fn(async () => ({ expired: 0 })), + nextExpiryAt: vi.fn(async () => null), sweepContinuations: vi.fn(async () => ({ retry: false })), })), decisionRetentionService: vi.fn(() => ({ @@ -918,7 +920,9 @@ describe("startServer feedback export wiring", () => { expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionContinuation)).toBe(true); const decisions = vi.mocked(decisionService).mock.results.at(-1)!.value; expect(decisions.sweepContinuations).toHaveBeenCalledTimes(1); - expect(decisions.sweepExpired).toHaveBeenCalledWith(expect.any(Date), { includeContinuations: false }); + expect(decisions.sweepTtlExpiry).toHaveBeenCalledTimes(1); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(true); + expect(decisions.sweepExpired).toHaveBeenCalledWith(expect.any(Date), { includeContinuations: false, includeExpiry: false }); } finally { setIntervalSpy.mockRestore(); } @@ -956,6 +960,7 @@ describe("startServer feedback export wiring", () => { expect(heartbeatServiceMock.reconcileCostAccounting).toHaveBeenCalledTimes(1); expect(environmentCustomImagesServiceMock.cleanupExpiredSetupSessions).not.toHaveBeenCalled(); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionContinuation)).toBe(false); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(false); } finally { setIntervalSpy.mockRestore(); } diff --git a/server/src/index.ts b/server/src/index.ts index afb2b803ac9..883590a67a4 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerDecisionExpiryWork } from "./services/decision-expiry-work.js"; import { registerDecisionContinuationWork } from "./services/decision-continuation-work.js"; import { registerDecisionArchiveDeliveryWork } from "./services/decision-archive-delivery-work.js"; @@ -1839,7 +1840,9 @@ async function startServerWithDatabaseTeardown( }); await step("decision_continuation_worker", () => registerDecisionContinuationWork(deliveryWork, decisionExecutor, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); - await step("decision_expiry", () => decisionExecutor.sweepExpired(new Date(), { includeContinuations: false })); + await step("decision_expiry_worker", () => registerDecisionExpiryWork(deliveryWork, decisionExecutor, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("decision_expiry", () => decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false })); const loginSchedule = loginCleanupSchedule(db, config.heartbeatSchedulerIntervalMs); await step("login_cleanup_workers", () => registerLoginCleanupWork(deliveryWork, [ { queue: DELIVERY_QUEUES.adapterLoginCleanup, run: () => adapterLoginReaper.sweep().then(logAdapterLoginReaperResult), next: loginSchedule.adapter }, @@ -1860,7 +1863,7 @@ async function startServerWithDatabaseTeardown( trackHeartbeatSchedulerWork((async () => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; scheduleRoutineStatusReconcile(); - trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired(new Date(), { includeContinuations: false }).catch((err: unknown) => { + trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false }).catch((err: unknown) => { logger.error({ err }, "decision expiry sweep failed"); })); trackHeartbeatSchedulerWork(runRetentionSweep().catch((err: unknown) => { diff --git a/server/src/services/deadline-work.ts b/server/src/services/deadline-work.ts index d6ff2841037..31482ccd8a0 100644 --- a/server/src/services/deadline-work.ts +++ b/server/src/services/deadline-work.ts @@ -9,6 +9,7 @@ export function registerDeadlineWork( next: () => Promise; canRun: () => boolean; retryMs: number; + continuationMs?: number; // Optional batching latency for notifications; already-known deadlines still preempt it. notificationCoalesceMs?: number; }, @@ -29,12 +30,14 @@ export function registerDeadlineWork( return; } const generation = ++requested; - earliest = Date.now() + task.retryMs; + const startedAt = Date.now(); + earliest = startedAt + task.retryMs; await task.run(signal); if (signal.aborted) return; const deadline = await task.next(); // Back off failed or still-overdue work, not a successfully read future deadline. - if (deadline === null || deadline > Date.now()) earliest = 0; + earliest = deadline === null || deadline > Date.now() + ? 0 : startedAt + (task.continuationMs ?? task.retryMs); next = deadline === null ? null : Math.max(earliest, deadline); coalesceUntil = Date.now() + (task.notificationCoalesceMs ?? 0); reconciled = generation; diff --git a/server/src/services/decision-expiry-work.ts b/server/src/services/decision-expiry-work.ts new file mode 100644 index 00000000000..b49d88a69c5 --- /dev/null +++ b/server/src/services/decision-expiry-work.ts @@ -0,0 +1,16 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { decisionService } from "./decisions.js"; + +export function registerDecisionExpiryWork( + coordinator: ReturnType, + service: Pick, "sweepTtlExpiry" | "nextExpiryAt">, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.decisionExpiry, { + retryMs: 30_000, continuationMs: 1000, canRun, + run: signal => service.sweepTtlExpiry(new Date(), signal), + next: service.nextExpiryAt, + }); +} diff --git a/server/src/services/decision-expiry.ts b/server/src/services/decision-expiry.ts new file mode 100644 index 00000000000..ea5236a6ac0 --- /dev/null +++ b/server/src/services/decision-expiry.ts @@ -0,0 +1,65 @@ +import { and, asc, eq, getTableColumns, lte, sql } from "drizzle-orm"; +import { decisions, type Db } from "@paperclipai/db"; +import { logActivity, publishActivity, type ActivityPublication } from "./activity-log.js"; +import { writeDecisionChange } from "./decision-work-notifications.js"; + +type Decision = typeof decisions.$inferSelect; +type ExpiryReason = "ttl" | "target_gone" | "target_completed"; +const open = sql`${decisions.status} = 'open'`; + +export function decisionExpiryService(db: Db, deliverContinuation: (id: string) => Promise) { + let recoveryCursor: { expiresAt: string; id: string } | undefined; + async function expire(decision: Decision, reason: ExpiryReason, now: Date) { + const publications: ActivityPublication[] = []; + const updated = await writeDecisionChange(db, async tx => { + const [row] = await tx.update(decisions).set({ + status: "expired", updatedAt: now, + metadata: sql`${decisions.metadata} || ${JSON.stringify({ expiredReason: reason, + ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) })}::jsonb`, + }).where(and(eq(decisions.id, decision.id), open, + reason === "ttl" ? lte(decisions.expiresAt, now) : undefined)).returning(); + if (!row) return null; + await logActivity(tx, { companyId: row.companyId, actorType: "system", actorId: "decision-expiry-sweeper", + agentId: row.originAgentId, runId: row.originRunId, action: "decision.expired", entityType: "decision", + entityId: row.id, details: { expiredReason: reason } }, publications); + return row; + }); + if (!updated) return 0; + for (const publication of publications) publishActivity(publication); + await deliverContinuation(updated.id); + return 1; + } + + async function due(now: Date, limit: number, cursor?: { expiresAt: string; id: string }) { + return db.select({ ...getTableColumns(decisions), cursorExpiresAt: sql`${decisions.expiresAt}::text` }) + .from(decisions).where(and(open, lte(decisions.expiresAt, now), cursor + ? sql`(${decisions.expiresAt}, ${decisions.id}) > (${cursor.expiresAt}::timestamptz, ${cursor.id}::uuid)` : undefined)) + .orderBy(asc(decisions.expiresAt), asc(decisions.id)).limit(limit); + } + + async function sweep(now = new Date(), signal?: AbortSignal) { + let expired = 0; + if (signal?.aborted) return { expired }; + const errors: unknown[] = []; + const configuredBatchSize = Number(process.env.PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE ?? 100); + const batchSize = Number.isFinite(configuredBatchSize) ? Math.max(1, Math.trunc(configuredBatchSize)) : 100; + const rows = await due(now, batchSize, recoveryCursor); + for (const row of rows) { + if (signal?.aborted) return { expired }; + try { expired += await expire(row, "ttl", now); } catch (error) { errors.push(error); } + recoveryCursor = { expiresAt: row.cursorExpiresAt, id: row.id }; + } + if (rows.length < batchSize) recoveryCursor = undefined; + if (errors.length === 1) throw errors[0]; + if (errors.length) throw new AggregateError(errors, "Decision expiry failed"); + return { expired }; + } + + async function next(): Promise { + const [row] = await db.select({ at: decisions.expiresAt }).from(decisions).where(open) + .orderBy(asc(decisions.expiresAt), asc(decisions.id)).limit(1); + return row?.at.getTime() ?? null; + } + + return { expire, due, sweep, next }; +} diff --git a/server/src/services/decision-work-notifications.ts b/server/src/services/decision-work-notifications.ts index af1ac08487f..06869ceff59 100644 --- a/server/src/services/decision-work-notifications.ts +++ b/server/src/services/decision-work-notifications.ts @@ -1,9 +1,18 @@ import type { Db } from "@paperclipai/db"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; +export function notifyDecisionCreated(transaction: object) { + return notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionExpiry); +} + +export async function notifyDecisionChange(transaction: object) { + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionContinuation); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionExpiry); +} + export function writeDecisionChange(db: Db, write: (transaction: Db) => Promise): Promise { return db.transaction(async transaction => { - await notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionContinuation); + await notifyDecisionChange(transaction); return write(transaction as unknown as Db); }); } diff --git a/server/src/services/decisions.ts b/server/src/services/decisions.ts index e37d1393b76..751e4c30406 100644 --- a/server/src/services/decisions.ts +++ b/server/src/services/decisions.ts @@ -15,7 +15,8 @@ import { import { decisionRetentionService, hashAttentionArchiveManifest } from "./decision-retention.js"; import { decisionContinuationService, type DecisionContinuationWake } from "./decision-continuations.js"; -import { writeDecisionChange } from "./decision-work-notifications.js"; +import { decisionExpiryService } from "./decision-expiry.js"; +import { notifyDecisionCreated, writeDecisionChange } from "./decision-work-notifications.js"; type Snapshot = { status: string; assigneeAgentId: string | null; assigneeUserId: string | null; updatedAt: string; descendantCount?: number; descendantIds?: string[]; childCount?: number; attentionArchive?: unknown }; @@ -124,6 +125,7 @@ function boardCanActDirectly(actor: AuthorizationActor, companyId: string) { export function decisionService(db: Db, options: DecisionServiceOptions) { const continuations = decisionContinuationService(db, options.wakeOriginAgent); + const expiry = decisionExpiryService(db, continuations.deliver); const scope = databaseWorkScopeKey(db); let flights = effectFlights.get(scope); if (!flights) effectFlights.set(scope, flights = new Map()); @@ -233,6 +235,7 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { ...(input.additionalTargetSnapshots ?? {}), }; const id = randomUUID(); + await notifyDecisionCreated(dbOrTx); const [created] = await dbOrTx.insert(decisions).values({ id, companyId: input.companyId, bundleId: input.bundleId ?? null, originAgentId: input.agentId, originIssueId: provenance.issueId, originRunId: input.runId, ruleKey: input.ruleKey ?? null, title: input.title, body: input.body, options: input.options, inputs: input.inputs ?? null, expiresAt, @@ -653,14 +656,7 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { } if (current.status !== "open") throw conflict("decision_already_resolved", { code: "decision_already_resolved" }); if (current.expiresAt <= new Date()) { - const [expired] = await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "expired", updatedAt: new Date(), metadata: { ...metadata, expiredReason: "ttl", - ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }) - .where(and(eq(decisions.id, current.id), eq(decisions.status, "open"), lte(decisions.expiresAt, new Date()))).returning()); - if (expired) { - await logActivity(db, { companyId: expired.companyId, actorType: "system", actorId: "decision-expiry-sweeper", agentId: expired.originAgentId, - runId: expired.originRunId, action: "decision.expired", entityType: "decision", entityId: expired.id, details: { expiredReason: "ttl" } }); - await continuations.deliver(expired.id); - } + await expiry.expire(current, "ttl", new Date()); throw conflict("decision_expired", { code: "decision_expired" }); } if (!current.options.some((option) => option.id === input.optionId)) throw unprocessable("Unknown optionId"); @@ -720,7 +716,7 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { return { ...bundle, decisions: created }; }); } - async function sweepExpired(now = new Date(), options: { includeContinuations?: boolean } = {}) { + async function sweepExpired(now = new Date(), options: { includeContinuations?: boolean; includeExpiry?: boolean } = {}) { const configuredBatchSize = Number(process.env.PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE ?? 100); const batchSize = Number.isFinite(configuredBatchSize) ? Math.max(1, Math.trunc(configuredBatchSize)) @@ -742,9 +738,7 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { await continuations.deliver(recovered.id); } if (options.includeContinuations !== false) await continuations.sweep(batchSize); - const ttlRows = await db.select().from(decisions) - .where(and(eq(decisions.status, "open"), lte(decisions.expiresAt, now))) - .orderBy(asc(decisions.expiresAt)).limit(batchSize); + const ttlRows = options.includeExpiry === false ? [] : await expiry.due(now, batchSize); const remaining = batchSize - ttlRows.length; const targetRows = remaining > 0 ? await db.select().from(decisions) @@ -766,15 +760,13 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { const targetsCompleted = !targetGone && strictTargetIds.size > 0 && targets.every((target) => target.status === "done") && targets.some((target) => snapshots[target.id]?.status !== "done"); - if (!targetGone && !targetsCompleted && decision.expiresAt >= now) continue; + if (!targetGone && !targetsCompleted && (options.includeExpiry === false || decision.expiresAt > now)) continue; const reason = targetGone ? "target_gone" : targetsCompleted ? "target_completed" : "ttl"; - const [updated] = await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "expired", updatedAt: now, metadata: { ...decision.metadata, expiredReason: reason, - ...(decision.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}) } }).where(and(eq(decisions.id, decision.id), eq(decisions.status, "open"))).returning()); - if (!updated) continue; expired += 1; - await logActivity(db, { companyId: updated.companyId, actorType: "system", actorId: "decision-expiry-sweeper", agentId: updated.originAgentId, runId: updated.originRunId, action: "decision.expired", entityType: "decision", entityId: updated.id, details: { expiredReason: reason } }); - await continuations.deliver(updated.id); } + expired += await expiry.expire(decision, reason, now); + } return { expired, resumed }; } - return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepExpired, sweepContinuations: continuations.sweep }; + return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepExpired, sweepContinuations: continuations.sweep, + sweepTtlExpiry: expiry.sweep, nextExpiryAt: expiry.next }; } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 31ece3af9fb..46e8242508c 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + decisionExpiry: "decision-expiry", decisionContinuation: "decision-continuation", decisionArchiveDelivery: "decision-archive-delivery", From 5787962454c5fe22329cbb7dd703327444d6c019 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 18:12:20 -0500 Subject: [PATCH 28/41] fix(evals): report setup-inclusive model token totals (#17946) https://entire.com/gh/paperclipai/-/trails/108/changes/paperclip/108 ## Thinking Path > - Paperclip Product E2E evaluations run real agent requests. > - Model accounting includes setup, canceled requests and earlier attempts. > - Heartbeat token totals can omit setup requests. > - The shared gateway journal retains final normalized usage for those requests. > - Reports should expose that usage once, with coverage and conflicts visible. ## Linked Issues or Issue Description Refs #17928. **What happened?** The two-case qualification reports showed complete model dollars from the shared ledger, while their headline heartbeat tokens omitted setup. For example, the verified Codex campaign has nine final requests, including setup requests, whose aggregate is only available in private receipts. **Expected behavior** The existing ledger report shows all-request token totals, a setup subtotal and missing/conflicting usage counts. Heartbeat and request totals remain separate. **Steps to reproduce** Run a budgeted native marker case with an asynchronous setup probe, then compare the heartbeat totals with final request observations in its retained ledger. ## What Changed - Aggregate final provider/gateway usage once per model request, including setup and canceled requests. Ignore provisional observations and infrastructure reservations. - Expose uncached input, cache-read input, cache-write input and output separately. Label reasoning as included in output. Preserve agreeing optional reasoning from partial final receipts only when it fits the complete output count. - Check every final provider/gateway observation for conflicting supplied fields, including partial receipts. Exclude ambiguous requests and report their conflicting coverage. Missing token usage remains visible. - Preserve retained reports without the optional token field; validate new counts and totals before dashboard regeneration. - Keep the journal-bound experiment identity stable when adding derived token presentation, so old accepted experiments can still be linked without a reimport. Journal changes still invalidate the receipt. ## Verification - Focused report and scheduled-operations tests: 54 passed. - Eval TypeScript check passed after refreshing ignored Runner declaration outputs. - Regression covers duplicate sources, provisional usage, setup, canceled requests, missing usage, conflicting final usage, infrastructure exclusion and invalid retained totals, including nonzero totals with zero contributing requests. All 32 partial-field subsets are checked in both observation orders. - Regenerated both real retained Claude and Codex reports successfully with all final request tokens and existing experiment receipts, using zero model calls. - Broader checks run in CI. No model calls are needed for report generation. ## Risks The subtotal requires final normalized gateway/provider token observations. Older journals and other telemetry sources can have incomplete coverage, which remains explicit. Token totals do not establish invoice reconciliation. The change does not alter paid selection, routing or budgets. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../runner-e2e/braintrust-experiment.test.ts | 29 +++++++- tests/runner-e2e/braintrust-experiment.ts | 4 +- tests/runner-e2e/operations-report.test.ts | 70 ++++++++++++++++++- tests/runner-e2e/operations-report.ts | 47 ++++++++++++- 4 files changed, 144 insertions(+), 6 deletions(-) diff --git a/tests/runner-e2e/braintrust-experiment.test.ts b/tests/runner-e2e/braintrust-experiment.test.ts index 41c61fc5b20..806bfc64a0a 100644 --- a/tests/runner-e2e/braintrust-experiment.test.ts +++ b/tests/runner-e2e/braintrust-experiment.test.ts @@ -4,7 +4,7 @@ import { promisify } from "node:util"; import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; -import { buildBraintrustExperiment, importBraintrustExperiment, renderBraintrustExperiment, type BraintrustExperimentReceipt } from "./braintrust-experiment.js"; +import { buildBraintrustExperiment, experimentEvidenceHash, importBraintrustExperiment, renderBraintrustExperiment, type BraintrustExperimentReceipt } from "./braintrust-experiment.js"; import { regenerateRunnerDashboard } from "./dashboard-regenerate.js"; import { buildRunnerCampaign } from "./history.js"; import { runnerMatrix } from "./catalog.js"; @@ -326,3 +326,30 @@ it.each(["public", "wrong-project", "wrong-hash", "partial-insert", "receiver-er expect(JSON.stringify(receipts)).not.toMatch(/private-key|private-upstream/); if (["partial-insert", "receiver-error"].includes(kind)) expect(receipts[0]).toMatchObject({ status: "created", acceptedRows: 0 }); }); + + +it("preserves retained experiment identity when adding journal-derived token presentation", async () => { + const { openEvalCostLedger } = await import("./cost-ledger.js"); + const { readEvalOperationsReport } = await import("./operations-report.js"); + const { campaign, originals } = fixture(); + const root = await mkdtemp(path.join(tmpdir(), "experiment-token-presentation-")), file = path.join(root, "ledger.jsonl"); + const ledger = await openEvalCostLedger(file, { campaignId: campaign.campaignId, maxUsd: "1", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 2, maxConcurrent: 1 }); + try { + await ledger.reserve("setup", { executionId: originals[0]!.executionId, attemptId: "first", phase: "setup" }, "0.2"); + await ledger.observe({ observationId: "usage", requestId: "setup", provider: "fixture", source: "gateway", currency: "USD", route: "direct", classification: "reported", costUsd: "0.05", final: true, + tokens: { input: 1, cachedRead: 2, cachedWrite: 3, output: 4 } }); + await ledger.finish("setup", "cancelled"); + const enriched = await readEvalOperationsReport(file); + if (enriched.status === "unavailable") throw new Error("Expected token report"); + campaign.operations = structuredClone(enriched); + delete campaign.operations.modelAccounting!.tokenUsage; + const legacyHash = experimentEvidenceHash(campaign); + const receipt = await importBraintrustExperiment({ projectId, apiKey: "key" }, campaign, originals, async () => {}, receiver(campaign, originals)); + campaign.braintrustExperiment = receipt; + campaign.operations = enriched; + expect(experimentEvidenceHash(campaign)).toBe(legacyHash); + expect(renderBraintrustExperiment(campaign)).toContain("2/2 rows accepted"); + campaign.operations.journalHash = "f".repeat(64); + expect(() => renderBraintrustExperiment(campaign)).toThrow("Invalid retained"); + } finally { await ledger.close(); await rm(root, { recursive: true, force: true }); } +}); diff --git a/tests/runner-e2e/braintrust-experiment.ts b/tests/runner-e2e/braintrust-experiment.ts index 6db75341ede..61c71d1341c 100644 --- a/tests/runner-e2e/braintrust-experiment.ts +++ b/tests/runner-e2e/braintrust-experiment.ts @@ -114,13 +114,15 @@ export interface BraintrustExperimentReceipt { } export function experimentEvidenceHash(campaign: RunnerE2ECampaign) { + const operations = campaign.operations ? structuredClone(campaign.operations) : null; + if (operations && operations.status !== "unavailable" && operations.modelAccounting) delete operations.modelAccounting.tokenUsage; return digest({ campaignId: campaign.campaignId, expected: [...campaign.expected].sort(), selected: campaign.results.map(result => ({ executionId: result.executionId, accountingIdentity: result.accountingIdentity, attempt: result.attempt, startedAt: result.attempt === 0 ? null : result.startedAt, sourceRevision: result.source?.sha ?? null, suiteDefinitionHash: result.suiteDefinitionHash ?? null, suiteId: result.suiteId, profileId: result.profileId, caseId: result.caseId, provider: result.provider, model: result.model, environmentId: result.environmentId, runtimeMode: result.runtimeMode })).sort((a, b) => digest(a).localeCompare(digest(b))), - accounting: campaign.accounting, operations: campaign.operations ?? null, feedbackReview: campaign.feedbackReview ?? null }); + accounting: campaign.accounting, operations, feedbackReview: campaign.feedbackReview ?? null }); } function safeBraintrustUrl(value: unknown): value is string { diff --git a/tests/runner-e2e/operations-report.test.ts b/tests/runner-e2e/operations-report.test.ts index a310ddd35e9..56a29536364 100644 --- a/tests/runner-e2e/operations-report.test.ts +++ b/tests/runner-e2e/operations-report.test.ts @@ -2,7 +2,7 @@ import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import path from "node:path"; import { expect, it } from "vitest"; -import { openEvalCostLedger } from "./cost-ledger.js"; +import { openEvalCostLedger, type EvalCostObservation } from "./cost-ledger.js"; import { readEvalOperationsReport, renderEvalOperationsReport, validateEvalOperationsReport } from "./operations-report.js"; import { buildRunnerCampaign } from "./history.js"; import { regenerateRunnerDashboard } from "./dashboard-regenerate.js"; @@ -91,3 +91,71 @@ it("keeps an in-flight final receipt reserved until the forwarding request finis expect(report).toMatchObject({ inflight: 1, knownUsd: "0.050000000", heldUsd: "0.150000000", committedUsd: "0.200000000", unresolvedRequests: 0 }); } finally { await ledger.close(); await rm(root, { recursive: true, force: true }); } }); + +it("reports final setup and canceled-request tokens once while exposing missing and conflicting usage", async () => { + const root = await mkdtemp(path.join(tmpdir(), "eval-token-report-")), file = path.join(root, "ledger.jsonl"); + const ledger = await openEvalCostLedger(file, { campaignId: "tokens", maxUsd: "10", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 10, maxConcurrent: 1 }); + const observation = { provider: "fixture", currency: "USD" as const, route: "direct" as const, classification: "reported" as const, costUsd: "0.05", final: true }; + try { + for (const [requestId, phase] of [["setup", "setup"], ["canceled", "agent"], ["missing", "agent"], ["conflict", "agent"], ["full-conflict", "agent"], ["reasoning-conflict", "agent"], ["overhead", "infrastructure"]] as const) { + await ledger.reserve(requestId, { executionId: "cell", attemptId: "first", phase }, "1"); + if (requestId !== "missing") { + const tokens = { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, ...(requestId === "reasoning-conflict" ? {} : { reasoning: 2 }) }; + await ledger.observe({ ...observation, observationId: requestId + "-partial", requestId, source: "gateway", tokens: { ...tokens, input: 1 }, final: false }); + await ledger.observe({ ...observation, observationId: requestId + "-gateway", requestId, source: "gateway", tokens }); + await ledger.observe({ ...observation, observationId: requestId + "-provider", requestId, source: "provider", tokens }); + if (requestId === "setup") await ledger.observe({ ...observation, observationId: "native-telemetry", requestId, source: "native_otlp", tokens: { ...tokens, input: 999 } }); + if (requestId === "conflict") await ledger.observe({ ...observation, observationId: "conflicting-usage", requestId, source: "provider", tokens: { input: 11 } }); + if (requestId === "reasoning-conflict") await ledger.observe({ ...observation, observationId: "partial-reasoning", requestId, source: "provider", tokens: { reasoning: 9 } }); + if (requestId === "full-conflict") await ledger.observe({ ...observation, observationId: "full-conflicting-usage", requestId, source: "provider", tokens: { ...tokens, input: 11 } }); + } + await ledger.finish(requestId, requestId === "canceled" ? "cancelled" : "succeeded"); + } + const report = await readEvalOperationsReport(file); + if (report.status === "unavailable") throw new Error("Expected readable token report"); + expect(report.modelAccounting?.tokenUsage).toEqual({ requests: 2, conflictingRequests: 3, + totals: { input: 20, cachedRead: 40, cachedWrite: 60, output: 10, reasoning: 4 }, + setup: { requests: 1, totals: { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, reasoning: 2 } } }); + const rendered = renderEvalOperationsReport(report); + expect(rendered).toContain("2/6 model requests have unambiguous final usage; 3 conflicting; 1 missing"); + expect(rendered).toContain("Reported reasoning: 4, included in output"); + expect(rendered).toContain("Setup token usage (1 requests, included above)"); + const emptySetup = structuredClone(report); + emptySetup.modelAccounting!.tokenUsage!.setup.requests = 0; + expect(() => validateEvalOperationsReport(emptySetup)).toThrow("Invalid retained"); + const emptyOverall = structuredClone(report); + emptyOverall.modelAccounting!.tokenUsage!.requests = 0; + emptyOverall.modelAccounting!.tokenUsage!.setup = { requests: 0, totals: { input: 0, cachedRead: 0, cachedWrite: 0, output: 0, reasoning: 0 } }; + expect(() => validateEvalOperationsReport(emptyOverall)).toThrow("Invalid retained"); + const changed = structuredClone(report); + changed.modelAccounting!.tokenUsage!.setup.totals.input = 21; + expect(() => validateEvalOperationsReport(changed)).toThrow("Invalid retained"); + changed.modelAccounting!.tokenUsage!.setup.totals.input = Number.MAX_SAFE_INTEGER + 1; + expect(() => validateEvalOperationsReport(changed)).toThrow("Invalid retained"); + } finally { await ledger.close(); await rm(root, { recursive: true, force: true }); } +}); + + +it("preserves agreed final usage across every partial-field subset and observation order", async () => { + const root = await mkdtemp(path.join(tmpdir(), "eval-token-subsets-")), file = path.join(root, "ledger.jsonl"); + const ledger = await openEvalCostLedger(file, { campaignId: "subsets", maxUsd: "10", deadlineAt: "2099-01-01T00:00:00Z", maxRequests: 64, maxConcurrent: 1 }); + const values = { input: 10, cachedRead: 20, cachedWrite: 30, output: 5, reasoning: 2 }; + const keys = ["input", "cachedRead", "cachedWrite", "output", "reasoning"] as const; + try { + for (let mask = 0; mask < 32; mask++) for (const reverse of [false, true]) { + const requestId = `request-${mask}-${reverse}`, partial: EvalCostObservation["tokens"] = {}; + keys.forEach((key, index) => { if (mask & 1 << index) partial[key] = values[key]; }); + await ledger.reserve(requestId, { executionId: "cell", attemptId: "attempt", phase: "agent" }, "0.2"); + const sources = [{ source: "gateway" as const, tokens: { input: 10, cachedRead: 20, cachedWrite: 30, output: 5 } }, { source: "provider" as const, tokens: partial }]; + if (reverse) sources.reverse(); + for (const observation of sources) await ledger.observe({ ...observation, observationId: requestId + observation.source, requestId, + provider: "fixture", currency: "USD", route: "direct", classification: "reported", costUsd: "0.05", final: true }); + await ledger.finish(requestId, "cancelled"); + } + const report = await readEvalOperationsReport(file); + if (report.status === "unavailable") throw new Error("Expected usable accounting for agreeing partial receipts"); + expect(report.modelAccounting?.tokenUsage).toMatchObject({ requests: 64, conflictingRequests: 0, + totals: { input: 640, cachedRead: 1280, cachedWrite: 1920, output: 320, reasoning: 64 } }); + expect(report.knownUsd).toBe("3.200000000"); + } finally { await ledger.close(); await rm(root, { recursive: true, force: true }); } +}); diff --git a/tests/runner-e2e/operations-report.ts b/tests/runner-e2e/operations-report.ts index 80c3affe067..96df96e2559 100644 --- a/tests/runner-e2e/operations-report.ts +++ b/tests/runner-e2e/operations-report.ts @@ -3,12 +3,17 @@ import { usdToUnits } from "../../packages/shared/src/money.js"; import { html, isPublicReportTimestamp } from "./report-format.js"; const money = (n: bigint) => `${n / 1_000_000_000n}.${String(n % 1_000_000_000n).padStart(9, "0")}`; +interface ModelTokenTotals { input: number; cachedRead: number; cachedWrite: number; output: number; reasoning: number } +interface ModelTokenUsage { requests: number; conflictingRequests: number; totals: ModelTokenTotals; setup: { requests: number; totals: ModelTokenTotals } } +const tokenFields = ["input", "cachedRead", "cachedWrite", "output", "reasoning"] as const; +const emptyTokenTotals = (): ModelTokenTotals => ({ input: 0, cachedRead: 0, cachedWrite: 0, output: 0, reasoning: 0 }); + interface EvalOperationsAvailableReport { version: 1; scope: "shared_gateway_ledger"; status?: never; accountingReadiness?: "model_requests"; - modelAccounting?: { requests: number; unresolvedRequests: number; conflictingRequests: number; inflight: number; knownUsd: string; heldUsd: string }; + modelAccounting?: { requests: number; unresolvedRequests: number; conflictingRequests: number; inflight: number; knownUsd: string; heldUsd: string; tokenUsage?: ModelTokenUsage }; ledgerCampaignId: string; journalHash: string; records: number; @@ -53,7 +58,20 @@ export function validateEvalOperationsReport(report: EvalOperationsReport): void if (report.accountingReadiness !== undefined && (report.accountingReadiness !== "model_requests" || !report.modelAccounting)) fail(); if (report.modelAccounting !== undefined) { const model = report.modelAccounting; - fields(model, ["requests", "unresolvedRequests", "conflictingRequests", "inflight", "knownUsd", "heldUsd"]); + fields(model, ["requests", "unresolvedRequests", "conflictingRequests", "inflight", "knownUsd", "heldUsd", "tokenUsage"]); + if (model.tokenUsage !== undefined) { + const usage = model.tokenUsage; + fields(usage, ["requests", "conflictingRequests", "totals", "setup"]); + fields(usage.setup, ["requests", "totals"]); + [usage.requests, usage.conflictingRequests, usage.setup.requests].forEach(count); + if (usage.requests + usage.conflictingRequests > model.requests || usage.setup.requests > usage.requests) fail(); + for (const [requests, totals] of [[usage.requests, usage.totals], [usage.setup.requests, usage.setup.totals]] as const) { + fields(totals, [...tokenFields]); + tokenFields.forEach(key => count(totals[key])); + if (totals.reasoning > totals.output || requests === 0 && tokenFields.some(key => totals[key] !== 0)) fail(); + } + if (tokenFields.some(key => usage.setup.totals[key] > usage.totals[key])) fail(); + } [model.requests, model.unresolvedRequests, model.conflictingRequests, model.inflight].forEach(count); if (model.requests > report.requests || model.unresolvedRequests > model.requests || model.unresolvedRequests > report.unresolvedRequests || model.conflictingRequests > model.unresolvedRequests || model.conflictingRequests > report.conflictingRequests @@ -104,11 +122,29 @@ async function readAvailableEvalOperationsReport(file: string): Promise cost ? reserved - cost : 0n); }, 0n); + const tokenUsage: ModelTokenUsage = { requests: 0, conflictingRequests: 0, totals: emptyTokenTotals(), setup: { requests: 0, totals: emptyTokenTotals() } }; + for (const request of loaded.state.requests.values()) { + if (request.identity.phase === "infrastructure") continue; + const candidates = request.observations.filter(observation => observation.final && observation.tokens + && ["gateway", "provider"].includes(observation.source)); + const values = tokenFields.map(key => new Set(candidates.flatMap(observation => observation.tokens![key] === undefined ? [] : [observation.tokens![key]!] ))); + const complete = candidates.find(observation => ["input", "cachedRead", "cachedWrite", "output"].every(key => observation.tokens![key as keyof ModelTokenTotals] !== undefined)); + if (values.some(value => value.size > 1) || complete && candidates.some(observation => (observation.tokens!.reasoning ?? 0) > complete.tokens!.output!)) { tokenUsage.conflictingRequests++; continue; } + if (!complete) continue; + const tokens = Object.fromEntries(tokenFields.map(key => [key, complete.tokens![key] ?? 0])) as unknown as ModelTokenTotals; + tokens.reasoning = candidates.find(observation => observation.tokens!.reasoning !== undefined)?.tokens?.reasoning ?? 0; + tokenUsage.requests++; + if (request.identity.phase === "setup") tokenUsage.setup.requests++; + for (const key of tokenFields) { + tokenUsage.totals[key] += tokens[key]; + if (request.identity.phase === "setup") tokenUsage.setup.totals[key] += tokens[key]; + } + } const report: EvalOperationsAvailableReport = { version: 1, scope: "shared_gateway_ledger", ledgerCampaignId: snapshot.campaignId, ...(loaded.state.limits.overhead?.accountingReadiness ? { accountingReadiness: loaded.state.limits.overhead.accountingReadiness } : {}), modelAccounting: { requests: modelRequests.length, unresolvedRequests: modelRequests.filter(request => !request.receiptComplete).length, conflictingRequests: modelRequests.filter(request => request.conflict).length, inflight: modelRequests.filter(request => !request.finished).length, - knownUsd: money(modelKnown), heldUsd: money(modelHeld) }, + knownUsd: money(modelKnown), heldUsd: money(modelHeld), tokenUsage }, journalHash: loaded.previous, records: loaded.sequence, deadlineAt: new Date(loaded.state.limits.deadlineAt).toISOString(), limitUsd: snapshot.limitUsd, knownUsd: snapshot.knownUsd, heldUsd: snapshot.heldUsd, committedUsd: snapshot.committedUsd, requests: snapshot.requests.length, unresolvedRequests: snapshot.unknownRequests, boundedUnresolvedRequests: snapshot.boundedUnresolvedRequests, conflictingRequests: snapshot.conflictingRequests, @@ -141,6 +177,11 @@ export function operationsReportText(report: EvalOperationsReport): string[] { ]; return [ ...(report.modelAccounting ? [`Model requests: ${report.modelAccounting.requests}; $${report.modelAccounting.knownUsd} known token cost; $${report.modelAccounting.heldUsd} additional held; ${report.modelAccounting.unresolvedRequests} unresolved.`] : []), + ...(report.modelAccounting?.tokenUsage ? (() => { + const model = report.modelAccounting!, usage = model.tokenUsage!, total = usage.totals, setup = usage.setup.totals; + return [`Ledger token usage: ${usage.requests}/${model.requests} model requests have unambiguous final usage; ${usage.conflictingRequests} conflicting; ${model.requests - usage.requests - usage.conflictingRequests} missing. Totals: ${total.input} uncached input / ${total.cachedRead} cache-read input / ${total.cachedWrite} cache-write input / ${total.output} output. Reported reasoning: ${total.reasoning}, included in output.`, + `Setup token usage (${usage.setup.requests} requests, included above): ${setup.input} uncached input / ${setup.cachedRead} cache-read input / ${setup.cachedWrite} cache-write input / ${setup.output} output. Completed, failed and canceled requests are included once; token coverage is separate from invoice reconciliation.`]; + })() : []), ...(report.accountingReadiness === "model_requests" ? [`Readiness scope: model requests; status: ${evalReadinessStatus(report)}. Operating costs remain separately unreconciled unless final receipts establish otherwise. Their reservations remain inside campaign admission; this does not establish a full-lifecycle operating-cost bound.`] : []), `Accounting status: ${evalAccountingStatus(report)}. ${report.boundedUnresolvedRequests ?? 0} pending charges have reviewed bounds; ${report.unresolvedRequests - (report.boundedUnresolvedRequests ?? 0)} do not. Provisional completion preserves holds and requires later reconciliation; it does not establish invoice completeness.`, `Shared request ledger ${report.ledgerCampaignId}: $${report.knownUsd} known subtotal; $${report.heldUsd} additional held reservations; $${report.committedUsd} committed against $${report.limitUsd}.`, From e9e84bb8a16393543effd1d45f34a4aad535d394 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 18:24:55 -0500 Subject: [PATCH 29/41] fix(server): recover decision effects from saved claim deadlines (#17482) ## Thinking Path > - Paperclip manages agent work and the decisions that authorize it. > - A committed decision can leave effects unfinished after an interrupted execution. > - Recovery must preserve that debt without scanning settled history forever. > - Claims already store the state and time needed to schedule recovery. > - A failed claim must get another attempt even while newer claims arrive. > - Malformed claims need a durable failure outcome instead of an endless retry. > - This PR uses committed signals, saved deadlines, and bounded recovery rotations. ## Linked Issues or Issue Description **What happened?** Periodic decision maintenance queries for unfinished effects even when none remain. A cursor that only wraps after a short page can postpone an old failure indefinitely under continuous new work. Missing saved deciders or chosen options can also leave non-executable claims scanning forever. **Expected behavior** Open decisions and completed history produce no recurring effect-recovery SQL. Committed claims wake recovery, and restart reconstructs their deadlines. Genuine failures retain retries. Known malformed outcomes settle with an audit and preserve completed effects. Recovery must not invent authority, repeat committed effects, or lose continuation delivery. **Steps to reproduce** Create an old failing claim with a one-row recovery page, then keep adding newer due claims. Before the rotation fix, the old failure is never revisited. Separately, recover a claim with a missing decider or chosen option: the old implementation skips or throws on every attempt without settling it. Creating an ordinary open decision also caused an unnecessary effect scan. Stacked on #17479. Other decision expiry, continuation, target and retention duties have separate PRs in the stack. ## What Changed - Coalesce sequential committed notifications for up to one second without delaying already saved deadlines. This bounds instance-wide recovery reads and adds no idle polling. - Register effect recovery with the shared scheduler and transaction-aware notifications. Derive deadlines from the existing claim timestamp and recovery grace. - Remove effect recovery from recurring aggregate maintenance when the worker owns it. Preserve the explicit combined recovery entry point. - Page within a frozen final timestamp/ID and retain PostgreSQL precision at both boundaries. Failed claims and healthy later claims both make progress. - Load the current claim and derive recovery authority inside the shared execution flight. Preserve foreground joining, current authorization and per-effect locks. - Recheck known malformed claims under a row lock. Save a failed outcome, or partial when an executed effect receipt exists, with `effectRecoveryError` and an atomic `decision.recovery_failed` audit. Publish activity after commit and use normal continuation delivery. - Avoid effect-recovery notifications for open-decision creation. Claim and settlement writes retain their existing notification boundary. - Add generated migration `0390_salty_killraven.sql`, the concurrent, idempotent partial index for running-claim lookup. No queue table is added. - Cover quietness, transaction and lifecycle boundaries, retry fairness, malformed outcomes, and concurrent repair. Document the contract in `doc/idle-decision-effect-recovery.md`. ## Verification Current candidate normally integrates landed expiry `3421366903`, including expiry-only creation signals and all 21 independent master changes. The test revision is `dba8a79d1e1a5b97b8b5f6dac0f7c7837536531a`; the publication adds only this evidence documentation. - **143 current cases pass** across all 26 recovery, 54 decision, 20 expiry, 42 startup and one scoped-deadline cases. Server types/build pass with the rebuilt Runner vendor (1,524 inputs and 1,563 outputs). These results follow the landed dependency reconciliation; prior 69- and 141-case results remain historical evidence for their stated revisions. - Generated migration `0390_salty_killraven.sql` adds the existing running-claim partial index through the concurrent executor. All 388 parent journal entries are unchanged; five generated snapshots remain. Schema drift, numbering/safety and actual migration apply/replay pass. - Existing regressions exercise failed-claim rotation under sustained arrivals, malformed decider/option outcomes, atomic audit failure, row-lock repair races, current authority, foreground effect joining, coalesced tenant notifications, transaction commit/rollback, drain/resume, paging with PostgreSQL timestamp precision, and shutdown/restart. Completed effects are not repeated. - The first integrated run passed 140 cases and failed an overlap test that assumed the effect deadline was the scheduler's earliest deadline. Continuation coalescing now creates an earlier registered wake. The test services that earlier wake before asserting the original effect grace deadline; it still requires one shared effect flight, two lock attempts including the test gate, one committed comment and one continuation. No production timing was relaxed. - Earlier red/green fairness, malformed-claim, notification amplification and fixture failures remain recorded in [the owning audit](doc/idle-decision-effect-recovery.md). Extra upstream embedded-only tests attempted during parent integration could not initialize locally; CI must exercise that environment. - Previous candidate `f54a707d74` received both 5/5 reviews but its CI server shard 6 was interrupted (run 40597921712). The step was cancelled, with no failing assertion or suite summary, and the verify gate also failed. No retry or passing claim was made for that run. The necessary parent repair is included here; fresh CI and both reviews are required. Commands: server typecheck/build and Vitest for `decision-effect-recovery-work`, `decisions-service`, `decision-expiry-work`, `server-startup-feedback-export`, and `scoped-deadline-work`; DB migration checks, schema snapshot drift and real migration apply/replay. No full local repository-suite or deployed sleep/cost claim is made. Fresh current-head CI and Greptile/Ponytail are required before merge. ## Risks Missing deciders and missing chosen options now produce durable failed or partial outcomes instead of remaining running forever. Their audit and receipts remain available; no effect executes under invented authority. Only these known malformed states are settled this way. Database, audit, and other execution failures retain retries. Continuations retain their existing at-least-once delivery contract. The supported writer model remains process-local notifications plus restart reconstruction. Independent SQL writers and multiple replicas gain no new wake protocol. Cloud admission and durable external alarms are unchanged. These tests establish worker behavior, not deployed host suspension or fleet savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-decision-effect-recovery.md | 255 +++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 105 +++- packages/db/src/client.ts | 2 +- .../src/migrations/0390_salty_killraven.sql | 1 + ...{0385_snapshot.json => 0390_snapshot.json} | 113 +++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/decisions.ts | 2 + .../decision-effect-recovery-work.test.ts | 500 ++++++++++++++++++ .../server-startup-feedback-export.test.ts | 7 +- server/src/index.ts | 7 +- .../services/decision-effect-recovery-work.ts | 17 + .../src/services/decision-effect-recovery.ts | 60 +++ .../services/decision-work-notifications.ts | 1 + server/src/services/decisions.ts | 93 ++-- .../services/delivery-work-notifications.ts | 1 + 15 files changed, 1131 insertions(+), 40 deletions(-) create mode 100644 doc/idle-decision-effect-recovery.md create mode 100644 packages/db/src/migrations/0390_salty_killraven.sql rename packages/db/src/migrations/meta/{0385_snapshot.json => 0390_snapshot.json} (99%) create mode 100644 server/src/__tests__/decision-effect-recovery-work.test.ts create mode 100644 server/src/services/decision-effect-recovery-work.ts create mode 100644 server/src/services/decision-effect-recovery.ts diff --git a/doc/idle-decision-effect-recovery.md b/doc/idle-decision-effect-recovery.md new file mode 100644 index 00000000000..b6724feed72 --- /dev/null +++ b/doc/idle-decision-effect-recovery.md @@ -0,0 +1,255 @@ +# Idle decision-effect recovery + +Interrupted decision effects now have a dedicated deadline worker. The main +maintenance tick no longer scans running decisions. Target invalidation remains +in that tick; TTL expiry and continuation delivery have their own workers. + +## Obligation and activation + +The existing decision row is durable truth. A row with `status = decided` and +`executionStatus = running` owes effect recovery. Its next deadline is +`updatedAt + PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS`, default 60 seconds. Zero is +supported; invalid or negative values retain the existing default. Reads do not +move the timestamp. Open decisions and completed history create no recovery debt. + +The existing `writeDecisionChange` transaction boundary signals effect recovery +for running claims, effect/archive settlement, cancellation, dismissal and expiry. +The generic `notifyDecisionChange` helper signals continuation and expiry only; +open-decision creation does not owe effect recovery. Notifications are awaited +before writes. Outer commit activates work; rollback does not publish a claim. +The additional topic does not wrap provider work or agent wakes in SQL transactions +and does not introduce a new queue table. + +Committed notification bursts coalesce for up to one second after a successful +pass. This bounds sequential tenant writes to at most one notification-driven +global pass per window, rather than only coalescing overlapping scans. A saved +recovery deadline can preempt that window. New obligations discovered by a signal +can wait up to one second; foreground effects still execute immediately. The +window alone never schedules work when there is no notification or saved debt. + +The supported writer model is the application DB signal owner and its child pools. +Independent SQL clients and multiple active writers need a separate notification +contract. Startup reconstructs committed claims; it does not observe future writes +from unsupported writers. + +## Ownership and recovery + +`decision-effect-recovery.ts` owns selection, grace, bounded paging, and earliest +deadline lookup. `decisions.ts` owns execution, current membership, effect claims, +audit, archive policy, and continuation delivery. The worker adapter only binds +that domain contract to the shared coordinator. Startup registers it only when +heartbeat scheduling is enabled. The public combined sweep retains its default +recovery behavior for existing callers. + +Each pass honors the existing configured batch size (default 100, finite values +clamped to at least one integer). Its timestamp/ID cursor advances past failed +candidates within a frozen final timestamp/ID. Newer due claims wait for the next +rotation, so a continuous stream cannot postpone an earlier failure forever. +Both cursor positions retain PostgreSQL precision. Genuine execution or settlement +failures retain their retry obligation. + +A missing saved decider or chosen option is a deterministic malformed outcome. +Recovery rechecks it under a row lock, records an audited failed outcome (partial +if a completed effect receipt exists), and preserves all effect receipts. It +creates no authority and executes no effect for the malformed claim. The failure +reason and continuation obligation are saved in the same transaction as the audit. +An audit or settlement failure rolls back and leaves the claim pending. A claim +repaired before the lock is acquired remains eligible for normal recovery. + +The decision service re-reads current status before recovery. Foreground and +recovery execution share the existing per-decision flight scoped to the DB owner. +Individual effects keep their durable records and transaction advisory locks. +Completed effects are not repeated after a failed final decision status write. +Recovery derives authority from the saved decider's current active membership. +It preserves the current authorization intersection and current target checks. + +The scheduler has a 30-second minimum retry spacing for pending or failed work. +A first claim or failed retry during drain retains its wake without passive SQL. +Shutdown stops subscriptions and checks cancellation between candidates. An +already admitted effect/continuation is awaited; later candidates remain durable +for restart. No periodic pass remains after all claims settle. + +## Existing guarantees and limits + +Regular effects preserve their existing per-effect transaction boundaries. +Continuation delivery retains its separate at-least-once wake/acknowledgement +contract. This migration does not establish exactly-once external delivery. + +Archive proposals have a separate versioned manifest contract. Recovery still +verifies the saved signature/manifest and current authority. A crash after an +archive commits but before the decision settles can encounter the changed +manifest version on restart. The existing result is a failed decision with an +archive error, while the archive remains committed once. The new worker preserves +and tests that outcome; it does not claim to repair this preexisting atomicity gap. + +Cloud sleep guards remain conservative. The internal next deadline is not a +persisted external alarm and does not authorize host sleep with running decisions. +This change establishes worker quiet, not actual instance/database suspension. + +## Verification + +The new real-PostgreSQL suite covers 17 cases: empty/open/completed-history quiet, +real foreground claim and failed settlement, exact grace and zero override, +restart from a claimed effect, active foreground/recovery overlap, outer commit +and rollback, first claim and failed retry through drain, revoked authority, +continuation failure, aggregate exclusion, fair paging past missing-decider and +malformed claims, archive recovery before and after an archive commit, a claim +created during delivery, and shutdown between candidates. Quiet assertions observe +SQL methods through one hour of fake time after settling. + +The combined selection passes 212 tests: the 17 new cases, 50 existing decision +service cases, 14 TTL cases, 39 startup cases, and 92 agent-history route cases. +The first 15-case recovery run also passed. Workspace typecheck/build and the +final server typecheck pass. No failure was suppressed or timeout increased. + +The current regenerated migration `0390_salty_killraven.sql` adds one partial index concurrently, with `IF NOT EXISTS`, through +the migration owner's top-level statement path. Generation retains the newest +five schema snapshots. Fresh application, replay, safety, and numbering checks +pass. With 40,000 completed decisions and one running claim, both due selection +and earliest-deadline lookup use the partial index and read two buffers (0.012 ms +and 0.003 ms in the local probe). The probe transaction is rolled back. + +Full PR CI and review must pass separately before this slice is ready. + +A follow-up regression seeds a legacy invalid claim with a PostgreSQL-generated +microsecond timestamp ahead of a valid claim. With a one-row page, the original +Date cursor selected the invalid claim again and starved the valid effect. That +regression failed before repair. Recovery now retains the timestamp as database +text for cursor comparisons, leaving business grace deadlines unchanged. The +unresolved claim remains durable debt and is revisited after paging wraps; the +valid claim progresses without repeating its completed effect. The regression +now holds a candidate unresolved through the dispatch callback, because known +malformed claims receive a durable failed outcome as described above. +All 18 effect-recovery lifecycle cases and server typechecking pass after repair. + +### Foreground join test timing + +CI exposed a timing assumption in the foreground/recovery overlap test: it +expected exactly two lifetime recovery sweeps. A controlled PostgreSQL replay +waited for the committed claim's deadline discovery before advancing time. It +recorded two completed, zero-resumption scans before the deadline (startup and +claim notification), followed by the legitimate deadline scan. The old assertion +then failed deterministically with three calls. No recurring idle poll or duplicate +effect was found in that sequence. + +The test now waits for discovery, measures the deadline recovery relative to that +point, and proves that its promise stays pending while foreground execution holds +the effect lock. Advancing through several retry periods cannot start another +sweep or effect. After release, recovery completes, one comment and one continuation +wake exist, and the worker performs no SQL through the existing hour-long quiet +check. Counting actual lock attempts also distinguishes joining from a second +execution blocked on the same PostgreSQL advisory lock. A temporary mutation that +removed flight joining made this stronger assertion fail; production code was +restored before final verification. No runtime code, retry policy, timeout, or +business deadline changed. + +All 123 tests across the four complete effect-recovery, decision-service, expiry, +and startup suites passed, followed by server typechecking. Evidence: +`/tmp/sleep-effect-join-red.log`, `/tmp/sleep-effect-join-green.log`, +`/tmp/sleep-effect-join-mutation.log`, `/tmp/sleep-effect-join-owning-tests.log`, +and `/tmp/sleep-effect-join-root-types.log` with their status files. The initial +failure and mutation failure are expected evidence, not passing runs. Remaining +review findings and fresh stack CI remain separate acceptance requirements. + + +### Recovery review repairs (October 26) + +Two regressions failed before repair: an old failure was not revisited while newer +due claims kept each page full, and creating an open decision caused an unnecessary +effect-recovery scan. Frozen rotation boundaries now preserve both retry fairness +and forward progress. Creation still signals its continuation/expiry consumers; +the effect queue is signaled by the existing claim/settlement write boundary. + +Four malformed-outcome cases also failed before repair. Recovery now records +`effectRecoveryError` and a `decision.recovery_failed` activity atomically. Live +activity publication follows commit. Existing completed effect receipts remain +unchanged, and the normal continuation owner delivers the result. A real audit +insert failure leaves both status and audit uncommitted, then later recovery +settles successfully. A real PostgreSQL row-lock test repairs a claim while +settlement waits; the locked recheck leaves that claim running, and its valid +effect executes once on the next pass. + +The shared execution flight now loads the current claim and derives recovery +authority together. Recovery no longer performs a redundant full-row lookup +before entering the flight. Foreground overlap, current membership checks, +per-effect locks, continuation behavior, and the resumption count are preserved. +Only the two known malformed states are terminalized; other failures retain their +existing retry policy. No schema, queue, or external-wake contract is added. + +All 24 recovery lifecycle cases and server typechecking pass. Controlled failures +and final evidence are retained in `/tmp/sleep-effect-fairness-red.log`, +`/tmp/sleep-effect-invalid-red.log`, `/tmp/sleep-effect-owner-tests.log`, and +`/tmp/sleep-effect-owner-types.log`. Broader stack and remote acceptance remain +separate requirements; these results do not prove host suspension. + +The owning branch also passes all 129 cases across the four complete recovery, +decision-service, expiry, and startup suites. Database-backed suites use separate +migrated disposable databases. See `/tmp/sleep-effect-review-owner-acceptance.log`. + + +### Bounded notification review repair + +Ten sequential real cancellation transactions reproduced ten extra recovery +scans. After enabling the existing shared notification-coalescing capability, +the same transactions cause one follow-up scan and then return to SQL quiet. +A separate real-database case preserves an already loaded deadline 500 milliseconds +away. The initial deadline fixture observed the scheduler while its SQL pass was +still active; waiting for that read removed the fixture race before the baseline +was recorded. The cancellation amplification still failed before the production fix. + +All 131 tests in the four complete recovery, decision-service, expiry and startup +suites pass at the owning branch, as does server typechecking. The helper's optional +coalescing capability is promoted from the later plugin scheduling slice into this +PR; consumers that omit it keep their previous behavior. The integrated stack is +verified separately before publication. The original full-suite and external-wake +limitations remain. + + +## Current parent integration + +The expiry and continuation review repairs are integrated normally. The running +claim index is regenerated as 0390 after parent 0388/0389, with concurrent creation +registered in the existing executor. All 388 parent journal entries are unchanged; +the five newest generated snapshots remain. Schema drift, numbering/safety and +fresh migration apply/replay pass. No generated snapshot was hand-edited. + +All 141 cases in the five complete effect-recovery, decision/continuation, expiry, +startup and scoped-deadline suites pass. This includes all 26 recovery cases and +the parent's deterministic cross-pool archive contention test. Server types and +build pass. The first integrated run passed 140 cases and failed an old scheduler +assertion: the coordinator now also holds an earlier continuation-coalescing +deadline. The overlap test services that earlier registered wake, then still +requires the original effect grace deadline, one active effect flight, two total +advisory-lock attempts (test gate plus foreground), one committed effect and one +continuation. No production timing or retry was changed to pass the test. + +Existing one-second effect-notification coalescing and thirty-second failure +retry remain. Continuation notification coalescing is owned separately by the +parent. Extra upstream embedded-only tests attempted during parent integration +could not initialize locally; fresh current-head CI must cover that environment. +Full CI and both automated reviews remain merge gates. + + +After normal integration of master-target expiry parent `f7c61e9912`, the +effect-recovery owner, worker, decision execution and notification implementations +are unchanged. All 69 current recovery/startup/deadline cases and server types/ +build pass with the updated Runner vendor. The earlier full 141-case selection +remains evidence at its stated revision. Parent migration history is an exact +prefix and five generated snapshots remain. Fresh head CI and both automated +reviews are required after this parent integration. + + +## Landed expiry integration and CI interruption + +Expiry-only creation notification from `aa03d2c4c2` is integrated. All 143 cases +in the complete recovery, decision, expiry, startup and scoped-deadline suites +pass, including single/bundle creation without continuation signals. The normal +merge of landed expiry `3421366903` also preserves all 21 independently merged +master files. The same 143 cases pass after that reconciliation. + +The previous candidate `f54a707d74` had both reviews at 5/5, but CI run +40597921712 failed because server shard 6 was interrupted. Its test step was +cancelled and orphan cleanup ran; no failing test assertion or suite summary was +reported. The workflow eventually completed with that shard and its dependent +verify gate failed. It was not retried or presented as passing. The necessary +expiry parent repair creates a new candidate with fresh CI and reviews required. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 0ff600d0569..c4fd7944142 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -16,7 +16,8 @@ Do not mark an item done from a timer removal or a green check alone. | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | -| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479), effects [#17482](https://github.com/paperclipai/paperclip/pull/17482), and targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates | +| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482), and targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates | + | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | | 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Slices 02–07 merged: [secret proposals](https://github.com/paperclipai/paperclip/pull/17446), [credentials](https://github.com/paperclipai/paperclip/pull/17450), [review expiry](https://github.com/paperclipai/paperclip/pull/17453), [approved recovery](https://github.com/paperclipai/paperclip/pull/17454), [interrupted claims](https://github.com/paperclipai/paperclip/pull/17458), [projection](https://github.com/paperclipai/paperclip/pull/17468) | @@ -527,3 +528,105 @@ Slice 10 creation signal follow-up: single and bundle creation now notify expiry only through the central owner. Terminal updates still signal continuation. Both real regressions failed before repair; all 117 current expiry/decision/startup/ deadline cases, server types and build pass. Fresh CI/reviews remain required. + + + + +Slice 10 is [#17479](https://github.com/paperclipai/paperclip/pull/17479), head +`63b58cc26f`, based on the repaired continuation branch. It is attached to this +chat and review was requested. Workspace typecheck/build passed on the combined +branch after parent propagation. Full CI and review are active, not complete. +Slice 09 has fresh Greptile 5/5 at `5f96892a58`, no unresolved review threads, +and CI active with no failure in the latest snapshot. The first eight published +slices remain open with their recorded passing checks. No merge is authorized. + +## Next owner audit: interrupted decision effects + +The next branch is `codex/sleep-11-decision-effect-recovery`, based on #17479. +No eleventh implementation or PR is claimed. The remaining combined decision +sweep selects `status = decided`, `executionStatus = running`, and +`updatedAt <= now - recoveryGraceMs`. The existing grace defaults to 60 seconds; +invalid values fall back and zero is permitted. The combined batch size defaults +to 100. Preserve these semantics while moving only interrupted effect recovery. +Target invalidation remains a separate pending reason after this migration. + +The existing decision update helper covers the running claim and terminal +settlements. Extend that boundary for effect-recovery intent. Creation is open +and does not itself constitute effect debt. The indexed minimum running claim +plus grace should drive the next recovery attempt; open/terminal history must +not keep it scanning. Preserve expiry and continuation as separate workers. + +Keep effect execution, signatures, recovery authority, effect claims, and +postcommit issue actions in `decisions.ts` and its domain collaborators. A +recovery selector may call a narrow decision-owned operation rather than move +that policy into the scheduler. The new per-owner execution flight already +coalesces foreground and recovery calls and re-reads the current decision before +execution. Reuse it. Do not hold a DB transaction while waiting for the whole +operation or replace durable per-effect claims with the in-memory flight. + +Regular effects use transaction-scoped advisory locks and saved effect execution +rows. Completed effects must not replay; claimed work can resume. Recovery derives +current user authority from stored `decidedByUserId` and current membership. +The existing combined sweep skips rows without a deciding user. Audit how such +invalid legacy rows are represented in readiness and sleep safety; do not allow +them to starve actionable rows or invent permission to execute them. Malformed +stored options or genuine execution failures must retain a visible obligation. + +Archive proposals have separate manifest/version checks and transaction ownership. +The parent race repair serializes archive execution through its state write in +one live owner. It is not a durable cross-process claim and does not establish +exactly-once recovery after a process dies between archive commit and decision +settlement. Preserve and explicitly test the supported failure behavior instead +of treating this limitation as solved by a new timer. + +Use bounded passes, advance past per-row failures, and propagate cancellation +between candidates. A first claim during drain, completion during recovery, and +an unresolved failed write must retain their existing notification semantics. +Reconstruct recovery deadlines on startup, keep new production SQL indexed, and +preserve the public combined sweep's default behavior while removing its effect +scan from the main tick. + +Required evidence includes a real claim with foreground interruption, exact +recovery grace, ordinary active execution coalescing, completed effect history, +claimed-effect restart, changed user authority, invalid legacy rows, provider/ +archive failure semantics, outer commit/rollback, failed retry through drain, +changes during execution, fair bounded paging, and shutdown. Run existing +archive and continuation regressions on the combined stack. The full original +inventory remains the goal after this slice. + + +## Slice 11: interrupted decision effects + +The effect-recovery owner and worker now use committed decision changes and the +minimum running claim timestamp plus the existing recovery grace. The aggregate +tick opts out of this duty while preserving target invalidation. Effect execution, +current authority, claims, audit, and archive policy remain with the decision +service. See [effect recovery](../idle-decision-effect-recovery.md). + +All 212 selected tests pass: 17 new recovery cases, 50 decision service cases, +14 TTL cases, 39 startup cases, and 92 agent-history route cases. Workspace +typecheck/build and final server typecheck pass. Migration apply/replay and a +40,000-row completed-history query probe pass; both lookups use the new partial +index with two buffers. PR creation and full CI/review are next. + +The continuation parent #17478 now has passing checks and Greptile 5/5 at +`5f96892a58`, with both review findings resolved. TTL #17479 has Greptile 5/5 at +`63b58cc26f` and no findings. Its first CI run timed out in the unchanged +agent-history route test. All 92 route tests passed locally; one failed-job retry +was requested. This evidence does not yet establish green TTL CI. The full +original inventory remains in scope; no stack PR has been merged. + +Current slice 11 parent integration passes all 141 effect/decision/expiry/startup/ +deadline cases, server types/build, generated snapshot drift and migration replay. +The recovery index is regenerated as `0390_salty_killraven.sql`; parent history +is unchanged. The initial overlap assertion failure and its earlier-queue deadline +explanation are recorded in the behavior audit. Seven slices are merged; this +PR and the remaining stack still require fresh current-head CI and both reviews. + + +Slice 10 merged after exact-head Greptile/Ponytail 5/5 and all 58 check contexts +passed or were neutral/skipped: `aa03d2c4c2` → `3421366903`. Slice 11 normally +integrates that landed parent and all 21 independent master files. All 143 cases +in its five complete suites pass. Its preceding CI shard interruption remains +recorded in the owning effect-recovery audit; it is not a passing receipt. Fresh +CI/reviews are required on the reconciled head. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index fa192adb0d1..ff714b6919d 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -508,7 +508,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0387_boring_professor_monster.sql", "0388_ambitious_micromax.sql", "0389_brave_hulk.sql", - + "0390_salty_killraven.sql", "0384_funny_adam_warlock.sql", diff --git a/packages/db/src/migrations/0390_salty_killraven.sql b/packages/db/src/migrations/0390_salty_killraven.sql new file mode 100644 index 00000000000..c653ae92d7f --- /dev/null +++ b/packages/db/src/migrations/0390_salty_killraven.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "decisions_running_recovery_idx" ON "decisions" USING btree ("updated_at","id") WHERE "decisions"."status" = 'decided' AND "decisions"."execution_status" = 'running'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0385_snapshot.json b/packages/db/src/migrations/meta/0390_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0385_snapshot.json rename to packages/db/src/migrations/meta/0390_snapshot.json index 5564d696be5..346524fc38b 100644 --- a/packages/db/src/migrations/meta/0385_snapshot.json +++ b/packages/db/src/migrations/meta/0390_snapshot.json @@ -1,6 +1,6 @@ { - "id": "9965d300-e355-4f47-ba0a-951ed3b4b8a2", - "prevId": "4f96f70a-fd19-4d35-ad8a-8d5559b83cbd", + "id": "f38ffbf3-c0ce-435e-aad8-9e5afa6b9417", + "prevId": "394e85a6-7fc2-4ab1-80a3-33bb4c7003aa", "version": "7", "dialect": "postgresql", "tables": { @@ -22110,6 +22110,34 @@ "concurrently": false, "method": "btree", "with": {} + }, + "decision_archive_notification_outbox_delivery_idx": { + "name": "decision_archive_notification_outbox_delivery_idx", + "columns": [ + { + "expression": "last_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "first" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decision_archive_notification_outbox\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -24030,6 +24058,66 @@ "method": "btree", "with": {} }, + "decisions_running_recovery_idx": { + "name": "decisions_running_recovery_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" = 'running'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_open_expiry_idx": { + "name": "decisions_open_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'open'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_pending_continuation_idx": { + "name": "decisions_pending_continuation_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"continuation_policy\" = 'wake_origin_agent' AND \"decisions\".\"metadata\" ->> 'continuationPending' = 'true' AND (\"decisions\".\"status\" IN ('expired', 'cancelled') OR (\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" IN ('succeeded', 'partial', 'failed')))", + "concurrently": false, + "method": "btree", + "with": {} + }, "decisions_bundle_idx": { "name": "decisions_bundle_idx", "columns": [ @@ -52909,6 +52997,27 @@ "method": "btree", "with": {} }, + "tool_action_requests_company_interaction_idx": { + "name": "tool_action_requests_company_interaction_idx", + "columns": [ + { + "expression": "company_id", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "interaction_id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "concurrently": false, + "method": "btree", + "with": {} + }, "tool_action_requests_issue_idx": { "name": "tool_action_requests_issue_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 14f1a4ad042..9db984a3918 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2717,6 +2717,13 @@ "when": 1793138457589, "tag": "0389_brave_hulk", "breakpoints": true + }, + { + "idx": 390, + "version": "7", + "when": 1793139659776, + "tag": "0390_salty_killraven", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/decisions.ts b/packages/db/src/schema/decisions.ts index 5fe03d8f090..012d2af3d58 100644 --- a/packages/db/src/schema/decisions.ts +++ b/packages/db/src/schema/decisions.ts @@ -69,6 +69,8 @@ export const decisions = pgTable( table.status, table.expiresAt, ), + runningRecoveryIdx: index("decisions_running_recovery_idx").on(table.updatedAt, table.id) + .where(sql`${table.status} = 'decided' AND ${table.executionStatus} = 'running'`), openExpiryIdx: index("decisions_open_expiry_idx").on(table.expiresAt, table.id) .where(sql`${table.status} = 'open'`), pendingContinuationIdx: index("decisions_pending_continuation_idx").on(table.id) diff --git a/server/src/__tests__/decision-effect-recovery-work.test.ts b/server/src/__tests__/decision-effect-recovery-work.test.ts new file mode 100644 index 00000000000..4358b8e3215 --- /dev/null +++ b/server/src/__tests__/decision-effect-recovery-work.test.ts @@ -0,0 +1,500 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, agents, applyPendingMigrations, authUsers, companies, companyMemberships, createDb, decisionEffectExecutions, + decisions, decisionRetention, decisionTargetIssues, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { decisionService } from "../services/decisions.js"; +import { decisionEffectRecoveryService } from "../services/decision-effect-recovery.js"; +import { registerDecisionEffectRecoveryWork } from "../services/decision-effect-recovery-work.js"; +import { registerDecisionContinuationWork } from "../services/decision-continuation-work.js"; +import { writeDecisionChange } from "../services/decision-work-notifications.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { hashAttentionArchiveManifest } from "../services/decision-retention.js"; +import type { AttentionArchiveManifestEntry } from "@paperclipai/shared"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_DECISION_EFFECTS_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("decision effect recovery worker", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let companyId: string, agentId: string, originId: string, targetId: string, runId: string, userId: string; + let coordinator: ReturnType | undefined; + let draining: boolean; + let recovery: ReturnType; + const errors: unknown[] = []; + const wake = vi.fn(async () => {}); + const service = () => decisionService(db, { wakeOriginAgent: wake }); + + beforeAll(async () => { + if (externalUrl) await applyPendingMigrations(externalUrl); + else temporary = await startEmbeddedPostgresTestDatabase("decision-effects-"); + db = createDb(externalUrl ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 30_000); + beforeEach(async () => { + vi.stubEnv("PAPERCLIP_DECISION_SIGNING_SECRET", "0123456789abcdef0123456789abcdef"); + companyId = randomUUID(); agentId = randomUUID(); originId = randomUUID(); targetId = randomUUID(); runId = randomUUID(); userId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Recovery", issuePrefix: `R${companyId.slice(0, 6)}`, requireBoardApprovalForNewAgents: false }); + await db.insert(authUsers).values({ id: userId, name: "Decider", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "member" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Proposer", role: "engineer", status: "active", adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} }); + await db.insert(issues).values([ + { id: originId, companyId, title: "Origin", status: "in_progress", priority: "medium", assigneeAgentId: agentId, responsibleUserId: userId }, + { id: targetId, companyId, title: "Target", status: "todo", priority: "medium", responsibleUserId: userId }, + ]); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", responsibleUserId: userId, contextSnapshot: { issueId: originId } }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + draining = false; errors.length = 0; wake.mockReset(); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs(); + await db.execute(sql`drop trigger if exists decision_effect_finish_fail on decisions`); + await db.execute(sql`drop function if exists decision_effect_finish_fail()`); + await db.delete(decisionEffectExecutions); await db.delete(decisionTargetIssues); await db.delete(decisions); + await db.delete(decisionRetention); await db.delete(activityLog); await db.delete(issueComments); await db.delete(heartbeatRuns); + await db.delete(issues); await db.delete(agents); await db.delete(companyMemberships); await db.delete(authUsers); await db.delete(companies); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + const actor = () => ({ type: "agent" as const, companyId, agentId, runId, source: "agent_jwt" as const, + onBehalfOfUserId: userId, onBehalfOfMemberships: [{ companyId, membershipRole: "member", status: "active" }] }); + const board = () => ({ type: "board" as const, userId, companyIds: [companyId], source: "session" as const, + memberships: [{ companyId, membershipRole: "member", status: "active" }] }); + const create = () => service().create({ companyId, agentId, runId, actor: actor(), title: "Comment?", body: "Body", + continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [ + { type: "comment_on_issue", targetIssueId: targetId, staleness: "lenient", bodyMarkdown: "hello" }, + ] }] }); + const decide = (id: string) => service().decide({ id, optionId: "yes", decidedByUserId: userId, userActor: board() }); + const claim = (id: string, updatedAt = new Date(), owner = db) => writeDecisionChange(owner, async tx => tx.update(decisions).set({ + status: "decided", executionStatus: "running", chosenOptionId: "yes", decidedByUserId: userId, + updatedAt, inputValues: {}, metadata: { continuationPending: true }, + }).where(eq(decisions.id, id))); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, + onError: error => errors.push(error) }); + await registerDecisionContinuationWork(coordinator, service(), () => !draining).ready; + recovery = service(); + vi.spyOn(recovery, "recoverEffects"); + await registerDecisionEffectRecoveryWork(coordinator, recovery, () => !draining).ready; + } + async function quiet() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"), executes = vi.spyOn(db, "execute"); + await vi.advanceTimersByTimeAsync(60 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); executes.mockRestore(); + } + async function settled(id: string, executionStatus = "succeeded") { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(await service().get(id)).toMatchObject({ executionStatus, metadata: { continuationPending: false } }); + }, { timeout: 5000 }); + } + const comments = () => db.select().from(issueComments).where(eq(issueComments.issueId, targetId)); + async function failSettlement() { + await db.execute(sql`create function decision_effect_finish_fail() returns trigger language plpgsql as $$ begin + if OLD.execution_status = 'running' and NEW.execution_status <> 'running' then raise exception 'settlement unavailable'; end if; + return NEW; end $$`); + await db.execute(sql`create trigger decision_effect_finish_fail before update on decisions for each row execute function decision_effect_finish_fail()`); + } + + it("stays quiet with open decisions and delivered history after a real decision", async () => { + await start(); await quiet(); const row = await create(); await quiet(); + await decide(row.id); await quiet(); expect(await comments()).toHaveLength(1); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("does not inspect effect recovery when an open decision is created", async () => { + await start(); await quiet(); + const sweeps = vi.mocked(recovery.recoverEffects); sweeps.mockClear(); + await create(); await quiet(); + expect(sweeps).not.toHaveBeenCalled(); + }); + + it("coalesces sequential committed cancellations without an idle recovery poll", async () => { + const rows = []; + for (let i = 0; i < 10; i++) rows.push(await create()); + // Freeze wall time while real SQL and transaction settlement continue. + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + await start(); + const scans = vi.mocked(recovery.recoverEffects); + expect(scans).toHaveBeenCalledTimes(1); + for (const row of rows) { + await service().cancel(row.id, { actorType: "user", actorId: userId }); + await vi.advanceTimersByTimeAsync(1); + await db.execute(sql`select pg_sleep(0.01)`); // Allow each old scan to finish, rather than only testing overlap coalescing. + } + expect(scans).toHaveBeenCalledTimes(1); + await vi.advanceTimersByTimeAsync(1000); + await vi.waitFor(() => expect(scans).toHaveBeenCalledTimes(2)); + await quiet(); + const count = scans.mock.calls.length; + await vi.advanceTimersByTimeAsync(3_600_000); + expect(scans).toHaveBeenCalledTimes(count); + expect(wake).toHaveBeenCalledTimes(10); + }); + + it("lets an already saved recovery deadline preempt notification coalescing", async () => { + const due = await create(), cancelled = await create(); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: false }); + const deadline = Date.now() + 500; + await claim(due.id, new Date(deadline - 60_000)); + await start(); + expect(coordinator!.nextWakeAt()).toBe(deadline); + await service().cancel(cancelled.id, { actorType: "user", actorId: userId }); + await vi.advanceTimersByTimeAsync(1); + await db.execute(sql`select pg_sleep(0.01)`); + expect(coordinator!.nextWakeAt()).toBe(deadline); + await vi.advanceTimersByTimeAsync(499); + await vi.waitFor(async () => expect(await comments()).toHaveLength(1)); + await settled(due.id); await quiet(); + expect(wake).toHaveBeenCalledTimes(2); + }); + + it("retries an old failure while newer due claims keep filling pages", async () => { + const failed = await create(), healthy = await create(); + const at = Date.now() - 600_000; + await claim(failed.id, new Date(at)); await claim(healthy.id, new Date(at + 1)); + // Preserve PostgreSQL precision at both ends of the first rotation. + await db.execute(sql`update decisions set updated_at = updated_at + interval '456 microseconds' + where id in (${failed.id}, ${healthy.id})`); + const failure = new Error("temporary recovery failure"); + const attempts: string[] = []; + const scan = decisionEffectRecoveryService(db, async id => { + attempts.push(id); + if (id === failed.id) throw failure; + await db.update(decisions).set({ executionStatus: "succeeded" }).where(eq(decisions.id, id)); + return true; + }); + await expect(scan.sweep(new Date(), 1)).rejects.toBe(failure); + const added: string[] = []; + for (let index = 0; index < 7; index++) { + const row = await create(); added.push(row.id); await claim(row.id, new Date(at + index + 2)); + try { await scan.sweep(new Date(), 1); } catch (error) { expect(error).toBe(failure); } + } + expect(attempts).toHaveLength(8); + expect(attempts.slice(0, 6)).toEqual([failed.id, healthy.id, failed.id, added[0], added[1], failed.id]); + expect(new Set(attempts.filter(id => id !== failed.id)).size).toBe(5); + expect((await service().get(failed.id))?.executionStatus).toBe("running"); + }); + + it("reactivates from the actual claim and recovers a failed settlement without repeating the effect", async () => { + await start(); await quiet(); const row = await create(); await failSettlement(); + await expect(decide(row.id)).rejects.toMatchObject({ cause: { message: "settlement unavailable" } }); + expect((await service().get(row.id))?.executionStatus).toBe("running"); expect(await comments()).toHaveLength(1); + await db.execute(sql`drop trigger decision_effect_finish_fail on decisions`); + await settled(row.id); await quiet(); expect(await comments()).toHaveLength(1); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("reconstructs the saved grace deadline at restart and recovers a claimed effect", async () => { + const row = await create(); const claimedAt = new Date(); await claim(row.id, claimedAt); + await db.insert(decisionEffectExecutions).values({ decisionId: row.id, effectIndex: 0, effectType: "comment_on_issue", targetIssueId: targetId, status: "claimed" }); + await start(); expect(coordinator!.nextWakeAt()).toBe(claimedAt.getTime() + 60_000); + await vi.advanceTimersByTimeAsync(30_000); expect(await comments()).toHaveLength(0); + await settled(row.id); await quiet(); expect(await comments()).toHaveLength(1); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("recovers at the exact grace boundary and preserves a zero grace override", async () => { + const row = await create(), at = new Date(); await claim(row.id, at); + expect(await service().recoverEffects(new Date(at.getTime() + 59_999))).toEqual({ resumed: 0 }); + expect(await service().recoverEffects(new Date(at.getTime() + 60_000))).toEqual({ resumed: 1 }); + vi.stubEnv("PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS", "0"); + const second = await create(); await claim(second.id, at); + expect(await service().recoverEffects(at)).toEqual({ resumed: 1 }); expect(await comments()).toHaveLength(2); + }); + + it("joins active foreground effects when their recovery deadline arrives", async () => { + const row = await create(); await start(); + const effectKey = `decision-effect:${row.id}:0`; + const previousDebug = db.$client.options.debug; + let effectLockAttempts = 0; + db.$client.options.debug = (_connection, query, parameters) => { + if (query.includes("pg_advisory_xact_lock") && parameters.includes(effectKey)) effectLockAttempts++; + }; + let release!: () => void, locked!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const acquired = new Promise(resolve => { locked = resolve; }); + const lock = db.transaction(async tx => { + await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${effectKey}, 0))`); + locked(); await gate; + }); + let deciding: ReturnType | undefined; + let recovering: ReturnType | undefined; + try { + await acquired; + deciding = decide(row.id); + await vi.waitFor(async () => expect((await service().get(row.id))?.executionStatus).toBe("running")); + const claimed = (await service().get(row.id))!; + const deadline = claimed.updatedAt.getTime() + 60_000; + await vi.waitFor(async () => { + const scheduled = coordinator!.nextWakeAt(); + if (scheduled !== null && scheduled < deadline) { + await vi.advanceTimersByTimeAsync(Math.max(1, scheduled - Date.now())); + } + expect(coordinator!.nextWakeAt()).toBe(deadline); + }); + await vi.waitFor(() => expect(effectLockAttempts).toBe(2)); // Test gate + foreground execution. + const sweeps = vi.mocked(recovery.recoverEffects); + const beforeDeadline = sweeps.mock.calls.length; + await vi.advanceTimersByTimeAsync(Math.max(0, deadline - Date.now())); + await vi.waitFor(() => expect(sweeps).toHaveBeenCalledTimes(beforeDeadline + 1)); + let recoverySettled = false; + recovering = Promise.resolve(sweeps.mock.results[beforeDeadline].value).then(result => { + recoverySettled = true; + return result; + }); + await vi.advanceTimersByTimeAsync(120_000); + expect(await comments()).toHaveLength(0); expect(wake).not.toHaveBeenCalled(); + expect(recoverySettled).toBe(false); + expect(sweeps).toHaveBeenCalledTimes(beforeDeadline + 1); + expect(effectLockAttempts).toBe(2); + release(); + await lock; await deciding; + await expect(recovering).resolves.toEqual({ resumed: 1 }); + await quiet(); + expect(effectLockAttempts).toBe(2); + expect(await comments()).toHaveLength(1); expect(wake).toHaveBeenCalledTimes(1); + } finally { + release(); + await Promise.allSettled([lock, deciding, recovering]); + db.$client.options.debug = previousDebug; + } + }); + + it("defers an outer transaction until commit and ignores rollback", async () => { + const row = await create(); await start(); await quiet(); + await expect(db.transaction(async tx => { + await claim(row.id, new Date(Date.now() - 120_000), tx as unknown as typeof db); + await vi.advanceTimersByTimeAsync(120_000); expect(await comments()).toHaveLength(0); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect((await service().get(row.id))?.status).toBe("open"); + await db.transaction(async tx => { + await claim(row.id, new Date(Date.now() - 120_000), tx as unknown as typeof db); + await vi.advanceTimersByTimeAsync(120_000); expect(await comments()).toHaveLength(0); + }); + await settled(row.id); await quiet(); expect(await comments()).toHaveLength(1); + }); + + it("retains a first claim through drain without passive SQL", async () => { + const row = await create(); await start(); await quiet(); draining = true; + await claim(row.id, new Date(Date.now() - 120_000)); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; await settled(row.id); await quiet(); expect(await comments()).toHaveLength(1); + }); + + it("retains a failed recovery through drain and does not replay the completed effect", async () => { + const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); await failSettlement(); await start(); + await vi.waitFor(() => expect(errors.length).toBeGreaterThan(0)); draining = true; + expect(await comments()).toHaveLength(1); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await db.execute(sql`drop trigger decision_effect_finish_fail on decisions`); + draining = false; await settled(row.id); await quiet(); expect(await comments()).toHaveLength(1); expect(wake).toHaveBeenCalledTimes(1); + for (const error of errors) expect(error).toMatchObject({ cause: { message: "settlement unavailable" } }); errors.length = 0; + }); + + it("rechecks current membership before executing a recovered effect", async () => { + const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); + await db.update(companyMemberships).set({ status: "inactive" }).where(eq(companyMemberships.principalId, userId)); + await start(); await settled(row.id, "failed"); await quiet(); expect(await comments()).toHaveLength(0); + const result = await service().outcome(row.id); + expect(result.executions[0]?.error).toBe("deny_decision_intersection"); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("leaves continuation failures to their owner after effect recovery finishes", async () => { + const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); + wake.mockRejectedValueOnce(new Error("wake unavailable")); await start(); await settled(row.id); await quiet(); + expect(await comments()).toHaveLength(1); expect(wake).toHaveBeenCalledTimes(2); + expect(errors.every(error => String(error).includes("wake unavailable"))).toBe(true); errors.length = 0; + }); + + it("keeps aggregate maintenance from recovering effects when the worker owns them", async () => { + const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); + expect(await service().sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false })) + .toEqual({ expired: 0, resumed: 0 }); + expect(await comments()).toHaveLength(0); + expect(await service().sweepExpired()).toEqual({ expired: 0, resumed: 1 }); expect(await comments()).toHaveLength(1); + }); + + it.each(["missing_decider", "missing_option"] as const)("settles %s as durable failure and then becomes quiet", async reason => { + const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); + await db.update(decisions).set(reason === "missing_decider" ? { decidedByUserId: null } : { chosenOptionId: "missing" }) + .where(eq(decisions.id, row.id)); + expect(await service().recoverEffects()).toEqual({ resumed: 1 }); + expect(await service().get(row.id)).toMatchObject({ executionStatus: "failed", metadata: { + effectRecoveryError: reason, continuationPending: false, + } }); + expect(await comments()).toHaveLength(0); + expect(await db.select().from(decisionEffectExecutions)).toHaveLength(0); + const audit = await db.select().from(activityLog).where(eq(activityLog.action, "decision.recovery_failed")); + expect(audit).toHaveLength(1); + expect(audit[0]).toMatchObject({ companyId, actorType: "system", actorId: "decision-executor", + entityId: row.id, details: { reason } }); + expect(wake).toHaveBeenCalledTimes(1); + await start(); await quiet(); + expect(wake).toHaveBeenCalledTimes(1); + }); + + it("preserves completed effect receipts when a malformed outcome settles as partial", async () => { + const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); + await db.update(decisions).set({ chosenOptionId: "missing" }).where(eq(decisions.id, row.id)); + const [receipt] = await db.insert(decisionEffectExecutions).values({ decisionId: row.id, effectIndex: 0, + effectType: "comment_on_issue", targetIssueId: targetId, status: "executed", result: { preserved: true }, executedAt: new Date() }).returning(); + expect(await service().recoverEffects()).toEqual({ resumed: 1 }); + expect((await service().get(row.id))?.executionStatus).toBe("partial"); + expect(await db.select().from(decisionEffectExecutions)).toEqual([receipt]); + expect(await comments()).toHaveLength(0); + expect(wake).toHaveBeenCalledTimes(1); + await start(); await quiet(); + }); + + it("keeps malformed claims pending when their failure audit cannot commit", async () => { + const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); + await db.update(decisions).set({ decidedByUserId: null }).where(eq(decisions.id, row.id)); + await db.execute(sql`create function decision_recovery_audit_fail() returns trigger language plpgsql as $$ begin + if NEW.action = 'decision.recovery_failed' then raise exception 'recovery audit unavailable'; end if; + return NEW; end $$`); + await db.execute(sql`create trigger decision_recovery_audit_fail before insert on activity_log + for each row execute function decision_recovery_audit_fail()`); + try { + await expect(service().recoverEffects()).rejects.toMatchObject({ cause: { message: "recovery audit unavailable" } }); + expect((await service().get(row.id))?.executionStatus).toBe("running"); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision.recovery_failed"))).toHaveLength(0); + expect(wake).not.toHaveBeenCalled(); + } finally { + await db.execute(sql`drop trigger decision_recovery_audit_fail on activity_log`); + await db.execute(sql`drop function decision_recovery_audit_fail()`); + } + expect(await service().recoverEffects()).toEqual({ resumed: 1 }); + expect((await service().get(row.id))?.executionStatus).toBe("failed"); + expect(wake).toHaveBeenCalledTimes(1); + await start(); await quiet(); + }); + + it("rechecks a malformed claim under lock before settling a concurrently repaired outcome", async () => { + const row = await create(); const at = new Date(Date.now() - 120_000); await claim(row.id, at); + await db.update(decisions).set({ chosenOptionId: "missing" }).where(eq(decisions.id, row.id)); + let release!: () => void, locked!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const acquired = new Promise(resolve => { locked = resolve; }); + const repair = db.transaction(async tx => { + await tx.select({ id: decisions.id }).from(decisions).where(eq(decisions.id, row.id)).for("update"); + locked(); await gate; + await tx.update(decisions).set({ chosenOptionId: "yes", updatedAt: at }).where(eq(decisions.id, row.id)); + }); + const previousDebug = db.$client.options.debug; + let attempts = 0; + let pending: ReturnType["recoverEffects"]> | undefined; + try { + await acquired; + db.$client.options.debug = (_connection, query, parameters) => { + if (query.includes("for update") && parameters.includes(row.id)) attempts++; + }; + pending = service().recoverEffects(); + await vi.waitFor(() => expect(attempts).toBe(1)); + release(); await repair; + expect(await pending).toEqual({ resumed: 0 }); + expect(await service().get(row.id)).toMatchObject({ executionStatus: "running", chosenOptionId: "yes" }); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision.recovery_failed"))).toHaveLength(0); + expect(wake).not.toHaveBeenCalled(); + expect(await service().recoverEffects()).toEqual({ resumed: 1 }); + expect(await comments()).toHaveLength(1); expect(wake).toHaveBeenCalledTimes(1); + } finally { + release(); await Promise.allSettled([repair, pending]); db.$client.options.debug = previousDebug; + } + await start(); await quiet(); + }); + + it("settles invalid legacy claims without inventing authority or blocking later effects", async () => { + vi.stubEnv("PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE", "1"); + const invalid = await create(), valid = await create(); + await claim(invalid.id, new Date(Date.now() - 180_000)); await claim(valid.id, new Date(Date.now() - 120_000)); + await db.update(decisions).set({ decidedByUserId: null }).where(eq(decisions.id, invalid.id)); + const recovery = service(); + expect(await recovery.recoverEffects()).toEqual({ resumed: 1 }); + expect(await comments()).toHaveLength(0); + expect(await recovery.recoverEffects()).toEqual({ resumed: 1 }); expect(await comments()).toHaveLength(1); + expect((await recovery.get(invalid.id))?.executionStatus).toBe("failed"); + expect(await recovery.nextEffectRecoveryAt()).toBeNull(); + }); + + it("pages past unresolved claims using database timestamp precision", async () => { + const pending = await create(), valid = await create(); + await claim(pending.id); await claim(valid.id, new Date(Date.now() - 120_000)); + await db.execute(sql`update decisions set updated_at = date_trunc('milliseconds', now() - interval '3 minutes') + + interval '456 microseconds' where id = ${pending.id}`); + const scan = decisionEffectRecoveryService(db, async id => { + if (id === pending.id) return false; + await decide(id); return true; + }); + expect(await scan.sweep(new Date(), 1)).toEqual({ resumed: 0 }); + expect(await scan.sweep(new Date(), 1)).toEqual({ resumed: 1 }); + expect(await comments()).toHaveLength(1); + expect((await service().get(pending.id))?.executionStatus).toBe("running"); + expect(await scan.next()).not.toBeNull(); + expect(await scan.sweep(new Date(), 1)).toEqual({ resumed: 0 }); + expect(await scan.sweep(new Date(), 1)).toEqual({ resumed: 0 }); + expect(await comments()).toHaveLength(1); + }); + + it.each([false, true])("preserves archive recovery semantics with prior committed archive: %s", async committed => { + await db.update(issues).set({ status: "in_review", assigneeUserId: userId }).where(eq(issues.id, targetId)); + const activityAt = new Date("2026-04-01T00:00:00.000Z"); + await db.insert(decisionRetention).values({ companyId, sourceKind: "review", sourceId: targetId, sourceActivityAt: activityAt }); + const manifest: AttentionArchiveManifestEntry[] = [{ companyId, sourceKind: "review", sourceId: targetId, + expectedVersion: 1, activityAt: activityAt.toISOString(), reason: "Archive" }]; + const row = await service().create({ companyId, actor: actor(), agentId, runId, title: "Archive?", body: "Exact set", + continuationPolicy: "wake_origin_agent", options: [{ id: "archive", label: "Archive", effects: [] }], + metadata: { kind: "attention_archive_proposal", manifestHash: hashAttentionArchiveManifest(manifest) }, + additionalTargetSnapshots: { [`attention:review:${targetId}`]: { status: "attention", assigneeAgentId: null, + assigneeUserId: null, updatedAt: activityAt.toISOString(), attentionArchive: manifest[0] } } }); + if (committed) { + await failSettlement(); + await expect(service().decide({ id: row.id, optionId: "archive", decidedByUserId: userId, userActor: board() })) + .rejects.toMatchObject({ cause: { message: "settlement unavailable" } }); + await db.execute(sql`drop trigger decision_effect_finish_fail on decisions`); + } else { + await writeDecisionChange(db, async tx => tx.update(decisions).set({ status: "decided", executionStatus: "running", + chosenOptionId: "archive", decidedByUserId: userId, updatedAt: new Date(Date.now() - 120_000), + metadata: { ...row.metadata, continuationPending: true } }).where(eq(decisions.id, row.id))); + } + await start(); await settled(row.id, committed ? "failed" : "succeeded"); await quiet(); + const [retained] = await db.select().from(decisionRetention); + expect(retained.archiveVersion).toBe(1); expect(retained.archivedAt).not.toBeNull(); + expect(wake).toHaveBeenCalledTimes(1); + if (committed) expect((await service().get(row.id))?.metadata).toHaveProperty("archiveProposalError"); + }); + + it("learns another claim written during delivery", async () => { + const first = await create(), second = await create(); await claim(first.id, new Date(Date.now() - 120_000)); + wake.mockImplementationOnce(async () => { await claim(second.id); }); + await start(); await settled(first.id); await settled(second.id); await quiet(); + expect(await comments()).toHaveLength(2); expect(wake).toHaveBeenCalledTimes(2); + }); + + it("finishes the admitted recovery and leaves later claims for restart during shutdown", async () => { + const first = await create(), second = await create(); + await claim(first.id, new Date(Date.now() - 180_000)); await claim(second.id, new Date(Date.now() - 120_000)); + let release!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + wake.mockImplementationOnce(async () => gate); + const starting = start(); let stopped = false; let stopping: Promise; + try { + await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(1)); + stopping = coordinator!.stop().then(() => { stopped = true; }); + await vi.advanceTimersByTimeAsync(60_000); expect(stopped).toBe(false); + } finally { release(); } + await starting; await stopping!; coordinator = undefined; + expect(wake).toHaveBeenCalledTimes(1); expect(await comments()).toHaveLength(1); + expect((await service().get(second.id))?.executionStatus).toBe("running"); + await start(); await quiet(); expect(await comments()).toHaveLength(2); expect(wake).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index c2ee1e0caa4..3c814c6dcfe 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -349,6 +349,8 @@ vi.mock("../services/index.js", () => ({ })), decisionService: vi.fn(() => ({ sweepExpired: vi.fn(async () => ({ expired: 0 })), + recoverEffects: vi.fn(async () => ({ resumed: 0 })), + nextEffectRecoveryAt: vi.fn(async () => null), sweepTtlExpiry: vi.fn(async () => ({ expired: 0 })), nextExpiryAt: vi.fn(async () => null), sweepContinuations: vi.fn(async () => ({ retry: false })), @@ -921,8 +923,10 @@ describe("startServer feedback export wiring", () => { const decisions = vi.mocked(decisionService).mock.results.at(-1)!.value; expect(decisions.sweepContinuations).toHaveBeenCalledTimes(1); expect(decisions.sweepTtlExpiry).toHaveBeenCalledTimes(1); + expect(decisions.recoverEffects).toHaveBeenCalledTimes(1); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionEffectRecovery)).toBe(true); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(true); - expect(decisions.sweepExpired).toHaveBeenCalledWith(expect.any(Date), { includeContinuations: false, includeExpiry: false }); + expect(decisions.sweepExpired).toHaveBeenCalledWith(expect.any(Date), { includeContinuations: false, includeExpiry: false, includeEffects: false }); } finally { setIntervalSpy.mockRestore(); } @@ -961,6 +965,7 @@ describe("startServer feedback export wiring", () => { expect(environmentCustomImagesServiceMock.cleanupExpiredSetupSessions).not.toHaveBeenCalled(); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionContinuation)).toBe(false); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(false); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionEffectRecovery)).toBe(false); } finally { setIntervalSpy.mockRestore(); } diff --git a/server/src/index.ts b/server/src/index.ts index 883590a67a4..0d27d6f5b5e 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerDecisionEffectRecoveryWork } from "./services/decision-effect-recovery-work.js"; import { registerDecisionExpiryWork } from "./services/decision-expiry-work.js"; import { registerDecisionContinuationWork } from "./services/decision-continuation-work.js"; @@ -1842,7 +1843,9 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("decision_expiry_worker", () => registerDecisionExpiryWork(deliveryWork, decisionExecutor, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); - await step("decision_expiry", () => decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false })); + await step("decision_effect_recovery_worker", () => registerDecisionEffectRecoveryWork(deliveryWork, decisionExecutor, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("decision_expiry", () => decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false })); const loginSchedule = loginCleanupSchedule(db, config.heartbeatSchedulerIntervalMs); await step("login_cleanup_workers", () => registerLoginCleanupWork(deliveryWork, [ { queue: DELIVERY_QUEUES.adapterLoginCleanup, run: () => adapterLoginReaper.sweep().then(logAdapterLoginReaperResult), next: loginSchedule.adapter }, @@ -1863,7 +1866,7 @@ async function startServerWithDatabaseTeardown( trackHeartbeatSchedulerWork((async () => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; scheduleRoutineStatusReconcile(); - trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false }).catch((err: unknown) => { + trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false }).catch((err: unknown) => { logger.error({ err }, "decision expiry sweep failed"); })); trackHeartbeatSchedulerWork(runRetentionSweep().catch((err: unknown) => { diff --git a/server/src/services/decision-effect-recovery-work.ts b/server/src/services/decision-effect-recovery-work.ts new file mode 100644 index 00000000000..c34ef251a7c --- /dev/null +++ b/server/src/services/decision-effect-recovery-work.ts @@ -0,0 +1,17 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import type { decisionService } from "./decisions.js"; + +export function registerDecisionEffectRecoveryWork( + coordinator: ReturnType, + service: Pick, "recoverEffects" | "nextEffectRecoveryAt">, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.decisionEffectRecovery, { + retryMs: 30_000, canRun, + notificationCoalesceMs: 1000, + run: signal => service.recoverEffects(new Date(), undefined, signal), + next: service.nextEffectRecoveryAt, + }); +} diff --git a/server/src/services/decision-effect-recovery.ts b/server/src/services/decision-effect-recovery.ts new file mode 100644 index 00000000000..9d987cc0568 --- /dev/null +++ b/server/src/services/decision-effect-recovery.ts @@ -0,0 +1,60 @@ +import { and, asc, desc, lte, sql } from "drizzle-orm"; +import { decisions, type Db } from "@paperclipai/db"; + +const running = sql`${decisions.status} = 'decided' AND ${decisions.executionStatus} = 'running'`; + +export function decisionSweepBatchSize() { + const value = Number(process.env.PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE ?? 100); + return Number.isFinite(value) ? Math.max(1, Math.trunc(value)) : 100; +} + +function recoveryGraceMs() { + const value = Number(process.env.PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS ?? 60_000); + return Number.isFinite(value) && value >= 0 ? value : 60_000; +} + +export function decisionEffectRecoveryService(db: Db, resume: (id: string) => Promise) { + type Position = { updatedAt: string; id: string }; + let cursor: Position | undefined; + let boundary: Position | undefined; + + async function sweep(now = new Date(), limit = decisionSweepBatchSize(), signal?: AbortSignal) { + let resumed = 0; + if (signal?.aborted) return { resumed }; + const dueBefore = new Date(now.getTime() - recoveryGraceMs()); + const position = { id: decisions.id, updatedAt: sql`${decisions.updatedAt}::text` }; + if (!boundary) { + // Finish this rotation even if newer due claims arrive on every page. + [boundary] = await db.select(position).from(decisions) + .where(and(running, lte(decisions.updatedAt, dueBefore))) + .orderBy(desc(decisions.updatedAt), desc(decisions.id)).limit(1); + if (!boundary) return { resumed }; + } + const rows = await db.select(position).from(decisions) + .where(and(running, lte(decisions.updatedAt, dueBefore), + sql`(${decisions.updatedAt}, ${decisions.id}) <= (${boundary.updatedAt}::timestamptz, ${boundary.id}::uuid)`, cursor + ? sql`(${decisions.updatedAt}, ${decisions.id}) > (${cursor.updatedAt}::timestamptz, ${cursor.id}::uuid)` : undefined)) + .orderBy(asc(decisions.updatedAt), asc(decisions.id)).limit(limit); + const errors: unknown[] = []; + for (const row of rows) { + if (signal?.aborted) return { resumed }; + try { if (await resume(row.id)) resumed++; } catch (error) { errors.push(error); } + cursor = row; + } + const last = rows.at(-1); + if (rows.length < limit || (last?.id === boundary.id && last.updatedAt === boundary.updatedAt)) { + cursor = undefined; boundary = undefined; + } + if (errors.length === 1) throw errors[0]; + if (errors.length) throw new AggregateError(errors, "Decision effect recovery failed"); + return { resumed }; + } + + async function next(): Promise { + const [row] = await db.select({ at: decisions.updatedAt }).from(decisions).where(running) + .orderBy(asc(decisions.updatedAt), asc(decisions.id)).limit(1); + return row ? row.at.getTime() + recoveryGraceMs() : null; + } + + return { sweep, next }; +} diff --git a/server/src/services/decision-work-notifications.ts b/server/src/services/decision-work-notifications.ts index 06869ceff59..081edc8fc39 100644 --- a/server/src/services/decision-work-notifications.ts +++ b/server/src/services/decision-work-notifications.ts @@ -13,6 +13,7 @@ export async function notifyDecisionChange(transaction: object) { export function writeDecisionChange(db: Db, write: (transaction: Db) => Promise): Promise { return db.transaction(async transaction => { await notifyDecisionChange(transaction); + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionEffectRecovery); return write(transaction as unknown as Db); }); } diff --git a/server/src/services/decisions.ts b/server/src/services/decisions.ts index 751e4c30406..9d915c7955d 100644 --- a/server/src/services/decisions.ts +++ b/server/src/services/decisions.ts @@ -1,5 +1,5 @@ import { randomUUID } from "node:crypto"; -import { and, asc, count, desc, eq, gt, gte, inArray, lte, sql } from "drizzle-orm"; +import { and, asc, count, desc, eq, gt, gte, inArray, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { databaseWorkScopeKey, companyMemberships, decisionBundles, decisionEffectExecutions, decisionRetention, decisions, decisionTargetIssues, heartbeatRuns, issueRelations, issues } from "@paperclipai/db"; import { ATTENTION_SOURCE_KINDS, decisionEffectTargetIssueIds } from "@paperclipai/shared"; @@ -15,6 +15,7 @@ import { import { decisionRetentionService, hashAttentionArchiveManifest } from "./decision-retention.js"; import { decisionContinuationService, type DecisionContinuationWake } from "./decision-continuations.js"; +import { decisionEffectRecoveryService, decisionSweepBatchSize } from "./decision-effect-recovery.js"; import { decisionExpiryService } from "./decision-expiry.js"; import { notifyDecisionCreated, writeDecisionChange } from "./decision-work-notifications.js"; @@ -24,7 +25,8 @@ type Snapshot = { status: string; assigneeAgentId: string | null; assigneeUserId export type DecisionServiceOptions = { wakeOriginAgent: DecisionContinuationWake }; const DAY = 86_400_000; type DecisionOutcome = typeof decisions.$inferSelect & { executions: Array }; -const effectFlights = new WeakMap>>(); +type EffectAttempt = { outcome: DecisionOutcome; resumed: boolean }; +const effectFlights = new WeakMap>>(); function targetIds(options: DecisionOption[]) { const result = new Set(); @@ -126,6 +128,7 @@ function boardCanActDirectly(actor: AuthorizationActor, companyId: string) { export function decisionService(db: Db, options: DecisionServiceOptions) { const continuations = decisionContinuationService(db, options.wakeOriginAgent); const expiry = decisionExpiryService(db, continuations.deliver); + const effectRecovery = decisionEffectRecoveryService(db, recoverDecisionEffects); const scope = databaseWorkScopeKey(db); let flights = effectFlights.get(scope); if (!flights) effectFlights.set(scope, flights = new Map()); @@ -557,21 +560,53 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { } } - function runEffects(decision: typeof decisions.$inferSelect, userActor: AuthorizationActor): Promise { - const existing = executing.get(decision.id); + function runEffects(id: string, userActor?: AuthorizationActor): Promise { + const existing = executing.get(id); if (existing) return existing; const attempt = (async () => { - const current = await get(decision.id); - if (!current) throw notFound("Decision not found"); - return current.executionStatus === "running" ? executeDecisionEffects(current, userActor) : outcome(current.id); - })().finally(() => { executing.delete(decision.id); }); - executing.set(decision.id, attempt); + const current = await get(id); + if (!current) return null; + if (current.status !== "decided" || current.executionStatus !== "running") { + return { outcome: await outcome(id), resumed: false }; + } + // Recovery derives authority from the current claim inside the shared flight. + const actor = userActor ?? (current.decidedByUserId + ? await recoveryActor(current.companyId, current.decidedByUserId) + : { type: "none" as const, source: "none" as const }); + return { outcome: await executeDecisionEffects(current, actor), resumed: true }; + })().finally(() => { executing.delete(id); }); + executing.set(id, attempt); return attempt; } + async function settleInvalidEffectOutcome(id: string) { + const publications: ActivityPublication[] = []; + await writeDecisionChange(db, async tx => { + const [current] = await tx.select().from(decisions).where(eq(decisions.id, id)).for("update"); + if (!current || current.status !== "decided" || current.executionStatus !== "running") return; + const reason = !current.decidedByUserId ? "missing_decider" + : !current.options.some(option => option.id === current.chosenOptionId) ? "missing_option" : null; + if (!reason) return; + const [executed] = await tx.select({ id: decisionEffectExecutions.id }).from(decisionEffectExecutions) + .where(and(eq(decisionEffectExecutions.decisionId, id), eq(decisionEffectExecutions.status, "executed"))).limit(1); + const executionStatus = executed ? "partial" : "failed"; + await tx.update(decisions).set({ executionStatus, updatedAt: new Date(), metadata: { ...current.metadata, + effectRecoveryError: reason, + ...(current.continuationPolicy === "wake_origin_agent" ? { continuationPending: true } : {}), + } }).where(eq(decisions.id, id)); + await logActivity(tx, { companyId: current.companyId, actorType: "system", actorId: "decision-executor", + agentId: current.originAgentId, runId: current.originRunId, responsibleUserIdOverride: current.decidedByUserId, + action: "decision.recovery_failed", entityType: "decision", entityId: id, + details: { reason, executionStatus, chosenOptionId: current.chosenOptionId, decidedByUserId: current.decidedByUserId }, + }, publications); + }); + for (const publication of publications) publishActivity(publication); + return outcome(id); + } + async function executeDecisionEffects(decision: typeof decisions.$inferSelect, userActor: AuthorizationActor) { const option = decision.options.find((item) => item.id === decision.chosenOptionId); - if (!option || !decision.decidedByUserId) throw unprocessable("Stored decision outcome is invalid"); + if (!option || !decision.decidedByUserId) return settleInvalidEffectOutcome(decision.id); const run = await db.select({ responsibleUserId: heartbeatRuns.responsibleUserId }).from(heartbeatRuns).where(eq(heartbeatRuns.id, decision.originRunId)).then((rows) => rows[0] ?? null); const metadata = decision.metadata as Record; if (metadata.kind === "attention_archive_proposal") { @@ -635,7 +670,9 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { } async function resumeDecision(decision: typeof decisions.$inferSelect, userActor: AuthorizationActor) { - const result = decision.executionStatus === "running" ? await runEffects(decision, userActor) : await outcome(decision.id); + const attempt = decision.executionStatus === "running" ? await runEffects(decision.id, userActor) : undefined; + if (attempt === null) throw notFound("Decision not found"); + const result = attempt?.outcome ?? await outcome(decision.id); await continuations.deliver(result.id); return outcome(decision.id); } @@ -716,27 +753,16 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { return { ...bundle, decisions: created }; }); } - async function sweepExpired(now = new Date(), options: { includeContinuations?: boolean; includeExpiry?: boolean } = {}) { - const configuredBatchSize = Number(process.env.PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE ?? 100); - const batchSize = Number.isFinite(configuredBatchSize) - ? Math.max(1, Math.trunc(configuredBatchSize)) - : 100; - const configuredRecoveryGraceMs = Number(process.env.PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS ?? 60_000); - const recoveryGraceMs = Number.isFinite(configuredRecoveryGraceMs) && configuredRecoveryGraceMs >= 0 - ? configuredRecoveryGraceMs - : 60_000; - const runningRows = await db.select().from(decisions) - .where(and(eq(decisions.status, "decided"), eq(decisions.executionStatus, "running"), - lte(decisions.updatedAt, new Date(now.getTime() - recoveryGraceMs)))) - .orderBy(asc(decisions.updatedAt)).limit(batchSize); - let resumed = 0; - for (const decision of runningRows) { - if (!decision.decidedByUserId) continue; - const recovered = await runEffects(decision, await recoveryActor(decision.companyId, decision.decidedByUserId)); - if (recovered.executionStatus === "running") continue; - resumed += 1; - await continuations.deliver(recovered.id); - } + async function recoverDecisionEffects(id: string) { + const attempt = await runEffects(id); + if (!attempt?.resumed || attempt.outcome.executionStatus === "running") return false; + await continuations.deliver(attempt.outcome.id); + return true; + } + + async function sweepExpired(now = new Date(), options: { includeContinuations?: boolean; includeExpiry?: boolean; includeEffects?: boolean } = {}) { + const batchSize = decisionSweepBatchSize(); + const { resumed } = options.includeEffects === false ? { resumed: 0 } : await effectRecovery.sweep(now, batchSize); if (options.includeContinuations !== false) await continuations.sweep(batchSize); const ttlRows = options.includeExpiry === false ? [] : await expiry.due(now, batchSize); const remaining = batchSize - ttlRows.length; @@ -768,5 +794,6 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { } return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepExpired, sweepContinuations: continuations.sweep, - sweepTtlExpiry: expiry.sweep, nextExpiryAt: expiry.next }; + sweepTtlExpiry: expiry.sweep, nextExpiryAt: expiry.next, + recoverEffects: effectRecovery.sweep, nextEffectRecoveryAt: effectRecovery.next }; } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 46e8242508c..908be12ff8d 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + decisionEffectRecovery: "decision-effect-recovery", decisionExpiry: "decision-expiry", decisionContinuation: "decision-continuation", From 8dfed893d8d3a7d7f09f5beb6a80a07da2ae3f89 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 18:53:50 -0500 Subject: [PATCH 30/41] fix(server): wake decision target checks from committed changes (#17486) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Decisions can depend on strict references to other tasks. > - A cancelled, deleted, or completed target can make a decision obsolete. > - The server currently scans open decisions on every maintenance tick to find those changes. > - The owning issue writes already have transaction boundaries that can signal the change. > - This pull request reconciles affected decisions after commit and reconstructs open targets at startup. > - Valid open decisions and completed history stop causing periodic SQL. ## Linked Issues or Issue Description **What existing behavior does this improve?** Decision target invalidation. This is slice 12 of the recurring-server sleep stack, based on #17482. TTL expiry is #17479 and continuation delivery is #17478. Retention remains separate work. **Subsystem affected** Server decision policy, issue mutation and deletion, and startup wiring. **Current behavior** The aggregate sweep repeatedly reads open decisions and their strict targets. It shares a maintenance tick with unrelated jobs even when every target remains valid. **Proposed behavior** Committed decision creation and terminal issue updates request a scoped check. Deletion requests a bounded full scan because its target links can cascade away. Startup reconstructs open target state. Once reconciliation settles, no target scan remains scheduled. **Reason and benefit** Remove normal empty polling without losing target invalidation, transactional delivery, or restart recovery. Reuse existing decision rows and target links instead of adding a table. **Breaking changes** No API or schema change. The main tick stops calling the combined decision sweep after its four duties have separate owners. The public combined method retains its default behavior. **Additional context** The initial decision implementation stored secondary links as well as primary links. This migration does not assume a new legacy backfill. Independent SQL writers still require an explicit signal integration. ## What Changed - Replace the periodic target scan with committed-change activation and bounded startup reconstruction. Keep the existing decision policy and expiry/activity owner. - Require production notifications to identify an issue or decision. Before deletion, lock the issue and capture its linked open decisions in the same company; emit their IDs before links cascade. This avoids resetting full scans and fences concurrent proposal creation. - Reuse that owner helper for issue removal and routine failure cleanup. No linked decisions means no target signal. Remove company deletion's ineffective global signal; its pre-existing restrictive decision foreign keys remain unchanged. - Retain PostgreSQL timestamp text in paging cursors so rows within one millisecond cannot repeat indefinitely. - Remove the unused combined internal decision sweep after its final duty is split; exercise each owner directly in the existing behavioral tests. - Keep query policy and notification discovery in `decision-targets.ts`. The coordinator registration contains lifecycle wiring only. Add no queue table, per-route recovery wrapper or external-wake exemption. ## Verification Current candidate normally integrates landed effect recovery `e9e84bb8a1`. Its runtime and acceptance tests are byte-identical to tested target head `85ce6545bf`; the only new source files are four independently merged Runner E2E accounting files, preserved exactly. The 248-case/type/build evidence comes from test revision `54936a919f36c2ea421e7ce8e15912b1105b8dcc`, followed by documentation-only changes. Fresh current-head CI and reviews are required. This remains the target-invalidation duty only. - **248 cases pass** in all seven complete suites: 25 target, 80 routine, 54 decision, 26 effect-recovery, 20 expiry, 42 startup and one scoped-deadline case. Database suites use separate migrated PostgreSQL fixtures. Server production typecheck and build pass. - **Four regressions failed before repair:** paging across microsecond timestamps, deletion behind a full valid startup page, unnecessary global work from unrelated deletion, and unnecessary work after deleting an empty company. All four pass. - A real concurrent proposal is observed waiting on the issue-row lock between pre-delete discovery and deletion commit. Removing that lock made the test fail because the proposal committed instead of waiting. Restoring the implementation passes. The mutation was restored before commit. - Existing real producer tests cover commit/rollback, deletion with scoped hints in one outer transaction, secondary targets, re-opened tasks, valid/lenient history quietness, startup paging, failure retry, drain/resume and shutdown. The routine failure case now requires no signal when its cleanup has no linked decision. - The earlier dropped-hint and drain-fixture failures remain in [the behavior audit](doc/idle-decision-targets.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. No new schema migration or table is added. The final review cleanup removes the combined internal `sweepExpired` method, cursor, flags and unused target-policy export. A repository/package/SDK audit found no remaining production caller or public exposure. Behavioral tests now call the dedicated workers; isolation tests prove TTL does not recover stale effects and effect recovery does not expire open decisions. HTTP and plugin SDK contracts are unchanged. The full 248-case selection passes after this cleanup and parent reconciliation; types/build pass with the updated Runner vendor. Current-head CI run 40602159553 (attempt 1) failed in the workspace CLI group: company import/export startup exceeded its 30-second health deadline, and the dense ZIP case exceeded its unchanged five-second timeout. The preceding UI group passed all 4,169 cases. Both CLI files and the transfer implementation are unchanged from the landed parent. The complete ZIP-transfer file passes all 17 cases locally on this exact head; the CLI startup E2E is skipped because the host's embedded PostgreSQL bootstrap fails. This does not prove Linux startup locally. The failure logs remain recorded; no timeout or assertion is changed. One bounded retry of the failed jobs is requested after the workflow completed. Its result and the current-head gates remain required before merge. Commands: server typecheck/build and Vitest for `decision-target-work`, `routines-service`, `decisions-service`, `decision-effect-recovery-work`, `decision-expiry-work`, `server-startup-feedback-export` and `scoped-deadline-work`. Local targeted acceptance does not establish deployed suspension or fleet savings. Fresh exact-head CI and Greptile/Ponytail remain merge gates. ## Risks - Deletion now holds the issue row lock while discovering dependent decisions and completing the existing removal transaction. This is needed to serialize new FK references. Provider calls are not placed inside that transaction. - Company deletion with decision-origin references still has the pre-existing restrictive FK limitation. Removing the futile notification does not implement a new company-data cleanup policy. - Supported writers share the application DB owner and maintain existing target links. Independent SQL writers, damaged links or multiple active processes need their own wake contract; startup still reconstructs all open targets. - Cloud sleep blockers and durable external alarms are unchanged. Worker quiet is not proof of deployed host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-decision-targets.md | 164 ++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 140 ++++++- .../decision-effect-recovery-work.test.ts | 13 +- .../__tests__/decision-expiry-work.test.ts | 23 +- .../__tests__/decision-target-work.test.ts | 366 ++++++++++++++++++ .../src/__tests__/decisions-service.test.ts | 29 +- server/src/__tests__/routines-service.test.ts | 13 +- .../server-startup-feedback-export.test.ts | 8 +- server/src/index.ts | 7 +- server/src/services/decision-target-work.ts | 17 + server/src/services/decision-targets.ts | 93 +++++ server/src/services/decisions.ts | 46 +-- .../services/delivery-work-notifications.ts | 1 + server/src/services/issues.ts | 6 + server/src/services/routines.ts | 2 + 15 files changed, 852 insertions(+), 76 deletions(-) create mode 100644 doc/idle-decision-targets.md create mode 100644 server/src/__tests__/decision-target-work.test.ts create mode 100644 server/src/services/decision-target-work.ts create mode 100644 server/src/services/decision-targets.ts diff --git a/doc/idle-decision-targets.md b/doc/idle-decision-targets.md new file mode 100644 index 00000000000..3ceb5153bd2 --- /dev/null +++ b/doc/idle-decision-targets.md @@ -0,0 +1,164 @@ +# Idle decision target invalidation + +Open decisions no longer require a periodic target scan. The target worker +reconciles committed issue changes and decision creation. Startup reconstructs +open decisions in bounded pages. After reconciliation, valid open decisions and +completed history cause no recurring SQL. TTL expiry, continuation delivery, +interrupted effects, and retention remain separate responsibilities. + +## Domain policy + +Only strict effect references participate. The existing shared +`decisionEffectTargetIssueIds` helper includes primary and secondary references. +A missing or cancelled target expires a decision with `target_gone`. If all +strict targets are done and at least one was not done in the proposal snapshot, +it expires with `target_completed`. Lenient decisions and proposals intentionally +acting on already-done targets remain open. A target that reopened before +reconciliation is checked in its current state; the notification is not authority +to expire it. The existing open-status compare-and-set arbitrates expiry against +other decision transitions. + +`decision-targets.ts` owns this policy and scoped reconciliation. It delegates +expiry, activity, and continuation to the existing expiry owner. Startup +registers the target worker when scheduling is enabled. The combined internal +`sweepExpired` method is removed after the final duty is split. No new table or +migration is added. + +## Producer audit + +| Producer | Intent before write | Scope | +| --- | --- | --- | +| Decision creation, including bundles and caller-owned transactions | Existing creation transaction, before insertion | Decision ID | +| `issueService.update` entering/reasserting done or cancelled | Existing issue update transaction, before the patch | Issue ID | +| Issue removal | Lock the issue and capture linked open decision IDs in the existing transaction, before deletion | Affected decision IDs; no signal when none | +| Routine dispatch failure deleting its created issue | Same locked pre-delete lookup in the existing dispatch transaction | Affected decision IDs; usually no signal | + +The existing target link table supports ordinary issue scopes. Before a target +is deleted, the decision owner locks its issue row, reads distinct linked open +decisions in the same company, and signals their IDs. The links may then cascade +without losing notification scope. The row lock closes the race in which a new +proposal could add its target reference after discovery but before deletion. +This remains part of the caller's existing transaction; no provider operation +is wrapped in a new transaction. Production hints require an issue or decision +ID, so routine cleanup and repeated removals do not restart full startup paging. + +Company deletion no longer signals target work. With no decision rows it cannot +create a same-company invalidation obligation. Pre-existing restrictive origin +run/issue foreign keys still prevent deleting some companies with decisions; +that separate limitation is unchanged. This migration does not claim to implement +complete decision-data deletion for those companies. + +Other direct issue writers were inspected: checkout/adoption enter in-progress; +conversation writers enter todo/in-progress/in-review; recovery enters blocked; +release preserves existing terminal status; administrative release changes locks +and assignment. Tree restoration excludes terminal snapshots. Route aliases and +raw privacy SQL do not write terminal status. These paths do not create new +invalidation obligations. They can restore a target before the worker runs; the +worker reads its current status. + +The initial decision feature (0197 / commit `9c1f8e7887`) already stored primary +and secondary target links in the creation transaction. The later shared-helper +extraction `ef33c1d9ed` preserved that behavior. There is no newly assumed legacy +secondary-link backfill. Direct SQL writers or damaged/incomplete link data are +outside the supported live writer contract; startup still examines all open rows. + +## Scheduling and lifecycle + +The owning service holds pending scopes and cursors, not a second durable queue. +The database decisions, options, snapshots, and target rows remain truth. Each +pass examines at most 100 rows/scopes, ordered by expiry and ID using the existing +open-decision index. The cursor retains PostgreSQL timestamp text, including +microseconds, rather than rounding through a JavaScript Date. An empty scope consumes one unit of the pass budget. It +rotates unfinished scopes so a long startup scan does not starve new changes. + +A settled notification replaces its scope with a fresh cursor. An admitted pass +cannot delete that replacement when it completes. Failed rows receive individual +retry scopes while later rows continue; query failures retain their scope. +Retries use the coordinator's existing 30-second cadence only while reconciliation +is unfinished. Unknown hints request a bounded full scan. Once all scopes settle, +`hasPending` returns from memory and no future target scan is scheduled. + +Intent registration is awaited before writes. Nested writes belong to the outer +transaction. Rollback does not publish a change; ambiguous outcomes retain the +existing XID reconciliation contract. Notifications cover the application's DB +owner and child pools, not arbitrary SQL writers or independent active processes. + +Drain preserves pending scopes without passive target SQL. Resume processes them. +Shutdown checks cancellation between rows and awaits admitted expiry/delivery. +Startup reconstructs anything left after process exit. Cloud's conservative +sleep guards are unchanged; this establishes worker quiet, not permission to +sleep with unsupported obligations or proof of host suspension. + +## Current verification + +All 248 cases in seven complete suites pass: 25 target, 80 routine, 54 decision, +26 effect-recovery, 20 expiry, 42 startup and one scoped-deadline case. PostgreSQL +suites use separate migrated external fixtures. Server typecheck and build pass. +No new migration or table is added. The normal parent merge preserves landed +expiry and all 21 independent master files. Server types and build use the +updated Runner vendor (1,524 inputs and 1,563 outputs). + +Four real regressions failed before repair: sub-millisecond paging, deletion +behind a full valid startup page, unrelated deletion notifications and successful +empty-company deletion notifications. They all pass. The new concurrency test +observes the proposal's actual PostgreSQL lock wait while deletion is held. +Temporarily removing the issue-row lock made the test fail because the proposal +was accepted instead of waiting; restoring it passes. The mutation was restored +before commit. No timeout or production hold was weakened. + +The combined internal method had no production caller after this split. The +server package exposes only its startup module; neither its exports nor the +plugin SDK exposes the decision service. Its tests now call the dedicated TTL, +target, effect and continuation workers, preserving their outcome assertions. +The obsolete cursor, scan options and exported target-policy method are removed. +HTTP and plugin SDK contracts are unchanged. + +Fresh current-head CI and both automated reviews remain required. Historical +fixture and CI failures below remain part of the verification record. Extra +upstream embedded-only tests attempted during parent integration could not +initialize locally; CI must cover that environment. + +## Earlier verification + +The initial 17-case real-PostgreSQL target suite passed. The combined selection +passed 233 tests: those 17, 17 effect-recovery cases, 14 TTL cases, 50 decision +service cases, 39 startup cases, 80 routine-service cases, and 16 routine-route +cases. The final target run passed all 18 cases, including a change to the same scope +during reconciliation. In total, 234 selected cases pass across these runs. + +Coverage includes actual issue update/removal, decision creation after cancellation, +secondary targets, all-target completion, intentional done targets, lenient history, +outer commit/rollback, startup after deleted links, first change and failed expiry +through drain, reopened targets, scope isolation, more than one page of valid +open rows, failure fairness, creation during delivery, and shutdown. The original routine +failure test observed intent and settlement around its cleanup transaction. The +current test requires no target signal when that created issue has no linked +decisions. +Quiet assertions observe SQL through one hour of fake time after settlement. + +Workspace typecheck/build passed before the final test-only addition. No runtime +test failure was suppressed or timeout increased. Existing warnings from startup +fixture signal listeners remain visible. The routine suite now supports the +validated disposable external test database path, so local embedded PostgreSQL +unavailability does not silently skip its 80 cases. + +A local query probe with 40,000 completed decisions, one open decision, and 40,001 +target links used existing indexes: scoped lookup read five buffers (0.012 ms), +full open-page lookup two (0.003 ms). The probe was rolled back. No new schema +index is needed for this change. Full PR CI and fresh review remain required. + +## Earlier review and CI repair + +Review found that an absent deletion hint was dropped when the outer transaction +also carried a scoped hint. Both delete-before-update and update-before-delete +regressions reproduced the missing expiry after the target links cascaded away. +That revision sent the explicit `all` hint alongside scoped hints; the current +pre-delete decision-ID contract below supersedes the global hint. +The repaired target suite passes 20 cases; the TTL suite passes 14 cases. + +The initial CI run failed the TTL worker's drain assertion. It measured SQL +before already-admitted sweeps had finished. The test now joins those specific +sweep promises before measuring passive SQL; unresolved-intent holds remain +intact. Waiting for the entire idle count to reach zero was tested and rejected: +that count correctly includes the retained recovery holds. No production drain +or retry behavior changed for this test repair. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index c4fd7944142..f81d138e2d9 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -16,7 +16,8 @@ Do not mark an item done from a timer removal or a green check alone. | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | -| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482), and targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates | +| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates | + | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | @@ -630,3 +631,140 @@ integrates that landed parent and all 21 independent master files. All 143 cases in its five complete suites pass. Its preceding CI shard interruption remains recorded in the owning effect-recovery audit; it is not a passing receipt. Fresh CI/reviews are required on the reconciled head. + + +Slice 11 is [#17482](https://github.com/paperclipai/paperclip/pull/17482), head +`a5b5994c1d`, based on #17479. It is attached and Greptile review was requested. +Its CI is active with no failure in the latest snapshot. Slice 10's one +serialized-job retry passed; all checks now pass at `63b58cc26f`, with Greptile +5/5 and no unresolved findings. The recorded original timeout remains part of +its verification history. + +## Next owner audit: decision target invalidation + +Branch `codex/sleep-12-decision-target-invalidation` starts from #17482. No target +migration is implemented or ready yet. This is the final responsibility inside +the current combined decision sweep, separate from retention and notifications. + +Preserve its exact policy: only strict effect references count. A missing target +or any cancelled target expires with `target_gone`. If all strict targets are +done and at least one was not done in the saved proposal snapshot, expire with +`target_completed`. Lenient decisions and proposals intentionally targeting +already-done tasks remain open. Include secondary effect references through the +existing `decisionEffectTargetIssueIds` helper. Do not duplicate provider/effect +payload interpretation inside the scheduler or broaden invalidation to every +updated timestamp. TTL validity remains separately enforced on use. + +The direct-writer audit found the main `issueService.update` boundary can enter +done/cancelled and already owns an outer transaction. Release preserves an +existing terminal status; checkout/adoption enter in-progress; conversation +writers enter todo/in-progress/in-review; recovery writers enter blocked. +Tree restoration explicitly excludes terminal snapshots. Administrative release +only changes locks/assignment. Route aliases `issueRows` write source-trust or +owner-notification metadata. Raw SQL updates found in issue/project privacy +propagation change privacy only. These are audit findings, not a substitute for +producer integration tests or final inspection. + +Three direct deletion paths need accounting: issue removal, routine dispatch +failure cleanup, and company deletion. Routine creation is visible through its +service while dispatch holds a separate transaction, so do not assume its failed +issue could never have become a target. Company deletion currently encounters +restricting decision/run references if decisions exist; do not silently fix or +relax those foreign keys in this slice. Single-target links cascade on issue +deletion, so an after-commit lookup through those links cannot identify deleted +targets. Use pre-write intent and either preserve the affected decision scope or +request a bounded startup-style scan for this rare path. + +A candidate design uses the existing coordinator's scoped hints: decision +creation signals its decision ID; terminal issue writes signal their issue ID; +deletion or unknown hints request a bounded full scan. Existing decision-target +links select affected open rows; the domain helper computes invalidation from +current targets and saved snapshots. Stable valid open decisions must become +quiet after their finite reconciliation, even with long retention or no near TTL. +No per-decision timer or new queue table is required. + +Keep scoped scan state in the owner, with bounded pages and fair rotation. +Changes arriving during a scan must reset/requeue that scope without being lost +when the admitted pass completes. Failed candidates retain retry debt while +allowing later scopes to progress. Startup reconstructs all open targets from +durable state; rollback and uncertain writes use the existing transaction +protocol. `hasPending` must not perform passive SQL during drain. Shutdown must +finish admitted expiry and leave later candidates for restart. A changed target +can reopen before reconciliation; inspect current state rather than blindly +expiring from the notification itself. + +Required tests include the actual issue update/removal producers, creation after +cancellation, intentional done-target proposals, secondary targets, lenient +history, no recurring SQL with valid open decisions, outer transactions, +rollback, restart with deleted target links, failed expiry, concurrent scope +changes, fair paging, drain/resume, and shutdown. Exercise relevant route/service +consumers when adding notification calls to their existing transaction boundary. +Do not remove the periodic target sweep before this coverage exists. + + +## Slice 12 implementation: decision targets + +The target owner, scoped notification adapter, and audited producer calls are +implemented. Startup reconstructs open targets; ordinary terminal writes select +linked decisions; deletion requests a bounded full scan after cascading links. +The aggregate decision scan is removed now that its four separate owners exist. +See [target invalidation](../idle-decision-targets.md) for scope and policy. + +The combined real-database/consumer selection passes 233 tests. Workspace +typecheck and build pass. The final target run also passes all 18 cases, including the additional +same-scope concurrent-change regression (234 unique selected cases in total). No new table or migration is introduced. Existing +indexes support both lookup shapes in the 40,000-row completed-history probe. + +Slice 11 #17482 has Greptile 5/5 at `a5b5994c1d` and no unresolved findings. +Its first browser shard 3/8 failed in agent-chat fixture cleanup: cancelling a +run hit a PostgreSQL lock timeout, and the following test saw the experimental +flag left enabled. No CI retry has been used. Reproduce the two tests on the +exact parent head before classifying a retry or repair. This parent is not yet +claimed green. The full original inventory remains in scope. + +Slice 11 #17482 now has all checks passing at `a5b5994c1d`, Greptile 5/5, +no unresolved findings, and no merge conflict. Its one browser-shard retry passed +after both affected cases passed locally on the exact head (2/2). The original +fixture-cleanup lock timeout remains documented in the PR. + +Slice 12 is #17486 at initial head `5062efb368`. Review found that deletion's +absent hint could be dropped when the same outer transaction included a scoped +hint. Both operation orders failed new regressions before the explicit `all` +hint repair. The repaired target/TTL selection passes 34 cases. CI also exposed +a TTL test race: passive-SQL measurement now joins already-admitted sweeps, +without dropping unresolved recovery holds. Fresh CI/review remain required. + +The fresh master audit at `991c47bc6e` identified an additional recurring-server +owner: the Grok routine dispatcher starts a one-second settings/dispatch scan +when public MCP is configured, even when Grok is disabled. Include this in the +full stack as slice 13, separately from decision retention. The disabled baseline +issued three settings queries in 3.2 seconds. Its durable prepared dispatches, +atomic claim, and existing uncertain-outcome policy must remain authoritative. + +Slice 12 review repair: deletion captures affected open decision IDs under an +issue-row lock before links cascade. Ordinary cleanup without linked decisions +and empty-company deletion no longer signal global work. PostgreSQL text cursors +preserve microsecond precision. Four real regressions failed before repair; +removing the new lock also failed a controlled concurrent-proposal test. All +246 target/routine/decision/effect/expiry/startup/deadline cases and server +types/build pass. The existing restrictive company-deletion FKs and explicit +combined-service compatibility path remain documented. Fresh head gates required. + + +Slice 12 final combined-sweep cleanup: the repository/package/SDK audit found no +production consumer or public exposure of `decisionService.sweepExpired` after +the final duty split. Removed that method, its cursor/flags/imports and the +unneeded target-policy export. Its behavioral tests now use the four dedicated +workers. Real isolation tests show TTL does not recover effects and effect +recovery does not expire open decisions. All 248 cases in seven complete suites, +server types and build pass after integrating parent `c304273b75`. This supersedes +the earlier manual-caller compatibility claim. Both creation signals remain: +expiry-only creation and scoped target invalidation. Fresh head gates required. + + +Slice 11 merged at `c304273b75` → `e9e84bb8a1` after exact-head Greptile/Ponytail +5/5 and all 58 checks passed or were neutral/skipped. Slice 12 normally merges +that landed dependency and preserves four independent Runner E2E accounting +files. All runtime and acceptance test sources are byte-identical to its tested +`85ce6545bf` tree (248 cases, server types/build). Fresh final-head CI and reviews +remain required after retargeting to master. diff --git a/server/src/__tests__/decision-effect-recovery-work.test.ts b/server/src/__tests__/decision-effect-recovery-work.test.ts index 4358b8e3215..6c1a17d2b39 100644 --- a/server/src/__tests__/decision-effect-recovery-work.test.ts +++ b/server/src/__tests__/decision-effect-recovery-work.test.ts @@ -316,12 +316,15 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes expect(errors.every(error => String(error).includes("wake unavailable"))).toBe(true); errors.length = 0; }); - it("keeps aggregate maintenance from recovering effects when the worker owns them", async () => { - const row = await create(); await claim(row.id, new Date(Date.now() - 120_000)); - expect(await service().sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false })) - .toEqual({ expired: 0, resumed: 0 }); + it("does not expire overdue open decisions during effect recovery", async () => { + const row = await create(); + await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(decisions.id, row.id)); + expect(await service().recoverEffects()).toEqual({ resumed: 0 }); + expect(await service().get(row.id)).toMatchObject({ status: "open" }); expect(await comments()).toHaveLength(0); - expect(await service().sweepExpired()).toEqual({ expired: 0, resumed: 1 }); expect(await comments()).toHaveLength(1); + expect(wake).not.toHaveBeenCalled(); + expect(await service().sweepTtlExpiry()).toEqual({ expired: 1 }); + expect(wake).toHaveBeenCalledTimes(1); }); it.each(["missing_decider", "missing_option"] as const)("settles %s as durable failure and then becomes quiet", async reason => { diff --git a/server/src/__tests__/decision-expiry-work.test.ts b/server/src/__tests__/decision-expiry-work.test.ts index 51c5c7c9673..7b35a9dde6a 100644 --- a/server/src/__tests__/decision-expiry-work.test.ts +++ b/server/src/__tests__/decision-expiry-work.test.ts @@ -62,8 +62,10 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes async function start() { coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => errors.push(error) }); - await registerDecisionContinuationWork(coordinator, service(), () => !draining).ready; - await registerDecisionExpiryWork(coordinator, service(), () => !draining).ready; + const continuation = service(), expiry = service(); + await registerDecisionContinuationWork(coordinator, continuation, () => !draining).ready; + await registerDecisionExpiryWork(coordinator, expiry, () => !draining).ready; + return { continuation, expiry }; } async function quiet() { await vi.waitFor(async () => { @@ -218,12 +220,14 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes }); it("keeps a failed expiry retry through drain and preserves pending state", async () => { - const row = await create(new Date(Date.now() + 30_000)); await start(); + const row = await create(new Date(Date.now() + 30_000)); const workers = await start(); + const sweeps = [vi.spyOn(workers.expiry, "sweepTtlExpiry"), vi.spyOn(workers.continuation, "sweepContinuations")]; await db.execute(sql`create function decision_expiry_fail() returns trigger language plpgsql as $$ begin if NEW.status = 'expired' then raise exception 'expiry unavailable'; end if; return NEW; end $$`); await db.execute(sql`create trigger decision_expiry_fail before update on decisions for each row execute function decision_expiry_fail()`); await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(errors.length).toBeGreaterThan(0); }); draining = true; expect((await service().get(row.id))?.status).toBe("open"); + await Promise.allSettled(sweeps.flatMap(sweep => sweep.mock.results.filter(result => result.type === "return").map(result => result.value))); const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); await db.execute(sql`drop trigger decision_expiry_fail on decisions`); @@ -247,12 +251,15 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes expect(wake).not.toHaveBeenCalled(); expect(await service().nextExpiryAt()).toBeNull(); }); - it("keeps TTL expiry out of the aggregate sweep when its worker owns the duty", async () => { + it("does not recover stale running effects during TTL expiry", async () => { const row = await create(); - await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(decisions.id, row.id)); - expect(await service().sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false })).toEqual({ expired: 0, resumed: 0 }); - expect((await service().get(row.id))?.status).toBe("open"); - expect(await service().sweepTtlExpiry()).toEqual({ expired: 1 }); expect(wake).toHaveBeenCalledTimes(1); + await db.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: "yes", + decidedByUserId: null, updatedAt: new Date(Date.now() - 120_000), expiresAt: new Date(Date.now() - 1000) }) + .where(eq(decisions.id, row.id)); + expect(await service().sweepTtlExpiry()).toEqual({ expired: 0 }); + expect(await service().get(row.id)).toMatchObject({ status: "decided", executionStatus: "running" }); + expect(wake).not.toHaveBeenCalled(); + expect(await service().nextExpiryAt()).toBeNull(); }); it("pages past a failed candidate and retries it without repeating completed expirations", async () => { diff --git a/server/src/__tests__/decision-target-work.test.ts b/server/src/__tests__/decision-target-work.test.ts new file mode 100644 index 00000000000..f9ec92d1068 --- /dev/null +++ b/server/src/__tests__/decision-target-work.test.ts @@ -0,0 +1,366 @@ +import { randomUUID } from "node:crypto"; +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, agents, authUsers, companies, companyMemberships, createDb, subscribeDatabaseWork, decisionEffectExecutions, + decisions, decisionRetention, decisionTargetIssues, heartbeatRuns, issueComments, issues } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { decisionService } from "../services/decisions.js"; +import { registerDecisionTargetWork } from "../services/decision-target-work.js"; +import { registerDecisionContinuationWork } from "../services/decision-continuation-work.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { issueService } from "../services/issues.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { deleteCompany } from "../services/company-deletion.js"; +import { notifyDecisionTargetIssueRemoval } from "../services/decision-targets.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_DECISION_TARGETS_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("decision target worker", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let companyId: string, agentId: string, originId: string, targetId: string, runId: string, userId: string; + let coordinator: ReturnType | undefined; + let draining: boolean; + let recovery: ReturnType; + const errors: unknown[] = []; + const wake = vi.fn(async () => {}); + const service = () => decisionService(db, { wakeOriginAgent: wake }); + + beforeAll(async () => { + if (!externalUrl) temporary = await startEmbeddedPostgresTestDatabase("decision-targets-"); + db = createDb(externalUrl ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 30_000); + beforeEach(async () => { + vi.stubEnv("PAPERCLIP_DECISION_SIGNING_SECRET", "0123456789abcdef0123456789abcdef"); + companyId = randomUUID(); agentId = randomUUID(); originId = randomUUID(); targetId = randomUUID(); runId = randomUUID(); userId = randomUUID(); + await db.insert(companies).values({ id: companyId, name: "Recovery", issuePrefix: `R${companyId.slice(0, 6)}`, requireBoardApprovalForNewAgents: false }); + await db.insert(authUsers).values({ id: userId, name: "Decider", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "member" }); + await db.insert(agents).values({ id: agentId, companyId, name: "Proposer", role: "engineer", status: "active", adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} }); + await db.insert(issues).values([ + { id: originId, companyId, title: "Origin", status: "in_progress", priority: "medium", assigneeAgentId: agentId, responsibleUserId: userId }, + { id: targetId, companyId, title: "Target", status: "todo", priority: "medium", responsibleUserId: userId }, + ]); + await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", responsibleUserId: userId, contextSnapshot: { issueId: originId } }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + draining = false; errors.length = 0; wake.mockReset(); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs(); + await db.execute(sql`drop trigger if exists decision_target_expire_fail on decisions`); + await db.execute(sql`drop function if exists decision_target_expire_fail()`); + await db.delete(decisionEffectExecutions); await db.delete(decisionTargetIssues); await db.delete(decisions); + await db.delete(decisionRetention); await db.delete(activityLog); await db.delete(issueComments); await db.delete(heartbeatRuns); + await db.delete(issues); await db.delete(agents); await db.delete(companyMemberships); await db.delete(authUsers); await db.delete(companies); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + + const actor = () => ({ type: "agent" as const, companyId, agentId, runId, source: "agent_jwt" as const, + onBehalfOfUserId: userId, onBehalfOfMemberships: [{ companyId, membershipRole: "member", status: "active" }] }); + const board = () => ({ type: "board" as const, userId, companyIds: [companyId], source: "session" as const, + memberships: [{ companyId, membershipRole: "member", status: "active" }] }); + const create = (staleness: "strict" | "lenient" = "strict") => service().create({ companyId, agentId, runId, actor: actor(), title: "Comment?", body: "Body", + continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [ + { type: "comment_on_issue", targetIssueId: targetId, staleness, bodyMarkdown: "hello" }, + ] }] }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, + onError: error => errors.push(error) }); + await registerDecisionContinuationWork(coordinator, service(), () => !draining).ready; + recovery = service(); + await registerDecisionTargetWork(coordinator, recovery, () => !draining).ready; + } + async function quiet() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"), executes = vi.spyOn(db, "execute"); + await vi.advanceTimersByTimeAsync(60 * 60_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); executes.mockRestore(); + } + async function expired(id: string, expiredReason = "target_gone") { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(await service().get(id)).toMatchObject({ status: "expired", metadata: { continuationPending: false, expiredReason } }); + }, { timeout: 5000 }); + } + async function failExpiry() { + await db.execute(sql`create function decision_target_expire_fail() returns trigger language plpgsql as $$ begin + if NEW.status = 'expired' then raise exception 'expiry unavailable'; end if; + return NEW; end $$`); + await db.execute(sql`create trigger decision_target_expire_fail before update on decisions for each row execute function decision_target_expire_fail()`); + } + + const change = (status: "done" | "cancelled") => issueService(db).update(targetId, { status, actorUserId: userId }); + + it("stays quiet with valid open decisions and wakes from the real terminal update", async () => { + await start(); await quiet(); const row = await create(); await quiet(); + await change("done"); await expired(row.id, "target_completed"); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("finishes startup paging across submillisecond expiry values", async () => { + const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions); + await db.insert(decisions).values(Array.from({ length: 101 }, () => ({ ...seed, id: randomUUID() }))); + await db.execute(sql`update decisions set expires_at = date_trunc('milliseconds', now() + interval '1 day') + interval '456 microseconds'`); + const owner = service(); + await owner.sweepInvalidTargets(); + expect(owner.hasDecisionTargetWork()).toBe(true); + await owner.sweepInvalidTargets(); + expect(owner.hasDecisionTargetWork()).toBe(false); + expect(wake).not.toHaveBeenCalled(); + }); + + it("reaches a deleted target behind a full page of valid startup decisions", async () => { + const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions); + const invalidId = "ffffffff-ffff-4fff-afff-ffffffffffff"; + const valid = Array.from({ length: 100 }, (_, index) => ({ ...seed, + id: `01000000-0000-4000-8000-${String(index).padStart(12, "0")}`, + options: [{ id: "yes", label: "Yes", effects: [{ type: "comment_on_issue" as const, targetIssueId: originId, staleness: "strict" as const, bodyMarkdown: "hello" }] }], + })); + await db.insert(decisions).values([...valid, { ...seed, id: invalidId }]); + await db.insert(decisionTargetIssues).values({ companyId, decisionId: invalidId, issueId: targetId }); + const owner = service(); + await owner.sweepInvalidTargets(); + expect(owner.hasDecisionTargetWork()).toBe(true); + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, (event, hints) => { + if (event === "settled") owner.decisionTargetsChanged(hints); + }); + try { + await issueService(db).remove(targetId); + expect(await owner.sweepInvalidTargets()).toEqual({ expired: 1 }); + expect((await owner.get(invalidId))?.metadata).toMatchObject({ expiredReason: "target_gone" }); + expect(owner.hasDecisionTargetWork()).toBe(false); + expect(wake).toHaveBeenCalledTimes(1); + } finally { unsubscribe(); } + }); + + it("fences new target references between deletion discovery and commit", async () => { + let entered!: () => void, release!: () => void; + const discovered = new Promise(resolve => { entered = resolve; }); + const gate = new Promise(resolve => { release = resolve; }); + const removing = db.transaction(async tx => { + await notifyDecisionTargetIssueRemoval(tx, targetId); + entered(); await gate; + await tx.delete(issues).where(eq(issues.id, targetId)); + }); + let proposing: Promise<{ accepted: boolean }> | undefined; + try { + await discovered; + proposing = create().then(() => ({ accepted: true }), () => ({ accepted: false })); + await vi.waitFor(async () => { + const blocked = await db.execute(sql`select pid from pg_stat_activity where datname = current_database() + and pid <> pg_backend_pid() and wait_event_type = 'Lock' and query like '%decision_target_issues%'`); + expect(blocked).toHaveLength(1); + }); + release(); await removing; + await expect(proposing).resolves.toEqual({ accepted: false }); + expect(await db.select().from(decisions)).toEqual([]); + expect(wake).not.toHaveBeenCalled(); + } finally { release(); await Promise.allSettled([removing, ...(proposing ? [proposing] : [])]); } + }); + + it("does not signal target work for removal with no linked decisions", async () => { + await create(); + const unrelatedId = randomUUID(); + await db.insert(issues).values({ id: unrelatedId, companyId, title: "Unrelated", status: "todo", responsibleUserId: userId }); + const settled = vi.fn(); + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, event => { if (event === "settled") settled(); }); + try { await issueService(db).remove(unrelatedId); expect(settled).not.toHaveBeenCalled(); } + finally { unsubscribe(); } + }); + + it("does not launch global target work after deleting an empty company", async () => { + const emptyCompanyId = randomUUID(); + await db.insert(companies).values({ id: emptyCompanyId, name: "Empty", issuePrefix: `E${emptyCompanyId.slice(0, 6)}` }); + const home = await mkdtemp(join(tmpdir(), "decision-target-company-")); + vi.stubEnv("PAPERCLIP_HOME", home); + const settled = vi.fn(); + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, event => { if (event === "settled") settled(); }); + try { + expect(await deleteCompany(db, emptyCompanyId)).toMatchObject({ id: emptyCompanyId }); + expect(settled).not.toHaveBeenCalled(); + } finally { unsubscribe(); await rm(home, { recursive: true, force: true }); } + }); + + it("keeps lenient and intentional already-done proposals open and quiet", async () => { + const lenient = await create("lenient"); await change("done"); const intentional = await create(); + await start(); await quiet(); + expect((await service().get(lenient.id))?.status).toBe("open"); + expect((await service().get(intentional.id))?.status).toBe("open"); expect(wake).not.toHaveBeenCalled(); + }); + + it("expires a decision created after its target was cancelled", async () => { + await change("cancelled"); await start(); await quiet(); const row = await create(); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("reacts to the real removal producer after target links cascade away", async () => { + const row = await create(); await start(); await quiet(); await issueService(db).remove(targetId); + expect(await db.select().from(decisionTargetIssues).where(eq(decisionTargetIssues.decisionId, row.id))).toHaveLength(0); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it.each(["cancelled", "deleted"] as const)("reconstructs %s target invalidation at startup", async action => { + const row = await create(); + if (action === "deleted") await db.delete(issues).where(eq(issues.id, targetId)); + else await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, targetId)); + await start(); await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("includes a strict secondary effect reference in notification scope", async () => { + const blockerId = randomUUID(); + await db.insert(issues).values({ id: blockerId, companyId, title: "Blocker", status: "todo", responsibleUserId: userId }); + const row = await service().create({ companyId, actor: actor(), agentId, runId, title: "Follow up?", body: "Body", + continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [ + { type: "create_issue", targetIssueId: targetId, staleness: "strict", draft: { title: "Next", blockedByIssueIds: [blockerId] } }, + ] }] }); + await start(); await quiet(); await issueService(db).update(blockerId, { status: "cancelled", actorUserId: userId }); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("waits for the last strict target to complete", async () => { + const blockerId = randomUUID(); + await db.insert(issues).values({ id: blockerId, companyId, title: "Blocker", status: "todo", responsibleUserId: userId }); + const row = await service().create({ companyId, actor: actor(), agentId, runId, title: "Follow up?", body: "Body", + continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [ + { type: "create_issue", targetIssueId: targetId, staleness: "strict", draft: { title: "Next", blockedByIssueIds: [blockerId] } }, + ] }] }); + await start(); await quiet(); await change("done"); await quiet(); + expect((await service().get(row.id))?.status).toBe("open"); + await issueService(db).update(blockerId, { status: "done", actorUserId: userId }); + await expired(row.id, "target_completed"); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("waits for the actual issue update's outer transaction and ignores rollback", async () => { + const row = await create(); await start(); await quiet(); + await expect(db.transaction(async tx => { + await issueService(db).update(targetId, { status: "cancelled", actorUserId: userId }, tx); + await vi.advanceTimersByTimeAsync(120_000); expect(wake).not.toHaveBeenCalled(); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect((await service().get(row.id))?.status).toBe("open"); + await db.transaction(async tx => { + await issueService(db).update(targetId, { status: "cancelled", actorUserId: userId }, tx); + await vi.advanceTimersByTimeAsync(120_000); expect(wake).not.toHaveBeenCalled(); + }); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it.each([true, false])("preserves a deletion mixed with a scoped hint in one transaction, deleteFirst=%s", async deleteFirst => { + const row = await create(); const otherId = randomUUID(); + await db.insert(issues).values({ id: otherId, companyId, title: "Unrelated", status: "todo", responsibleUserId: userId }); + await start(); await quiet(); + await db.transaction(async tx => { + const scoped = issueService(tx as unknown as typeof db); + if (deleteFirst) await scoped.remove(targetId); + await scoped.update(otherId, { status: "done", actorUserId: userId }); + if (!deleteFirst) await scoped.remove(targetId); + }); + expect(await db.select().from(decisionTargetIssues).where(eq(decisionTargetIssues.decisionId, row.id))).toHaveLength(0); + await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("retains a first target change through drain without passive SQL", async () => { + const row = await create(); await start(); await quiet(); draining = true; await change("cancelled"); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("checks current state if a cancelled target reopened before reconciliation", async () => { + const row = await create(); await start(); await quiet(); draining = true; await change("cancelled"); + await issueService(db).update(targetId, { status: "todo", actorUserId: userId }); + draining = false; await quiet(); expect((await service().get(row.id))?.status).toBe("open"); expect(wake).not.toHaveBeenCalled(); + }); + + it("retains failed expiry through drain and retries after the database recovers", async () => { + const row = await create(); await start(); await quiet(); await failExpiry(); await change("cancelled"); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(errors.length).toBeGreaterThan(0); }); + draining = true; + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await db.execute(sql`drop trigger decision_target_expire_fail on decisions`); + draining = false; await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + for (const error of errors) expect(error).toMatchObject({ cause: { message: "expiry unavailable" } }); errors.length = 0; + }); + + it("does not inspect unrelated open decisions on an ordinary scoped change", async () => { + const unrelated = await create(); + const otherId = randomUUID(); await db.insert(issues).values({ id: otherId, companyId, title: "Other", status: "todo", responsibleUserId: userId }); + const affected = await service().create({ companyId, actor: actor(), agentId, runId, title: "Other?", body: "Body", + continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [ + { type: "comment_on_issue", targetIssueId: otherId, staleness: "strict", bodyMarkdown: "hello" }, + ] }] }); + await start(); await quiet(); + await db.update(decisions).set({ options: sql`'null'::jsonb` }).where(eq(decisions.id, unrelated.id)); + await issueService(db).update(otherId, { status: "cancelled", actorUserId: userId }); + await expired(affected.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1); + }); + + it("bounds startup pages and continues after more than one page of valid decisions", async () => { + const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions); + await db.insert(decisions).values(Array.from({ length: 101 }, () => ({ ...seed, id: randomUUID() }))); + await start(); expect(recovery.hasDecisionTargetWork()).toBe(true); + await quiet(); expect(await db.select().from(decisions).where(eq(decisions.status, "open"))).toHaveLength(101); + expect(wake).not.toHaveBeenCalled(); + }); + + it("continues past a failing first page and retains its failed candidate", async () => { + const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions); + const ids = Array.from({ length: 101 }, (_, i) => `12000000-0000-4000-8000-${String(i).padStart(12, "0")}`); + await db.insert(decisions).values(ids.map(id => ({ ...seed, id }))); + await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, targetId)); + await db.execute(sql`create function decision_target_expire_fail() returns trigger language plpgsql as $$ begin + if NEW.id = '12000000-0000-4000-8000-000000000000' and NEW.status = 'expired' then raise exception 'expiry unavailable'; end if; + return NEW; end $$`); + await db.execute(sql`create trigger decision_target_expire_fail before update on decisions for each row execute function decision_target_expire_fail()`); + await start(); expect(wake).toHaveBeenCalledTimes(99); expect((await service().get(ids.at(-1)!))?.status).toBe("open"); + await expired(ids.at(-1)!); expect((await service().get(ids[0]))?.status).toBe("open"); + await db.execute(sql`drop trigger decision_target_expire_fail on decisions`); + await expired(ids[0]); await quiet(); expect(wake).toHaveBeenCalledTimes(101); + for (const error of errors) expect(error).toMatchObject({ cause: { message: "expiry unavailable" } }); errors.length = 0; + }); + + it("retains a change to the same issue scope while its earlier pass is still running", async () => { + await change("done"); const intentional = await create(); + await db.update(decisions).set({ expiresAt: new Date(Date.now() + 24 * 60 * 60_000) }).where(eq(decisions.id, intentional.id)); + await issueService(db).update(targetId, { status: "todo", actorUserId: userId }); + const ordinary = await create(); await start(); await quiet(); + wake.mockImplementationOnce(async () => { await change("cancelled"); }); + await change("done"); await expired(ordinary.id, "target_completed"); await expired(intentional.id); await quiet(); + expect(wake).toHaveBeenCalledTimes(2); + }); + + it("learns a decision created during invalidation delivery", async () => { + const first = await create(); await change("cancelled"); let nextId = ""; + wake.mockImplementationOnce(async () => { nextId = (await create()).id; }); + await start(); await expired(first.id); await vi.waitFor(() => expect(nextId).not.toBe("")); + await expired(nextId); await quiet(); expect(wake).toHaveBeenCalledTimes(2); + }); + + it("finishes admitted invalidation and leaves later rows for restart during shutdown", async () => { + const first = await create(), second = await create(); await change("cancelled"); + let release!: () => void; const gate = new Promise(resolve => { release = resolve; }); + wake.mockImplementationOnce(async () => gate); + const starting = start(); let stopped = false; let stopping: Promise; + try { + await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(1)); + stopping = coordinator!.stop().then(() => { stopped = true; }); + await vi.advanceTimersByTimeAsync(60_000); expect(stopped).toBe(false); + } finally { release(); } + await starting; await stopping!; coordinator = undefined; + expect(wake).toHaveBeenCalledTimes(1); + const states = await Promise.all([service().get(first.id), service().get(second.id)]); + expect(states.map(row => row!.status).sort()).toEqual(["expired", "open"]); + await start(); await quiet(); expect(wake).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/__tests__/decisions-service.test.ts b/server/src/__tests__/decisions-service.test.ts index 63719ba1ea7..7fc2cc50c0d 100644 --- a/server/src/__tests__/decisions-service.test.ts +++ b/server/src/__tests__/decisions-service.test.ts @@ -225,12 +225,12 @@ describePg("decisionService", () => { ...original(source, options), archiveReviewedManifest: async () => { throw new Error("archive_failure"); }, })) : undefined; try { - expect(await service().sweepExpired()).toMatchObject({ resumed: 1 }); + expect(await service().recoverEffects()).toMatchObject({ resumed: 1 }); const saved = await service().get(row.id); expect(saved).toMatchObject({ executionStatus: "failed", metadata: { archiveProposalError: failure, continuationPending: false } }); expect(saved!.metadata.continuationDeliveredAt).toEqual(expect.any(String)); expect(wakes).toHaveLength(1); - await service().sweepExpired(); + await service().recoverEffects(); expect(wakes).toHaveLength(1); } finally { factory?.mockRestore(); } }); @@ -512,7 +512,7 @@ describePg("decisionService", () => { expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1); }); - it("recovers stale running decisions from the bounded server sweep", async () => { + it("recovers stale running decisions from the effect worker", async () => { process.env.PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS = "0"; const created = await createCommentDecision(); await db.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: "yes", decidedByUserId, @@ -520,7 +520,7 @@ describePg("decisionService", () => { await db.insert(decisionEffectExecutions).values({ decisionId: created.id, effectIndex: 0, effectType: "comment_on_issue", targetIssueId, status: "claimed" }); - expect(await service().sweepExpired()).toEqual({ expired: 0, resumed: 1 }); + expect(await service().recoverEffects()).toEqual({ resumed: 1 }); expect((await service().get(created.id))?.executionStatus).toBe("succeeded"); expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1); expect(wakes).toEqual([{ companyId, agentId, issueId: originIssueId, decisionId: created.id, outcome: "decided" }]); @@ -538,7 +538,7 @@ describePg("decisionService", () => { expect((await service().get(created.id))?.metadata).toMatchObject({ continuationPending: true }); expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1); - await service().sweepExpired(); + await service().sweepContinuations(); expect(wakes).toEqual([{ companyId, agentId, issueId: originIssueId, decisionId: created.id, outcome: "decided" }]); expect((await service().get(created.id))?.metadata).toMatchObject({ continuationPending: false }); @@ -613,8 +613,8 @@ describePg("decisionService", () => { const second = await createCommentDecision("lenient", { idempotencyKey: "batch-2", expiresAt: nearFutureExpiry() }); await expireDecisionNow(first.id); await expireDecisionNow(second.id); - expect((await service().sweepExpired()).expired).toBe(1); - expect((await service().sweepExpired()).expired).toBe(1); + expect((await service().sweepTtlExpiry()).expired).toBe(1); + expect((await service().sweepTtlExpiry()).expired).toBe(1); }); it("falls back to the default sweep batch size for invalid configuration", async () => { @@ -624,7 +624,7 @@ describePg("decisionService", () => { await expireDecisionNow(first.id); await expireDecisionNow(second.id); - await expect(service().sweepExpired()).resolves.toMatchObject({ expired: 2 }); + await expect(service().sweepTtlExpiry()).resolves.toMatchObject({ expired: 2 }); }); it("expires TTL and target-gone decisions and wakes the origin agent", async () => { @@ -633,7 +633,8 @@ describePg("decisionService", () => { await db.update(decisions).set({ expiresAt: new Date(0) }).where(eq(decisions.id, ttl.id)); await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, targetIssueId)); await expireDecisionNow(ttl.id); - expect((await service().sweepExpired()).expired).toBe(2); + expect((await service().sweepTtlExpiry()).expired).toBe(1); + expect((await service().sweepInvalidTargets()).expired).toBe(1); const rows = await db.select().from(decisions); expect(rows.find((row) => row.id === ttl.id)?.metadata).toMatchObject({ expiredReason: "ttl" }); expect(rows.find((row) => row.id === gone.id)?.metadata).toMatchObject({ expiredReason: "target_gone" }); @@ -645,7 +646,7 @@ describePg("decisionService", () => { const lenient = await createCommentDecision("lenient", { idempotencyKey: "lenient-survives" }); await db.update(issues).set({ status: "done" }).where(eq(issues.id, targetIssueId)); - expect((await service().sweepExpired()).expired).toBe(1); + expect((await service().sweepInvalidTargets()).expired).toBe(1); expect((await service().get(completed.id))?.metadata).toMatchObject({ expiredReason: "target_completed" }); expect((await service().get(lenient.id))?.status).toBe("open"); @@ -656,7 +657,7 @@ describePg("decisionService", () => { await db.update(issues).set({ status: "done" }).where(eq(issues.id, targetIssueId)); const reopen = await createCommentDecision("strict", { idempotencyKey: "already-done" }); - expect((await service().sweepExpired()).expired).toBe(0); + expect((await service().sweepInvalidTargets()).expired).toBe(0); expect((await service().get(reopen.id))?.status).toBe("open"); }); @@ -691,7 +692,7 @@ describePg("decisionService", () => { }); await db.update(issues).set({ status: "done" }).where(eq(issues.id, blockerId)); - expect((await service().sweepExpired()).expired).toBe(1); + expect((await service().sweepInvalidTargets()).expired).toBe(1); expect((await service().get(created.id))?.metadata).toMatchObject({ expiredReason: "target_completed" }); }); @@ -716,7 +717,7 @@ describePg("decisionService", () => { await service().decide({ id: acceptedAgain.id, optionId: "assign", decidedByUserId, userActor: boardActor() }); await service().dismiss(rejected.id, decidedByUserId, boardActor(), "Not this time"); await expireDecisionNow(stale.id); - await service().sweepExpired(); + await service().sweepTtlExpiry(); const stats = await service().stats(companyId, { originAgentId: agentId }); expect(stats.filters).toEqual({ originAgentId: agentId, since: null }); @@ -825,7 +826,7 @@ describePg("decisionService", () => { if (action === "decide") await svc.decide({ id: row.id, optionId: "yes", decidedByUserId, userActor: boardActor() }); if (action === "cancel") await svc.cancel(row.id, { actorType: "agent", actorId: agentId }); if (action === "dismiss") await svc.dismiss(row.id, decidedByUserId, boardActor()); - if (action === "expire") { await expireDecisionNow(row.id); await svc.sweepExpired(new Date(), { includeContinuations: false }); } + if (action === "expire") { await expireDecisionNow(row.id); await svc.sweepTtlExpiry(); } await quiet(); await delivered(row.id); expect(wake).toHaveBeenCalledTimes(1); expect(wake.mock.calls[0][0]).toMatchObject({ decisionId: row.id, companyId, agentId, issueId: originIssueId }); diff --git a/server/src/__tests__/routines-service.test.ts b/server/src/__tests__/routines-service.test.ts index 22c40d25aa1..399a8bb5400 100644 --- a/server/src/__tests__/routines-service.test.ts +++ b/server/src/__tests__/routines-service.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { subscribeDatabaseWork } from "@paperclipai/db"; import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; import { createHmac, randomUUID } from "node:crypto"; @@ -38,7 +39,8 @@ import * as providerRegistry from "../secrets/provider-registry.ts"; import { routineService } from "../services/routines.ts"; import { secretService } from "../services/secrets.ts"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ROUTINES_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; const originalSecretsProviderEnv = process.env.PAPERCLIP_SECRETS_PROVIDER; const originalPaperclipApiUrlEnv = process.env.PAPERCLIP_API_URL; @@ -55,8 +57,8 @@ describeEmbeddedPostgres("routine service live-execution coalescing", () => { beforeAll(async () => { process.env.PAPERCLIP_API_URL = "http://localhost:3100"; - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-routines-service-"); - db = createDb(tempDb.connectionString); + if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-routines-service-"); + db = createDb(externalUrl ?? tempDb!.connectionString); }, 20_000); afterEach(async () => { @@ -2167,7 +2169,10 @@ describeEmbeddedPostgres("routine service live-execution coalescing", () => { }, }); - const run = await svc.runRoutine(routine.id, { source: "manual" }); + const events: string[] = []; + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, event => { events.push(event); }); + const run = await svc.runRoutine(routine.id, { source: "manual" }).finally(unsubscribe); + expect(events).toEqual([]); expect(run.status).toBe("failed"); expect(run.failureReason).toContain("queue unavailable"); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 3c814c6dcfe..12f46927634 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -348,7 +348,9 @@ vi.mock("../services/index.js", () => ({ list: vi.fn(async () => ({ items: [], nextCursor: null })), })), decisionService: vi.fn(() => ({ - sweepExpired: vi.fn(async () => ({ expired: 0 })), + sweepInvalidTargets: vi.fn(async () => ({ expired: 0 })), + decisionTargetsChanged: vi.fn(), + hasDecisionTargetWork: vi.fn(() => false), recoverEffects: vi.fn(async () => ({ resumed: 0 })), nextEffectRecoveryAt: vi.fn(async () => null), sweepTtlExpiry: vi.fn(async () => ({ expired: 0 })), @@ -926,7 +928,8 @@ describe("startServer feedback export wiring", () => { expect(decisions.recoverEffects).toHaveBeenCalledTimes(1); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionEffectRecovery)).toBe(true); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(true); - expect(decisions.sweepExpired).toHaveBeenCalledWith(expect.any(Date), { includeContinuations: false, includeExpiry: false, includeEffects: false }); + expect(decisions.sweepInvalidTargets).toHaveBeenCalledTimes(1); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionTargets)).toBe(true); } finally { setIntervalSpy.mockRestore(); } @@ -966,6 +969,7 @@ describe("startServer feedback export wiring", () => { expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionContinuation)).toBe(false); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(false); expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionEffectRecovery)).toBe(false); + expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionTargets)).toBe(false); } finally { setIntervalSpy.mockRestore(); } diff --git a/server/src/index.ts b/server/src/index.ts index 0d27d6f5b5e..4ea740f7113 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { registerDecisionTargetWork } from "./services/decision-target-work.js"; import { registerDecisionEffectRecoveryWork } from "./services/decision-effect-recovery-work.js"; import { registerDecisionExpiryWork } from "./services/decision-expiry-work.js"; import { registerDecisionContinuationWork } from "./services/decision-continuation-work.js"; @@ -1845,7 +1846,8 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("decision_effect_recovery_worker", () => registerDecisionEffectRecoveryWork(deliveryWork, decisionExecutor, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); - await step("decision_expiry", () => decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false })); + await step("decision_target_worker", () => registerDecisionTargetWork(deliveryWork, decisionExecutor, + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); const loginSchedule = loginCleanupSchedule(db, config.heartbeatSchedulerIntervalMs); await step("login_cleanup_workers", () => registerLoginCleanupWork(deliveryWork, [ { queue: DELIVERY_QUEUES.adapterLoginCleanup, run: () => adapterLoginReaper.sweep().then(logAdapterLoginReaperResult), next: loginSchedule.adapter }, @@ -1866,9 +1868,6 @@ async function startServerWithDatabaseTeardown( trackHeartbeatSchedulerWork((async () => { if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return; scheduleRoutineStatusReconcile(); - trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false }).catch((err: unknown) => { - logger.error({ err }, "decision expiry sweep failed"); - })); trackHeartbeatSchedulerWork(runRetentionSweep().catch((err: unknown) => { logger.error({ err }, "decision retention sweep failed"); })); diff --git a/server/src/services/decision-target-work.ts b/server/src/services/decision-target-work.ts new file mode 100644 index 00000000000..219c9ad6f2b --- /dev/null +++ b/server/src/services/decision-target-work.ts @@ -0,0 +1,17 @@ +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; +import type { decisionService } from "./decisions.js"; + +export function registerDecisionTargetWork( + coordinator: ReturnType, + service: Pick, "sweepInvalidTargets" | "decisionTargetsChanged" | "hasDecisionTargetWork">, + canRun: () => boolean, +) { + return coordinator.register(DELIVERY_QUEUES.decisionTargets, { + retryMs: 30_000, + pausePassiveMaintenance: true, + onSettled: service.decisionTargetsChanged, + run: signal => canRun() ? service.sweepInvalidTargets(new Date(), signal) : Promise.resolve(), + hasPending: async () => service.hasDecisionTargetWork(), + }); +} diff --git a/server/src/services/decision-targets.ts b/server/src/services/decision-targets.ts new file mode 100644 index 00000000000..8334dbaa2a7 --- /dev/null +++ b/server/src/services/decision-targets.ts @@ -0,0 +1,93 @@ +import { and, asc, eq, getTableColumns, inArray, sql } from "drizzle-orm"; +import { decisions, decisionTargetIssues, issues, type Db } from "@paperclipai/db"; +import { decisionEffectTargetIssueIds } from "@paperclipai/shared"; +import type { decisionExpiryService } from "./decision-expiry.js"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +type Decision = typeof decisions.$inferSelect; +type Cursor = { expiresAt: string; id: string }; +type Scope = { kind: "all" | "issue" | "decision"; id?: string; cursor?: Cursor }; +const open = sql`${decisions.status} = 'open'`; + +export function notifyDecisionTargetWork(transaction: object, hint: `issue:${string}` | `decision:${string}`) { + return notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionTargets, hint); +} + +export async function notifyDecisionTargetIssueRemoval(transaction: Pick, issueId: string) { + const [issue] = await transaction.select({ companyId: issues.companyId }).from(issues) + .where(eq(issues.id, issueId)).for("update"); + if (!issue) return; + const linked = await transaction.selectDistinct({ decisionId: decisionTargetIssues.decisionId }).from(decisionTargetIssues) + .innerJoin(decisions, eq(decisions.id, decisionTargetIssues.decisionId)) + .where(and(eq(decisionTargetIssues.companyId, issue.companyId), eq(decisionTargetIssues.issueId, issueId), eq(decisions.companyId, issue.companyId), open)); + for (const row of linked) await notifyDecisionTargetWork(transaction, `decision:${row.decisionId}`); +} + +export function decisionTargetService(db: Db, expire: ReturnType["expire"]) { + const scopes = new Map([["all", { kind: "all" }]]); + + function changed(hints?: ReadonlySet) { + if (!hints?.size) { scopes.set("all", { kind: "all" }); return; } + for (const hint of hints) { + const [kind, id] = hint.split(":"); + if ((kind === "issue" || kind === "decision") && id) scopes.set(hint, { kind, id }); + else scopes.set("all", { kind: "all" }); + } + } + + async function reason(decision: Decision): Promise<"target_gone" | "target_completed" | null> { + const strictIds = new Set(decision.options.flatMap(option => option.effects + .filter(effect => effect.staleness === "strict").flatMap(decisionEffectTargetIssueIds))); + if (strictIds.size === 0) return null; + const targets = await db.select({ id: issues.id, status: issues.status }).from(issues) + .where(and(eq(issues.companyId, decision.companyId), inArray(issues.id, [...strictIds]))); + if (targets.length !== strictIds.size || targets.some(target => target.status === "cancelled")) return "target_gone"; + const snapshots = decision.targetSnapshots as Record; + return targets.every(target => target.status === "done") && targets.some(target => snapshots[target.id]?.status !== "done") + ? "target_completed" : null; + } + + async function sweep(now = new Date(), signal?: AbortSignal) { + let expired = 0, inspected = 0; + const errors: unknown[] = []; + for (const [key, scope] of [...scopes]) { + if (signal?.aborted || inspected >= 100) break; + if (scopes.get(key) !== scope) continue; + scopes.delete(key); + const limit = 100 - inspected; + try { + const rows = await db.select({ ...getTableColumns(decisions), cursorExpiresAt: sql`${decisions.expiresAt}::text` }).from(decisions).where(and(open, + scope.kind === "decision" ? eq(decisions.id, scope.id!) : undefined, + scope.kind === "issue" ? inArray(decisions.id, db.select({ id: decisionTargetIssues.decisionId }) + .from(decisionTargetIssues).where(eq(decisionTargetIssues.issueId, scope.id!))) : undefined, + scope.cursor ? sql`(${decisions.expiresAt}, ${decisions.id}) > (${scope.cursor.expiresAt}::timestamptz, ${scope.cursor.id}::uuid)` : undefined, + )).orderBy(asc(decisions.expiresAt), asc(decisions.id)).limit(limit); + inspected += Math.max(1, rows.length); + for (const row of rows) { + if (signal?.aborted) { + if (!scopes.has(key)) scopes.set(key, scope); + return { expired }; + } + try { + const invalid = await reason(row); + if (invalid) expired += await expire(row, invalid, now); + } catch (error) { + errors.push(error); + const retry = `decision:${row.id}`; + if (!scopes.has(retry)) scopes.set(retry, { kind: "decision", id: row.id }); + } + scope.cursor = { expiresAt: row.cursorExpiresAt, id: row.id }; + } + if (rows.length === limit && !scopes.has(key)) scopes.set(key, scope); + } catch (error) { + errors.push(error); inspected++; + if (!scopes.has(key)) scopes.set(key, scope); + } + } + if (errors.length === 1) throw errors[0]; + if (errors.length) throw new AggregateError(errors, "Decision target reconciliation failed"); + return { expired }; + } + + return { sweep, changed, hasPending: () => scopes.size > 0 }; +} diff --git a/server/src/services/decisions.ts b/server/src/services/decisions.ts index 9d915c7955d..282327d433f 100644 --- a/server/src/services/decisions.ts +++ b/server/src/services/decisions.ts @@ -1,5 +1,6 @@ +import { decisionTargetService, notifyDecisionTargetWork } from "./decision-targets.js"; import { randomUUID } from "node:crypto"; -import { and, asc, count, desc, eq, gt, gte, inArray, sql } from "drizzle-orm"; +import { and, asc, count, desc, eq, gte, inArray, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { databaseWorkScopeKey, companyMemberships, decisionBundles, decisionEffectExecutions, decisionRetention, decisions, decisionTargetIssues, heartbeatRuns, issueRelations, issues } from "@paperclipai/db"; import { ATTENTION_SOURCE_KINDS, decisionEffectTargetIssueIds } from "@paperclipai/shared"; @@ -15,7 +16,7 @@ import { import { decisionRetentionService, hashAttentionArchiveManifest } from "./decision-retention.js"; import { decisionContinuationService, type DecisionContinuationWake } from "./decision-continuations.js"; -import { decisionEffectRecoveryService, decisionSweepBatchSize } from "./decision-effect-recovery.js"; +import { decisionEffectRecoveryService } from "./decision-effect-recovery.js"; import { decisionExpiryService } from "./decision-expiry.js"; import { notifyDecisionCreated, writeDecisionChange } from "./decision-work-notifications.js"; @@ -128,13 +129,13 @@ function boardCanActDirectly(actor: AuthorizationActor, companyId: string) { export function decisionService(db: Db, options: DecisionServiceOptions) { const continuations = decisionContinuationService(db, options.wakeOriginAgent); const expiry = decisionExpiryService(db, continuations.deliver); + const targets = decisionTargetService(db, expiry.expire); const effectRecovery = decisionEffectRecoveryService(db, recoverDecisionEffects); const scope = databaseWorkScopeKey(db); let flights = effectFlights.get(scope); if (!flights) effectFlights.set(scope, flights = new Map()); const executing = flights; const authz = authorizationService(db); - let targetSweepCursor: string | null = null; type CreateInput = { companyId: string; actor: AuthorizationActor; agentId: string; runId: string; bundleId?: string | null; ruleKey?: string | null; title: string; body: string; options: DecisionOption[]; inputs?: DecisionInput[] | null; expiresAt?: Date | null; idempotencyKey?: string | null; continuationPolicy?: "none" | "wake_origin_agent"; metadata?: Record }; @@ -239,6 +240,7 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { }; const id = randomUUID(); await notifyDecisionCreated(dbOrTx); + await notifyDecisionTargetWork(dbOrTx, `decision:${id}`); const [created] = await dbOrTx.insert(decisions).values({ id, companyId: input.companyId, bundleId: input.bundleId ?? null, originAgentId: input.agentId, originIssueId: provenance.issueId, originRunId: input.runId, ruleKey: input.ruleKey ?? null, title: input.title, body: input.body, options: input.options, inputs: input.inputs ?? null, expiresAt, @@ -760,40 +762,8 @@ export function decisionService(db: Db, options: DecisionServiceOptions) { return true; } - async function sweepExpired(now = new Date(), options: { includeContinuations?: boolean; includeExpiry?: boolean; includeEffects?: boolean } = {}) { - const batchSize = decisionSweepBatchSize(); - const { resumed } = options.includeEffects === false ? { resumed: 0 } : await effectRecovery.sweep(now, batchSize); - if (options.includeContinuations !== false) await continuations.sweep(batchSize); - const ttlRows = options.includeExpiry === false ? [] : await expiry.due(now, batchSize); - const remaining = batchSize - ttlRows.length; - const targetRows = remaining > 0 - ? await db.select().from(decisions) - .where(and(eq(decisions.status, "open"), ...(targetSweepCursor ? [gt(decisions.id, targetSweepCursor)] : []))) - .orderBy(asc(decisions.id)).limit(remaining) - : []; - targetSweepCursor = targetRows.length === remaining && targetRows.length > 0 ? targetRows[targetRows.length - 1]!.id : null; - const rows = [...new Map([...ttlRows, ...targetRows].map((row) => [row.id, row])).values()]; let expired = 0; - for (const decision of rows) { const strictTargetIds = new Set(decision.options.flatMap((option) => option.effects.filter((effect) => effect.staleness === "strict").flatMap(decisionEffectTargetIssueIds))); - const targets = strictTargetIds.size > 0 - ? await db.select({ id: issues.id, status: issues.status }).from(issues).where(and(eq(issues.companyId, decision.companyId), inArray(issues.id, [...strictTargetIds]))) - : []; - const targetGone = targets.length !== strictTargetIds.size || targets.some((target) => target.status === "cancelled"); - // A decision whose strict targets all completed after it was proposed is moot: - // the strict guard would skip its effects anyway, so retire it instead of - // leaving it pending until TTL. Targets that were already done at proposal - // time don't count — those decisions intentionally act on a done issue. - const snapshots = decision.targetSnapshots as Record; - const targetsCompleted = !targetGone && strictTargetIds.size > 0 && - targets.every((target) => target.status === "done") && - targets.some((target) => snapshots[target.id]?.status !== "done"); - if (!targetGone && !targetsCompleted && (options.includeExpiry === false || decision.expiresAt > now)) continue; - const reason = targetGone ? "target_gone" : targetsCompleted ? "target_completed" : "ttl"; - expired += await expiry.expire(decision, reason, now); - } - return { expired, resumed }; - } - - return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepExpired, sweepContinuations: continuations.sweep, + return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepContinuations: continuations.sweep, sweepTtlExpiry: expiry.sweep, nextExpiryAt: expiry.next, - recoverEffects: effectRecovery.sweep, nextEffectRecoveryAt: effectRecovery.next }; + recoverEffects: effectRecovery.sweep, nextEffectRecoveryAt: effectRecovery.next, + sweepInvalidTargets: targets.sweep, decisionTargetsChanged: targets.changed, hasDecisionTargetWork: targets.hasPending }; } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 908be12ff8d..899ca920654 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + decisionTargets: "decision-targets", decisionEffectRecovery: "decision-effect-recovery", decisionExpiry: "decision-expiry", decisionContinuation: "decision-continuation", diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 391d08c660f..1e77167e85d 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,4 +1,6 @@ +import { notifyDecisionTargetWork, notifyDecisionTargetIssueRemoval } from "./decision-targets.js"; import { assertOnboardingFirstTaskCreationAllowed } from "./onboarding-first-task-admission.js"; + import { resolveConnectionIntentContinuationSource, resolveItemVerdictContinuationWake } from "./chat-interaction-continuation.js"; import { insertIssueComment } from "./issue-comment-writes.js"; import { measureRunTransaction } from "./execution-transaction-timing.js"; @@ -11384,6 +11386,9 @@ export function issueService(db: Db) { (patch.status !== undefined || patch.assigneeAgentId !== undefined || patch.assigneeUserId !== undefined))) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors); } + if (patch.status === "done" || patch.status === "cancelled") { + await notifyDecisionTargetWork(tx, `issue:${id}`); + } const updated = await tx .update(issues) .set(patch) @@ -11791,6 +11796,7 @@ export function issueService(db: Db) { try { await notifyTerminalWorkspaceWork(tx, `issue:${id}`); await requestRestorePointDeletion(tx, { sourceIssueId: id }); + await notifyDecisionTargetIssueRemoval(tx, id); removedIssue = await tx .delete(issues) .where(eq(issues.id, id)) diff --git a/server/src/services/routines.ts b/server/src/services/routines.ts index a5384e1fc71..50c200490b7 100644 --- a/server/src/services/routines.ts +++ b/server/src/services/routines.ts @@ -1,3 +1,4 @@ +import { notifyDecisionTargetIssueRemoval } from "./decision-targets.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { verifyAppWebhook } from "./app-webhook.js"; import crypto from "node:crypto"; @@ -1964,6 +1965,7 @@ export function routineService( return updated ?? createdRun; } catch (error) { if (createdIssue) { + await notifyDecisionTargetIssueRemoval(txDb, createdIssue.id); await txDb.delete(issues).where(eq(issues.id, createdIssue.id)); } const failureReason = error instanceof Error ? error.message : String(error); From 2adcde162e33fe5898f654f6955e7e7e016bbfa6 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 19:07:53 -0500 Subject: [PATCH 31/41] fix(server): wake Grok dispatch from committed reservations (#17490) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages agent work and should become idle when no work is owed. > - Grok routine dispatch already persists each outbound intent before delivery. > - Its dispatcher starts a one-second timer whenever public MCP is configured. > - That timer queries settings even when Grok is disabled, and scans dispatches when enabled but idle. > - The reservation transaction and feature-setting transaction already define when work can become available. > - This PR connects those writes to the shared coordinator and retains the Grok service's claim and delivery policy. > - The worker becomes quiet without adding a queue table or replaying uncertain sends. ## Linked Issues or Issue Description Stacked on #17486. This PR owns only Grok routine dispatch. It does not include decision retention or other remaining maintenance owners. Related open Grok PR #17278 concerns Runner credential-lease fixtures, not this dispatcher. The duplicate search found no overlapping dispatcher migration. **What existing behavior does this improve?** Grok routine dispatch recovery in the server. **Current behavior** With public MCP configured, the dispatcher reads instance settings every second, including when Grok is disabled. With Grok enabled, it scans for prepared dispatches every second. The baseline regression observed three queries in 3.2 seconds with Grok disabled. **Proposed behavior** Reconcile once at startup. Wake from committed dispatch reservations and experimental-setting changes. Schedule another pass only while prepared dispatches remain or reconciliation failed. Disabled and settled workers perform no recurring SQL or provider calls. **Reason and benefit** Remove unnecessary tenant database activity while preserving dispatch recovery, current authorization, provider outcome semantics, and shutdown ownership. This follows the existing sleep-aware maintenance direction; ROADMAP review found no competing dispatcher redesign. **Breaking changes** No public API or provider payload change. The internal dispatcher start method receives the shared coordinator; that coordinator now owns shutdown. Sending, accepted, unknown, and terminal dispatches remain non-replayable. This does not extend Cloud wake coverage. ## What Changed - Register the Grok worker with the existing coordinator; remove its independent timer. - Signal reservation and experimental-setting writes before commit. Nested writes follow their outer transaction. - Preserve ten-row batches, durable send claims, current authority checks, and existing cancellation/uncertainty policy. - Join admitted work during shutdown and leave later rows for restart. - Add a concurrent, replay-safe partial index for prepared dispatch lookup. Keep only the five newest generated snapshots. - Reuse the existing real OAuth/agent/routine fixture for lifecycle tests and add a validated external PostgreSQL test option. - Document producers, recovery, quiet conditions, and remaining host-sleep limitations. ## Verification Test revision: `6d81746fe260ee879442e7e0c879cfe88c250c8e`, followed by docs-only head `679f523d78`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed target `8dfed893d8`. Its entire runtime and acceptance-test sources match that reviewed/tested tree; only four independently merged Runner E2E accounting files and the ledger differ, preserved exactly. Migration 0391 and the parent journal prefix are unchanged. Local checks were not repeated on unchanged runtime; fresh final-head CI and reviews are required. - **273 cases pass** in nine complete suites: real-PostgreSQL dispatch lifecycle, Grok qualification, server startup, scoped deadlines and all five instance-settings suites. Server typecheck and production build pass with the updated Runner vendor. - The baseline disabled-feature regression failed with three SQL reads. Existing lifecycle cases cover first reservation, restart, non-replayable history, re-enable, outer commit/rollback, drain, failure/retry, concurrent enqueue, paging, foreground send race, expiry, settings commit/rollback, and shutdown. - Generated migration `0391_third_roulette.sql` adds only the concurrent/idempotent prepared-dispatch index. All 389 parent journal entries are unchanged; five generated snapshots remain (0387–0391). Migration safety/numbering, schema drift and actual fresh apply/replay on two isolated PostgreSQL databases pass. - The earlier 259-case selection and 40,000-row generic-plan probe remain historical evidence: both unchanged queries used the partial index, six shared buffers including the routine join, 0.032ms / 0.025ms local execution. No new deployed suspension or cost claim is made. - Earlier browser CI failed during agent-chat fixture restoration (lock timeout/HTTP 500), followed by a test seeing the experiment left enabled. Those failures remain recorded in the owning audit; they were not dispatcher assertions. Later pre-reconciliation `a74305e2b4` checks were terminal and green. Fresh CI and both 5/5 bot reviews are required for this head. Commands: server Vitest for `grok-dispatch-work`, `grok-bot-routine`, `server-startup-feedback-export`, `scoped-deadline-work` and the five `instance-settings-*` suites; server typecheck/build; DB migration safety/numbering, snapshot drift and fresh migration apply/replay. ## Risks - Signals are local to the shared database owner and its child pools. Independent SQL writers and other replicas need explicit wake integration. Startup recovers committed prepared rows. - Disabled Grok retains prepared rows and resumes after a committed settings change. This does not relax Cloud's binding/assignment/hold guards. - Existing send-error cancellation policy is preserved. Sending or unknown outcomes are not reset or automatically replayed; this is not an exactly-once delivery change. - Shutdown awaits the current send within existing transport timeouts and stops before the next row. - The index adds write/storage cost but bounds queries independently of completed history. Existing migration execution runs concurrent index creation outside a transaction. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-grok-dispatch.md | 121 ++++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 40 +++- packages/db/src/client.ts | 1 + .../db/src/migrations/0391_third_roulette.sql | 1 + ...{0386_snapshot.json => 0391_snapshot.json} | 114 +++++++++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/grok_bot.ts | 2 + server/src/__tests__/grok-bot-routine.test.ts | 41 +---- .../src/__tests__/grok-dispatch-work.test.ts | 174 ++++++++++++++++++ .../__tests__/helpers/grok-routine-fixture.ts | 33 ++++ server/src/app.ts | 4 +- .../services/delivery-work-notifications.ts | 1 + server/src/services/grok-bot/dispatcher.ts | 54 +++--- server/src/services/grok-bot/routine.ts | 2 + server/src/services/instance-settings.ts | 1 + 15 files changed, 535 insertions(+), 61 deletions(-) create mode 100644 doc/idle-grok-dispatch.md create mode 100644 packages/db/src/migrations/0391_third_roulette.sql rename packages/db/src/migrations/meta/{0386_snapshot.json => 0391_snapshot.json} (99%) create mode 100644 server/src/__tests__/grok-dispatch-work.test.ts create mode 100644 server/src/__tests__/helpers/grok-routine-fixture.ts diff --git a/doc/idle-grok-dispatch.md b/doc/idle-grok-dispatch.md new file mode 100644 index 00000000000..799dcce7855 --- /dev/null +++ b/doc/idle-grok-dispatch.md @@ -0,0 +1,121 @@ +# Idle Grok routine dispatch + +The Grok dispatcher previously started a one-second timer whenever public MCP +was configured. Each tick read experimental settings, even when Grok was +otherwise disabled. Enabled ticks scanned prepared dispatches regardless of +whether any work existed. The disabled baseline regression observed three +settings queries in 3.2 seconds. + +The dispatcher now registers with the existing delivery coordinator and shared +scheduler. It performs one startup reconciliation. The worker becomes quiet +when Grok is disabled or when no prepared dispatch remains. Quiet means no +recurring worker SQL or provider calls, including with retained terminal history. +This does not establish whole-instance or database suspension. + +## Ownership and producers + +`grokRoutineService.reserve` is the only supported creator of prepared dispatches. +It signals intent before its insert in the existing transaction. Route verification, +manual retry of a confirmed rejection, and native-broker assignment admission all +use this service. The native broker passes its transaction, so the wake follows +the outer commit. Savepoint release is not publication; rollback does not send. + +`instanceSettingsService.updateExperimental` signals the same queue before its +existing transactional update. This covers enablePublicMcp, enableNativeRunner, +and enableGrokBot changes, including caller-owned outer transactions. It permits +prepared work retained while disabled to resume when enabled. Managed settings +and environment overlays are process configuration and are reconstructed on restart. +Other settings writers update general/default-environment fields, not these flags. + +All other dispatch writers move rows away from prepared: send claims, delivery +settlement, acknowledgement, explicit recovery, assignment completion, and +revocation. None reopens a row as prepared. They cannot introduce hidden new work. +A previously scheduled backlog pass may observe cancellation and disarm. + +Signals cover the current process's shared database owner and child pools. +Independent SQL writers or another process need explicit ingress integration; +this change does not claim cross-process notifications. Startup reconstructs +committed prepared rows from the existing dispatch table. No queue table is added. + +## Delivery and recovery + +The Grok service continues to own current authority checks, binding locks, +prepared-to-sending compare-and-set, secret access, provider delivery, and outcome +settlement. The worker does not interpret provider payloads or relocate policy +into app wiring. A foreground send and background recovery share the durable +claim; only the winner contacts the provider. + +Each pass processes at most ten prepared rows. The next lookup uses the same +predicate and schedules the existing one-second cadence only while prepared work +remains, or after a reconciliation failure. New commits wake the shared worker; +its deadline adapter preserves requested generations through concurrent changes +and drain. The partial created_at/id index keeps both selection and pending-work +lookup independent of retained completed history. The migration builds it +concurrently, is replay-safe, and uses the normal snapshot-retention workflow. + +The existing send-error policy is preserved: if sending throws and the row is +still prepared, the dispatcher cancels it. A row already claimed as sending is +never reset or replayed. Accepted, sending, unknown, acknowledged, rejected, and +cancelled rows do not become automatic retry work. Explicit operator retry still +creates a new intent only for confirmed rejection. This migration does not claim +exactly-once delivery or resolve uncertain provider outcomes automatically. + +## Lifecycle and sleep boundary + +The app registers only when public MCP is configured, after the shared coordinator +exists. Registration does not delay readiness for a provider send. The coordinator +tracks startup work and joins admitted sends on shutdown. Abort prevents the next +row from starting; existing HTTP timeouts bound a current transport operation. +Later prepared rows remain durable for restart. + +Drain and warm standby retain pending requests without passive worker queries. +Resume uses the same coordinator retry path; a first write or failed sweep cannot +be forgotten merely because dispatch was paused. A disabled feature may retain +prepared rows without scheduling a scan; a committed settings change reactivates +it. Disabled feature state is not permission to discard durable work or relax +Cloud sleep guards. Existing binding/assignment/hold blockers remain unchanged. + +## Verification + +- The baseline failed the disabled-feature quiet assertion with three SQL reads. +- 21 real-PostgreSQL lifecycle cases cover disabled/configured idle, first real + reservation, prepared startup recovery, all six non-prepared states, re-enable, + outer commit/rollback, drain/startup, failed scan through drain, concurrent + enqueue, two-page draining, foreground claim overlap, expired unsent work, + settings transaction rollback/commit, and shutdown joining a send. +- The existing 45 Grok qualification cases and 39 startup cases pass together + with the new suite: 105 cases total. +- The four affected settings suites pass 154 cases (259 selected cases total). +- Workspace typecheck, production build, migration checks and replay pass. A + generic-plan probe with 40,000 completed dispatches and one prepared row uses + the partial index for both queries (six shared buffers including the routine + join; 0.032ms batch and 0.025ms next-work execution in this local probe). +- Remote PR review and CI must pass before the slice is ready. + +The tests use the existing real OAuth/agent/routine fixture and a controlled +provider response. A validated disposable external PostgreSQL option permits +local execution where embedded PostgreSQL is unavailable; CI retains embedded +PostgreSQL by default. Provider-host suspension is outside this local proof. + + +## Current parent reconciliation + +The normal merge of target parent `85ce6545bf` preserves its complete tree outside +this duty's explicit files. Grok dispatch, reservation, settings and app wiring +are unchanged from the preceding candidate. The index is regenerated as +`0391_third_roulette.sql`, with concurrent/idempotent execution. All 389 parent +journal entries are unchanged; five generated snapshots remain (0387–0391). +Schema drift, migration safety/numbering and fresh apply/replay on two isolated +PostgreSQL databases pass. + +All 273 cases in nine complete suites pass at `6d81746fe2`: dispatch lifecycle, +Grok qualification, server startup, scoped deadlines and all five instance-settings +suites. Server typecheck and production build pass with the current Runner vendor. +The earlier 259-case and populated-plan measurements above remain historical +evidence; fresh exact-head reviews and CI are required. + +An earlier browser CI run failed during agent-chat fixture restoration with a +lock timeout/HTTP 500; the subsequent case observed the experiment left enabled. +Those failures were not dispatcher assertions and are retained in the evidence. +The later pre-reconciliation `a74305e2b4` checks were terminal and green. Neither +that older result nor local acceptance substitutes for this candidate's CI. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index f81d138e2d9..8d5a064676e 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -16,7 +16,7 @@ Do not mark an item done from a timer removal or a green check alone. | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | -| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates | +| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) merged | @@ -32,6 +32,7 @@ Do not mark an item done from a timer removal or a green check alone. | 14 | Skill-version retention and native-harness backup expiration | Separate retention owners and existing retention guarantees | Pending | | 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Pending | | 16 | Aggregate maintenance tick | Retire enabled and disabled branches after all constituent owners migrate | Pending | +| 17 | Grok routine dispatch | Committed reservations and feature changes; prepared-only recovery | Slice 13 implemented, local verification in progress | ## Stack and verification contract @@ -768,3 +769,40 @@ that landed dependency and preserves four independent Runner E2E accounting files. All runtime and acceptance test sources are byte-identical to its tested `85ce6545bf` tree (248 cases, server types/build). Fresh final-head CI and reviews remain required after retargeting to master. + +## Slice 13: Grok routine dispatch + +The one-second idle scan is replaced with a shared-coordinator worker. The existing +reservation and experimental-settings transactions notify it; the Grok service +retains all authority, claims, delivery, and uncertainty handling. Prepared-only +selection uses a concurrent partial index. The 21 lifecycle cases, 45 existing +Grok cases, and 39 startup cases pass together (105). Workspace typecheck passes. +See [Grok dispatch](../idle-grok-dispatch.md). Build, populated query proof, PR +publication, and fresh review/CI remain required. Decision retention and every +other pending inventory owner remain in scope. + + +Slice 13 final local verification also passes all 154 affected settings cases +(259 selected tests total), workspace typecheck, production build, migration +checks/replay, and a generic-plan query probe with 40,000 completed dispatches. +Both queries use the prepared-only partial index with six shared buffers, +including the routine join. Remote CI/review remain required. + + +Slice 13 normally integrates target parent `85ce6545bf`. Its Grok implementation +is unchanged; migration 0391 is regenerated after the current parent's history. +All 389 parent journal entries are preserved and five snapshots remain. All 273 +cases across nine complete dispatch/qualification/startup/deadline/settings +suites, server types/build, schema drift, migration safety and fresh apply/replay +pass. Prior browser fixture-cleanup failure evidence remains in its owning audit. +Fresh current-head CI and Greptile/Ponytail are required before merge. + + +Slice 12 merged `e6dff4b06f` → `8dfed893d8` after exact-head Greptile/Ponytail 5/5 +and all 58 checks passed or were neutral/skipped. One failed-job retry passed; +the original CLI startup/ZIP timeouts and local startup limitation remain in its +PR evidence. Slice 13 normally integrates that landed dependency and four +independent Runner E2E accounting files. All runtime and acceptance sources match +its tested/reviewed `679f523d78` tree; the 273-case, type/build and migration +receipts retain their original revisions. Migration 0391 and all parent journal +entries are unchanged. Fresh final-head CI and reviews required against master. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index ff714b6919d..0b743db6fce 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -509,6 +509,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0388_ambitious_micromax.sql", "0389_brave_hulk.sql", "0390_salty_killraven.sql", + "0391_third_roulette.sql", "0384_funny_adam_warlock.sql", diff --git a/packages/db/src/migrations/0391_third_roulette.sql b/packages/db/src/migrations/0391_third_roulette.sql new file mode 100644 index 00000000000..473dde35901 --- /dev/null +++ b/packages/db/src/migrations/0391_third_roulette.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "grok_dispatch_prepared_idx" ON "grok_bot_dispatches" USING btree ("created_at","id") WHERE "grok_bot_dispatches"."status" = 'prepared'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0386_snapshot.json b/packages/db/src/migrations/meta/0391_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0386_snapshot.json rename to packages/db/src/migrations/meta/0391_snapshot.json index 8dd26eef4fb..29b3a477203 100644 --- a/packages/db/src/migrations/meta/0386_snapshot.json +++ b/packages/db/src/migrations/meta/0391_snapshot.json @@ -1,6 +1,6 @@ { - "id": "da3229b5-4eab-4d8b-ba6b-d756e6c42430", - "prevId": "9965d300-e355-4f47-ba0a-951ed3b4b8a2", + "id": "e080fc1b-af8e-4012-8ce5-3b5065b65143", + "prevId": "f38ffbf3-c0ce-435e-aad8-9e5afa6b9417", "version": "7", "dialect": "postgresql", "tables": { @@ -22110,6 +22110,34 @@ "concurrently": false, "method": "btree", "with": {} + }, + "decision_archive_notification_outbox_delivery_idx": { + "name": "decision_archive_notification_outbox_delivery_idx", + "columns": [ + { + "expression": "last_attempt_at", + "isExpression": false, + "asc": true, + "nulls": "first" + }, + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decision_archive_notification_outbox\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} } }, "foreignKeys": { @@ -24030,6 +24058,66 @@ "method": "btree", "with": {} }, + "decisions_running_recovery_idx": { + "name": "decisions_running_recovery_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" = 'running'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_open_expiry_idx": { + "name": "decisions_open_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'open'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_pending_continuation_idx": { + "name": "decisions_pending_continuation_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"continuation_policy\" = 'wake_origin_agent' AND \"decisions\".\"metadata\" ->> 'continuationPending' = 'true' AND (\"decisions\".\"status\" IN ('expired', 'cancelled') OR (\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" IN ('succeeded', 'partial', 'failed')))", + "concurrently": false, + "method": "btree", + "with": {} + }, "decisions_bundle_idx": { "name": "decisions_bundle_idx", "columns": [ @@ -32019,6 +32107,28 @@ "method": "btree", "with": {} }, + "grok_dispatch_prepared_idx": { + "name": "grok_dispatch_prepared_idx", + "columns": [ + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"grok_bot_dispatches\".\"status\" = 'prepared'", + "concurrently": false, + "method": "btree", + "with": {} + }, "grok_dispatch_company_status_idx": { "name": "grok_dispatch_company_status_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 9db984a3918..c75fc030ee5 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2724,6 +2724,13 @@ "when": 1793139659776, "tag": "0390_salty_killraven", "breakpoints": true + }, + { + "idx": 391, + "version": "7", + "when": 1793142971791, + "tag": "0391_third_roulette", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/grok_bot.ts b/packages/db/src/schema/grok_bot.ts index da978d3a629..1ad24895a18 100644 --- a/packages/db/src/schema/grok_bot.ts +++ b/packages/db/src/schema/grok_bot.ts @@ -1,3 +1,4 @@ +import { sql } from "drizzle-orm"; import { index, integer, jsonb, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core"; import { companies } from "./companies.js"; import { agents } from "./agents.js"; @@ -49,6 +50,7 @@ export const grokBotDispatches = pgTable("grok_bot_dispatches", { updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), }, t => [ uniqueIndex("grok_dispatch_event_uq").on(t.bindingId, t.bindingGeneration, t.routineRevision, t.sourceEventId), + index("grok_dispatch_prepared_idx").on(t.createdAt, t.id).where(sql`${t.status} = 'prepared'`), index("grok_dispatch_company_status_idx").on(t.companyId, t.status), index("grok_dispatch_assignment_kind_idx").on(t.assignmentId, t.kind), ]); diff --git a/server/src/__tests__/grok-bot-routine.test.ts b/server/src/__tests__/grok-bot-routine.test.ts index 1bc93d8ff41..434da7cfcbb 100644 --- a/server/src/__tests__/grok-bot-routine.test.ts +++ b/server/src/__tests__/grok-bot-routine.test.ts @@ -4,7 +4,7 @@ import { grokBotRoutes } from "../routes/grok-bot.js"; import { publicMcpIngressRoutes } from "../routes/public-mcp.js"; import { createPublicMcpExecutor } from "../services/public-mcp/capabilities.js"; import { createDotRunnerMcpTools } from "../services/dot-runner-broker.js"; -import { createHash, randomBytes, randomUUID } from "node:crypto"; +import { randomBytes, randomUUID } from "node:crypto"; import { and, eq, sql } from "drizzle-orm"; import { beforeAll, afterAll, afterEach, describe, expect, it, vi } from "vitest"; import { agents, authUsers, companies, companyMemberships, createDb, dotAgentBindings, grokBotDispatches, @@ -12,7 +12,8 @@ import { agents, authUsers, companies, companyMemberships, createDb, dotAgentBin import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { accessService } from "../services/access.js"; import { instanceSettingsService } from "../services/instance-settings.js"; -import { createPublicMcpOAuth } from "../services/public-mcp/oauth.js"; +import { grokRoutineFixture } from "./helpers/grok-routine-fixture.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { dotRunnerBroker } from "../services/dot-runner-broker.js"; import { createGrokRoutineDispatcher } from "../services/grok-bot/dispatcher.js"; import { grokRoutineService } from "../services/grok-bot/routine.js"; @@ -26,44 +27,22 @@ import { hasRequiredWorkspaceRecovery } from "../services/workspace-restore-reco import { heartbeatService } from "../services/heartbeat.js"; import { environmentRuntimeService } from "../services/environment-runtime.js"; -let temporary: Awaited>; +let temporary: Awaited> | undefined; let db: ReturnType; beforeAll(async () => { - temporary = await startEmbeddedPostgresTestDatabase("paperclip-grok-routine-"); - db = createDb(temporary.connectionString); + const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_GROK_ROUTINE_TEST_DATABASE_URL"); + if (!externalUrl) temporary = await startEmbeddedPostgresTestDatabase("paperclip-grok-routine-"); + db = createDb(externalUrl ?? temporary!.connectionString); vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", "enforce"); vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", randomBytes(32).toString("base64")); await instanceSettingsService(db).updateExperimental({ enablePublicMcp: true, enableNativeRunner: true, enableGrokBot: true, enableOpenAiDot: true }); }, 120000); -afterAll(async () => { try { await temporary?.cleanup(); } finally { vi.unstubAllEnvs(); } }); +afterAll(async () => { try { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); } finally { vi.unstubAllEnvs(); } }); afterEach(() => vi.restoreAllMocks()); -async function fixture() { - const userId = randomUUID(); - await db.insert(authUsers).values({ id: userId, name: "Grok operator", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); - const [company] = await db.insert(companies).values({ name: "Grok test", issuePrefix: "GK" + randomBytes(3).toString("hex") }).returning(); - await db.insert(companyMemberships).values({ companyId: company!.id, principalType: "user", principalId: userId, membershipRole: "owner", status: "active" }); - await accessService(db).setPrincipalPermission(company!.id, "user", userId, "agents:create", true, userId); - const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Grok test", adapterType: "paperclip_runner", status: "active", adapterConfig: { provider: "grok_bot", allowUnmeteredProvider: true, lifecycleMode: "per_turn" } }).returning(); - const oauth = createPublicMcpOAuth(db, { origin: "https://paperclip.example", resource: "https://paperclip.example/mcp/runner" }); - const client = await oauth.register({ client_name: "Synthetic Grok client", redirect_uris: ["https://client.example/callback"] }, randomUUID()); - const verifier = randomBytes(32).toString("base64url"); - const input = { client_id: client.client_id, redirect_uri: "https://client.example/callback", response_type: "code", resource: oauth.config.resource, - scope: "paperclip:agent offline_access", code_challenge: createHash("sha256").update(verifier).digest("base64url"), code_challenge_method: "S256" }; - const requestId = (await oauth.authorize(input)).split("/").at(-1)!; - const pairing = await dotRunnerBroker(db).createPairing({ companyId: company!.id, agentId: agent!.id, operatorId: userId, provider: "grok_bot" }); - const consent = await oauth.consentDotPairing(requestId, pairing.pairingCode); - const tokens = await oauth.token({ grant_type: "authorization_code", client_id: client.client_id, redirect_uri: input.redirect_uri, - resource: oauth.config.resource, code: new URL(consent.redirectUrl).searchParams.get("code"), code_verifier: verifier }); - const principal = await oauth.authenticate(tokens.access_token); - const delivery = vi.fn().mockResolvedValue({ delivery: "accepted", httpStatus: 200 }); - const service = grokRoutineService(db, delivery); - const config = { companyId: company!.id, agentId: agent!.id, bindingId: pairing.bindingId, operatorId: userId, - expectedRevision: 0, botName: "Synthetic Bot", webhookUrl: "https://bot.example.test/routines/synthetic", webhookKey: "synthetic-sender-key" }; - await service.configure(config); - return { company: company!, agent: agent!, principal, pairing, service, delivery, config, oauth, tokens }; -} +const fixture = () => grokRoutineFixture(db); + async function verification(f: Awaited>, sourceEventId = randomUUID()) { return f.service.reserve({ companyId: f.company.id, agentId: f.agent.id, bindingId: f.pairing.bindingId, sourceEventId, kind: "verify" }); } diff --git a/server/src/__tests__/grok-dispatch-work.test.ts b/server/src/__tests__/grok-dispatch-work.test.ts new file mode 100644 index 00000000000..6d1c243dc5a --- /dev/null +++ b/server/src/__tests__/grok-dispatch-work.test.ts @@ -0,0 +1,174 @@ +import { randomBytes, randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { createDb, grokBotDispatches } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { grokRoutineFixture } from "./helpers/grok-routine-fixture.js"; +import { createGrokRoutineDispatcher } from "../services/grok-bot/dispatcher.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { instanceSettingsService } from "../services/instance-settings.js"; +import { grokRoutineService } from "../services/grok-bot/routine.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_GROK_DISPATCH_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("Grok dispatch worker", () => { + let temporary: Awaited> | undefined; + let db: ReturnType; + let coordinator: ReturnType | undefined; + let worker: ReturnType; + let draining: boolean; + const errors: unknown[] = []; + const delivery = vi.fn(async () => ({ delivery: "accepted" as const, httpStatus: 200 })); + const enable = (enableGrokBot = true) => instanceSettingsService(db).updateExperimental({ enablePublicMcp: true, enableNativeRunner: true, enableGrokBot }); + const reserve = (f: Awaited>, database = db) => grokRoutineService(database, delivery).reserve({ + companyId: f.company.id, agentId: f.agent.id, bindingId: f.pairing.bindingId, sourceEventId: randomUUID(), kind: "verify", + }); + const status = async (id: string) => (await db.select().from(grokBotDispatches).where(eq(grokBotDispatches.id, id)))[0]?.status; + + beforeAll(async () => { + if (!externalUrl) temporary = await startEmbeddedPostgresTestDatabase("grok-dispatch-"); + db = createDb(externalUrl ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 120_000); + beforeEach(async () => { + vi.stubEnv("PAPERCLIP_ISSUE_PRIVACY_MODE", "enforce"); + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", randomBytes(32).toString("base64")); + await db.delete(grokBotDispatches); + await enable(); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + draining = false; errors.length = 0; delivery.mockReset(); + delivery.mockResolvedValue({ delivery: "accepted", httpStatus: 200 }); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs(); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + async function start() { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => errors.push(error) }); + worker = createGrokRoutineDispatcher(db, () => !draining, delivery); + await worker.start(coordinator).ready; + } + async function quiet() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull(); + }, { timeout: 5000 }); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"), executes = vi.spyOn(db, "execute"); + await vi.advanceTimersByTimeAsync(60_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); executes.mockRestore(); + } + async function delivered(id: string) { + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(await status(id)).toBe("accepted"); }, { timeout: 5000 }); + } + + it("does not query every second with Grok disabled", async () => { + await enable(false); await start(); await quiet(); expect(delivery).not.toHaveBeenCalled(); + }); + it("is quiet when configured and wakes from the real reservation", async () => { + const f = await grokRoutineFixture(db); await start(); await quiet(); + const row = await reserve(f); await delivered(row.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(1); + }); + it("reconstructs prepared work on restart", async () => { + const f = await grokRoutineFixture(db); const row = await reserve(f); + await start(); await delivered(row.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(1); + }); + it.each(["sending", "accepted", "unknown", "acknowledged", "rejected", "cancelled"] as const)("does not replay %s history", async state => { + const f = await grokRoutineFixture(db); const row = await reserve(f); + await db.update(grokBotDispatches).set({ status: state }).where(eq(grokBotDispatches.id, row.id)); + await start(); await quiet(); expect(await status(row.id)).toBe(state); expect(delivery).not.toHaveBeenCalled(); + }); + it("keeps disabled prepared work quiet and wakes from committed settings", async () => { + const f = await grokRoutineFixture(db); const row = await reserve(f); await enable(false); + await start(); await quiet(); expect(await status(row.id)).toBe("prepared"); + await enable(); await delivered(row.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(1); + }); + it.each([true, false])("waits for the outer transaction, commit=%s", async commit => { + const f = await grokRoutineFixture(db); await start(); await quiet(); + let release!: () => void; const held = new Promise(resolve => { release = resolve; }); + let created!: Awaited>; + const transaction = db.transaction(async tx => { + created = await reserve(f, tx as unknown as typeof db); await held; + if (!commit) throw new Error("rollback fixture"); + }).catch(error => { if (commit || error.message !== "rollback fixture") throw error; }); + await vi.waitFor(() => expect(created).toBeDefined()); + await vi.advanceTimersByTimeAsync(2000); expect(delivery).not.toHaveBeenCalled(); + release(); await transaction; + if (commit) await delivered(created.id); + else expect(await status(created.id)).toBeUndefined(); + await quiet(); expect(delivery).toHaveBeenCalledTimes(commit ? 1 : 0); + }); + it.each(["startup", "first write"] as const)("preserves %s during drain without passive SQL", async when => { + const f = await grokRoutineFixture(db); + if (when === "first write") { await start(); await quiet(); } + draining = true; const row = await reserve(f); if (when === "startup") await start(); + await vi.advanceTimersByTimeAsync(2000); await vi.waitFor(() => expect(idleWorkSnapshot().active).toBeLessThanOrEqual(1)); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(10_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + expect(delivery).not.toHaveBeenCalled(); draining = false; + await delivered(row.id); await quiet(); + }); + it("retains a failed scan through drain and resumes its retry", async () => { + const f = await grokRoutineFixture(db); await start(); await quiet(); + draining = true; const row = await reserve(f); + const read = vi.spyOn(db, "select").mockImplementationOnce(() => { throw new Error("scan unavailable"); }); + draining = false; await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(errors).toHaveLength(1); }); + read.mockRestore(); draining = true; errors.length = 0; + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(10_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; await delivered(row.id); await quiet(); + }); + it("preserves an enqueue during provider delivery", async () => { + const first = await grokRoutineFixture(db), second = await grokRoutineFixture(db); + await start(); await quiet(); let next!: Awaited>; + delivery.mockImplementationOnce(async () => { next = await reserve(second); return { delivery: "accepted", httpStatus: 200 }; }); + const row = await reserve(first); await delivered(row.id); await vi.waitFor(() => expect(next).toBeDefined()); + await delivered(next.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(2); + }); + it("drains more than one page then stops with retained history", async () => { + const rows = []; + for (let i = 0; i < 11; i++) rows.push(await reserve(await grokRoutineFixture(db))); + await start(); await delivered(rows.at(-1)!.id); await quiet(); + expect(delivery).toHaveBeenCalledTimes(11); + expect((await db.select().from(grokBotDispatches)).every(row => row.status === "accepted")).toBe(true); + }); + it("shares the durable claim with a simultaneous foreground send", async () => { + const f = await grokRoutineFixture(db); await start(); await quiet(); + const row = await reserve(f); + await Promise.all([grokRoutineService(db, delivery).send(f.company.id, f.agent.id, row.id), worker.tick()]); + await delivered(row.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(1); + }); + it("cancels an expired unsent dispatch without contacting the provider", async () => { + const f = await grokRoutineFixture(db); const row = await reserve(f); + await db.update(grokBotDispatches).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(grokBotDispatches.id, row.id)); + await start(); await quiet(); expect(await status(row.id)).toBe("cancelled"); expect(delivery).not.toHaveBeenCalled(); + }); + it("does not enable pending dispatches before the settings transaction commits", async () => { + const f = await grokRoutineFixture(db); const row = await reserve(f); await enable(false); + await start(); await quiet(); + await expect(db.transaction(async tx => { + await instanceSettingsService(tx as unknown as typeof db).updateExperimental({ enableGrokBot: true }); + await vi.advanceTimersByTimeAsync(2000); expect(delivery).not.toHaveBeenCalled(); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect(await status(row.id)).toBe("prepared"); + await db.transaction(async tx => { + await instanceSettingsService(tx as unknown as typeof db).updateExperimental({ enableGrokBot: true }); + await vi.advanceTimersByTimeAsync(2000); expect(delivery).not.toHaveBeenCalled(); + }); + await delivered(row.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(1); + }); + it("joins an admitted send on shutdown and leaves later rows for restart", async () => { + const first = await grokRoutineFixture(db), second = await grokRoutineFixture(db); + let release!: () => void; const held = new Promise(resolve => { release = resolve; }); + delivery.mockImplementationOnce(async () => { await held; return { delivery: "accepted", httpStatus: 200 }; }); + const one = await reserve(first); const two = await reserve(second); + const starting = start(); await vi.waitFor(() => expect(delivery).toHaveBeenCalledTimes(1)); + let stopped = false; const stopping = coordinator!.stop().then(() => { stopped = true; }); + await vi.advanceTimersByTimeAsync(2000); expect(stopped).toBe(false); + release(); await stopping; await starting; + expect(await status(one.id)).toBe("accepted"); expect(await status(two.id)).toBe("prepared"); + await start(); await delivered(two.id); await quiet(); expect(delivery).toHaveBeenCalledTimes(2); + }); +}); diff --git a/server/src/__tests__/helpers/grok-routine-fixture.ts b/server/src/__tests__/helpers/grok-routine-fixture.ts new file mode 100644 index 00000000000..b955a5bbd92 --- /dev/null +++ b/server/src/__tests__/helpers/grok-routine-fixture.ts @@ -0,0 +1,33 @@ +import { createHash, randomBytes, randomUUID } from "node:crypto"; +import { vi } from "vitest"; +import { agents, authUsers, companies, companyMemberships, type createDb } from "@paperclipai/db"; +import { accessService } from "../../services/access.js"; +import { createPublicMcpOAuth } from "../../services/public-mcp/oauth.js"; +import { dotRunnerBroker } from "../../services/dot-runner-broker.js"; +import { grokRoutineService } from "../../services/grok-bot/routine.js"; + +export async function grokRoutineFixture(db: ReturnType) { + const userId = randomUUID(); + await db.insert(authUsers).values({ id: userId, name: "Grok operator", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + const [company] = await db.insert(companies).values({ name: "Grok test", issuePrefix: "GK" + randomBytes(3).toString("hex") }).returning(); + await db.insert(companyMemberships).values({ companyId: company!.id, principalType: "user", principalId: userId, membershipRole: "owner", status: "active" }); + await accessService(db).setPrincipalPermission(company!.id, "user", userId, "agents:create", true, userId); + const [agent] = await db.insert(agents).values({ companyId: company!.id, name: "Grok test", adapterType: "paperclip_runner", status: "active", adapterConfig: { provider: "grok_bot", allowUnmeteredProvider: true, lifecycleMode: "per_turn" } }).returning(); + const oauth = createPublicMcpOAuth(db, { origin: "https://paperclip.example", resource: "https://paperclip.example/mcp/runner" }); + const client = await oauth.register({ client_name: "Synthetic Grok client", redirect_uris: ["https://client.example/callback"] }, randomUUID()); + const verifier = randomBytes(32).toString("base64url"); + const input = { client_id: client.client_id, redirect_uri: "https://client.example/callback", response_type: "code", resource: oauth.config.resource, + scope: "paperclip:agent offline_access", code_challenge: createHash("sha256").update(verifier).digest("base64url"), code_challenge_method: "S256" }; + const requestId = (await oauth.authorize(input)).split("/").at(-1)!; + const pairing = await dotRunnerBroker(db).createPairing({ companyId: company!.id, agentId: agent!.id, operatorId: userId, provider: "grok_bot" }); + const consent = await oauth.consentDotPairing(requestId, pairing.pairingCode); + const tokens = await oauth.token({ grant_type: "authorization_code", client_id: client.client_id, redirect_uri: input.redirect_uri, + resource: oauth.config.resource, code: new URL(consent.redirectUrl).searchParams.get("code"), code_verifier: verifier }); + const principal = await oauth.authenticate(tokens.access_token); + const delivery = vi.fn().mockResolvedValue({ delivery: "accepted", httpStatus: 200 }); + const service = grokRoutineService(db, delivery); + const config = { companyId: company!.id, agentId: agent!.id, bindingId: pairing.bindingId, operatorId: userId, + expectedRevision: 0, botName: "Synthetic Bot", webhookUrl: "https://bot.example.test/routines/synthetic", webhookKey: "synthetic-sender-key" }; + await service.configure(config); + return { company: company!, agent: agent!, principal, pairing, service, delivery, config, oauth, tokens }; +} diff --git a/server/src/app.ts b/server/src/app.ts index d7d0e4ee9ca..f5f083ebf38 100644 --- a/server/src/app.ts +++ b/server/src/app.ts @@ -1094,7 +1094,6 @@ export async function createApp( api.use(publicMcpManagementRoutes(publicMcpOAuth, dotOAuth)); api.use(dotRunnerRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner")); api.use(grokBotRoutes(db, publicMcpOAuth.config.origin + "/mcp/runner")); - grokDispatcher.start(); } api.use(museBoardRoutes(db, opts.authPublicBaseUrl)); @@ -1285,6 +1284,7 @@ export async function createApp( agentLifecycle.setWake(agentLifecycleDelivery.wake); publicMcpEvents?.start(deliveryWork); dotMcpEvents?.start(deliveryWork); + if (publicMcpOAuth) grokDispatcher.start(deliveryWork); const stopMuseMaintenance = startMuseMaintenance(db, deliveryWork); const resourceLifecycleDelivery = registerPluginLifecycleDelivery(deliveryWork, db, workerManager, lifecycle); await resourceLifecycleDelivery.ready; @@ -1461,7 +1461,7 @@ export async function createApp( scheduler.stop(); await agentLifecycle.stop(); await publicMcpEvents?.stop(); - await dotMcpEvents?.stop(); grokDispatcher.stop(); + await dotMcpEvents?.stop(); jobCoordinator.stop(); resourceLifecycleDelivery.dispose(); await deliveryWork.stop(); diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 899ca920654..59389821c89 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -50,6 +50,7 @@ export const DELIVERY_QUEUES = { museMaintenance: "muse-maintenance", mcpEvents: "mcp-events", dotEvents: "dot-events", + grokDispatch: "grok-dispatch", skillSourceSync: "skill-source-sync", fastResponse: "fast-responses", feedback: "feedback-exports", diff --git a/server/src/services/grok-bot/dispatcher.ts b/server/src/services/grok-bot/dispatcher.ts index 864e3707114..b27d77d7a38 100644 --- a/server/src/services/grok-bot/dispatcher.ts +++ b/server/src/services/grok-bot/dispatcher.ts @@ -1,40 +1,44 @@ -import { and, asc, eq } from "drizzle-orm"; +import { and, asc, eq, sql } from "drizzle-orm"; import { grokBotDispatches, grokBotRoutines, type Db } from "@paperclipai/db"; import { grokRoutineService } from "./routine.js"; import type { deliverGrokRoutineWake } from "./webhook.js"; import { externalBotEnabled } from "../external-bot-authority.js"; +import { registerDeadlineWork } from "../deadline-work.js"; +import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../delivery-work-notifications.js"; -/** Drain only unsent durable intents. Sending/accepted/unknown work never retries. */ export function createGrokRoutineDispatcher(db: Db, isBackgroundWorkEnabled: () => boolean = () => true, deliver?: typeof deliverGrokRoutineWake) { - let timer: ReturnType | undefined; - let running = false; - let stopped = false; - async function tick() { - if (running || stopped || !isBackgroundWorkEnabled() || !await externalBotEnabled(db, "grok_bot")) return; - running = true; - try { + let running: Promise | null = null; + let enabled = false; + const prepared = () => db.select({ dispatch: grokBotDispatches, agentId: grokBotRoutines.agentId }).from(grokBotDispatches) + .innerJoin(grokBotRoutines, and(eq(grokBotRoutines.id, grokBotDispatches.routineId), eq(grokBotRoutines.companyId, grokBotDispatches.companyId))) + .where(sql`${grokBotDispatches.status} = 'prepared'`) + .orderBy(asc(grokBotDispatches.createdAt), asc(grokBotDispatches.id)); + function tick(signal?: AbortSignal): Promise { + if (running) return running; + running = (async () => { + if (!isBackgroundWorkEnabled() || signal?.aborted) return; + enabled = await externalBotEnabled(db, "grok_bot"); + if (!enabled) return; const service = grokRoutineService(db, deliver); - // Bot-initiated callers poll their durable request while admission is - // pending. Delay does not prove that the originating conversation ended. - // Starting a webhook here would create a competing Grok interaction. - // An accepted session already has a routine handling it. Quiet MCP - // traffic does not prove that routine ended. Keep feedback and receipts - // durable for the Bot to read instead of starting overlapping work. - const rows = await db.select({ dispatch: grokBotDispatches, agentId: grokBotRoutines.agentId }).from(grokBotDispatches) - .innerJoin(grokBotRoutines, and(eq(grokBotRoutines.id, grokBotDispatches.routineId), eq(grokBotRoutines.companyId, grokBotDispatches.companyId))) - .where(eq(grokBotDispatches.status, "prepared")).orderBy(asc(grokBotDispatches.createdAt)).limit(10); - for (const { dispatch, agentId } of rows) { - if (stopped || !isBackgroundWorkEnabled()) break; + for (const { dispatch, agentId } of await prepared().limit(10)) { + if (signal?.aborted || !isBackgroundWorkEnabled()) break; try { await service.send(dispatch.companyId, agentId, dispatch.id); } catch { - // No HTTP effect started while status is still prepared. Rejected - // authority is not a reason to keep attempting a stale wake. await db.update(grokBotDispatches).set({ status: "cancelled", updatedAt: new Date() }) .where(and(eq(grokBotDispatches.id, dispatch.id), eq(grokBotDispatches.status, "prepared"))); } } - } finally { running = false; } + })().finally(() => { running = null; }); + return running; } - return { tick, start() { stopped = false; timer ??= setInterval(() => { void tick().catch(() => {}); }, 1000); timer.unref(); }, - stop() { stopped = true; if (timer) clearInterval(timer); timer = undefined; } }; + return { + tick, + start: (coordinator: ReturnType) => registerDeadlineWork(coordinator, DELIVERY_QUEUES.grokDispatch, { + run: tick, + next: async () => enabled && (await prepared().limit(1)).length > 0 ? Date.now() : null, + canRun: isBackgroundWorkEnabled, + retryMs: 1000, + }), + }; } diff --git a/server/src/services/grok-bot/routine.ts b/server/src/services/grok-bot/routine.ts index 6e6acb5ac62..144936ae652 100644 --- a/server/src/services/grok-bot/routine.ts +++ b/server/src/services/grok-bot/routine.ts @@ -1,3 +1,4 @@ +import { DELIVERY_QUEUES, notifyDeliveryWork } from "../delivery-work-notifications.js"; import { createHash, randomBytes, randomUUID, timingSafeEqual } from "node:crypto"; import { and, asc, desc, eq, inArray, isNull, isNotNull, lte, or, sql } from "drizzle-orm"; import { z } from "zod"; @@ -155,6 +156,7 @@ export function grokRoutineService(db: Db, deliver: Delivery = deliverGrokRoutin await tx.update(routines).set({ verifiedAt: null, verifiedGrantId: null, updatedAt: new Date() }).where(eq(routines.id, routine.id)); await tx.update(bindings).set({ status: "connected", readyAt: null, updatedAt: new Date() }).where(eq(bindings.id, binding.id)); } + await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch); const [row] = await tx.insert(dispatches).values({ id: correlationId, companyId: input.companyId, routineId: routine.id, bindingId: binding.id, bindingGeneration: binding.generation, grantId: grant.id, routineRevision: routine.revision, sourceEventId: z.string().min(1).max(240).parse(input.sourceEventId), kind: input.kind, diff --git a/server/src/services/instance-settings.ts b/server/src/services/instance-settings.ts index 1d2395ffebe..d13920e8b9f 100644 --- a/server/src/services/instance-settings.ts +++ b/server/src/services/instance-settings.ts @@ -612,6 +612,7 @@ export function instanceSettingsService(db: Db, options: InstanceSettingsService const [updated] = await db.transaction(async tx => { await notifyDeliveryWork(tx, DELIVERY_QUEUES.mcpEvents, "*"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.dotEvents, "*"); + await notifyDeliveryWork(tx, DELIVERY_QUEUES.grokDispatch); return tx .update(instanceSettings) .set({ From 553a004926281e879b5823987fc057e544fa686a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 19:23:24 -0500 Subject: [PATCH 32/41] fix(server): schedule approval retention from durable deadlines (#17497) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages agent work and should become idle when it owes no background work. > - The attention feed archives old pending approvals under the existing 90-day policy. > - The server rebuilds the feed on each maintenance tick, even when approval state has not changed. > - Some approvals have never been read, so retention rows alone cannot define the work. > - Approval records and committed eligibility changes define the real deadline and wake triggers. > - This PR gives approval retention its own deadline worker and reuses the existing retention and archive-delivery contracts. > - The approval lane stops idle scans while other attention sources remain separate migration work. ## Linked Issues or Issue Description Stacked on #17490. This PR owns approval retention only. Archive notification delivery is already separated in #17475. Other attention-source retention and remaining recurring server duties remain in the stack inventory. The related-PR search found no overlapping approval-retention worker migration. **What existing behavior does this improve?** Automatic archival of old pending approvals from attention. **Current behavior** The shared maintenance tick builds the attention feed for each active company. That query discovers pending approvals and materializes retention records before checking their age. An unchanged or empty approval source is read repeatedly. **Proposed behavior** Reconstruct eligible approvals at startup. Schedule the earliest approval activity timestamp plus 90 days. Wake after committed changes to approvals, links, issue visibility, company activity, or retention controls. Stop scheduling when no eligible approval remains. Archive records through the existing version checks, audit writes, and notification outbox. **Reason and benefit** Idle approval retention should generate no recurring SQL. Unseen approvals still need eventual archival. A stale candidate must not archive an approval that was resubmitted while the worker waited for its source lock. **Breaking changes** None intended. The archive age, eligibility, keep controls, audit, and notification policy remain. Normal attention reads still include approvals. No Cloud sleep blocker is relaxed. ## What Changed - Restrict tool-review signals to their existing terminal transition owners (approve/reject/cancel). Creation/link/revision/resubmit signal retention only. - Coalesce retention notifications under one thirty-second aggregate read budget. Known deadlines still preempt it; no notification or durable deadline means no scan. - Add a bounded approval-retention worker. It uses the shared deadline coordinator and reads pending approvals directly. - Share first-linked-issue ordering and execution visibility through `approval-visibility.ts`, used by both the feed and worker. Preserve actor-specific read filtering and locked archival rechecks. - Materialize unseen approvals on demand inside manual keep/archive/revive transactions. Use the saved activity timestamp and existing versioned state; retain company scoping and rollback on failure. - Add awaited pre-write signals to approval, budget, tool, link, visibility, company, and retention mutation owners. Preserve outer transaction ownership; keep provider calls outside SQL wrappers. - Narrow retention inputs to the source fields the existing service uses. Signal materialization only for inserted or changed state so unchanged reads converge. - Exclude approval collection from the maintenance feed before its query. Other source kinds remain in that sweep. - Add the pending-only approval deadline index with concurrent, idempotent creation. Keep the five newest migration snapshots. - Add real-Postgres lifecycle tests and startup wiring tests. Enable validated external test databases for existing attention and budget suites. - Document producer coverage, quiet conditions, recovery and remaining scope in `doc/idle-approval-retention.md` and the stack ledger. ## Verification Current test revision: `dd844f4e8cbbf1db681a774dc31cd552301bef48`, followed by docs-only `3efda33812`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed Grok `2adcde162e`. All runtime and acceptance-test sources are byte-identical to that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration 0392 and the parent journal prefix are unchanged. Local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the approval-retention duty. - **315 cases pass** across eight complete suites: approval retention/service, attention, decision retention, budgets, tool gateway, server startup and scoped deadlines. All 30 real-PostgreSQL lifecycle cases pass. Server production typecheck and build pass. - **Four regressions failed before repair**: unseen young keep/archive/revive returned 404; the manual keep rollback case could not reach its update. They now pass. A failed keep update leaves no retention row or audit; retry succeeds with one audit. - Both linked-issue orders are checked against the feed: hidden first/visible later is excluded; visible first/hidden later remains eligible. The feed and worker use the same owner query, with actor read filtering preserved. - Generated concurrent/idempotent migration `0392_small_frightful_four.sql` adds only the pending approval index. Every historical parent SQL migration and all 390 parent journal entries are unchanged. Five generated snapshots remain (0388–0392). Schema drift, migration safety/numbering and actual fresh apply/replay on five isolated PostgreSQL databases pass. - Existing precision, transaction, drain/resume, failure fairness and shutdown tests remain. Earlier transaction-stub, budget-import, diagnostic-client and CI failures are recorded in the owning audit and stack ledger; later passes do not erase them. - Three additional real regressions failed before the wake repair: ordinary mutations emitted tool-review signals in both resolution cases, and sequential tenant writes caused repeated global reads. Tests now prove terminal approve/reject/cancel signals and one aggregate reconciliation per thirty-second window with a different company's pending approval retained. The aggregate budget is stricter than a separate per-company budget and matches the old default cadence; known deadlines preempt it. - The initial coalescing fixture expected the concrete pending-work hold to clear before reconciliation; it now preserves that hold and measures SQL. The first broad repair passed 313 cases but failed the existing parked-review cancel case. The original cancel producer is restored; all 315 now pass without weakening assertions. At old head `5991f257bb`, security was action-required and Greptile returned service error `8fb1835c-e564-476b-b802-d5be164ea915`. Fresh repaired-head results are required, not assumed. Commands: server Vitest for `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB snapshot drift, migration checks and fresh apply/replay. Fresh exact-head Greptile/Ponytail and CI remain merge gates. No full local repository suite or deployed host suspension/cost claim is made. Earlier stack verification below is historical evidence at the explicitly named revisions, not this head's acceptance: Approved delegated Test-tab calls now evaluate caller and target policies in the existing claim transaction. Real HTTP regressions first showed a missing caller charge and shared-limit self-rejection. All 78 connection-management cases and 92 gateway cases now pass on the owning branch, along with server production typechecking. All 78 connection-management cases also pass at combined tip `d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover shared charges, separate charges, rollback on denial, duplicate approval, and rejection of a later authorized call. Intermediate counter/identity fixture failures and a reused-database Slack collision are recorded in [the owning audit](https://github.com/paperclipai/paperclip/blob/20557d2c3dd0949c734eaa8095fd78b9b07d7fd5/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes). The earlier combined checks below remain evidence for their stated revisions. Current-head CI and review remain required. Earlier published head: `20557d2c3dd0949c734eaa8095fd78b9b07d7fd5`. This remains a single-duty PR stacked on #17490. - Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5` with normal merges. Every branch passed generated-schema checks and its target-branch migration-order gate. Existing SQL is unchanged; the 25 stack migrations follow master’s new migration. - Duty-specific producer coverage, lifecycle tests, and earlier failure records: [2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/76a49810c7ff403ae7720bef0d453747bd844731/doc/plans/2026-10-24-recurring-server-sleep-stack.md). - Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355 startup/cancellation/recovery/dispatch/preservation tests**, **75 connection-management tests**, and production server typechecking pass. These combined checks do not mean every intermediate PR independently ran the full repository suite. - Real master-to-stack database upgrade passes: **379 unchanged prior migration receipts plus 25 new migrations**, with the gateway responsibility fixture preserved. - Owning-branch checks: 84 runner installation/settlement cases plus runner typechecking and generated TypeScript/Rust profile agreement; 253 GitHub review workflow cases. Stop-attribution and held-worker readiness regressions failed before their fixes. - Failure accounting: the initial combined run passed 349 cases and failed six idle assertions in a reused preservation database containing old cancelled runs. Repeating the combined command with a fresh preservation database passed. Earlier invalid project selection, merge import/type errors, and the interrupted full local suite are not counted as passing checks. - **Latest-head CI and review remain required.** Prior CI findings include GitHub idle observation, route teardown, attention cleanup, touch-picker E2E, workspace health, and voice-cleanup timing; local passes do not erase those failures. No full local repository-suite, deployed sleep, fleet cost, or live Copilot qualification claim is made. ## Risks - Signals cover the application database owner and its child pools. Independent SQL writers and other active processes need an explicit wake integration. Startup reconstructs committed durable state. - The source row lock prevents stale approval-activity archival. It does not add cross-table serializability for concurrent link, visibility, or company-policy changes. - Newly changed eligibility can wait up to the thirty-second notification window, matching the former default maintenance cadence. Saved deadlines still preempt; no new idle polling is added. - Failed records retain recovery debt. Bounded cursor progress lets later rows proceed without dropping failures. - The shared retention transaction now groups materialization and activity-version updates. Existing attention and retention consumers are tested against PostgreSQL. - Kept, archived, resolved or inactive-company approvals can become quiet, but their supported mutations must retain their wake notifications. - The local scheduler cannot wake stopped compute. This PR neither adds approval deadlines to Cloud's external alarm contract nor claims measured host or database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-approval-retention.md | 164 +++++++ ...2026-10-24-recurring-server-sleep-stack.md | 224 +++++++++- packages/db/src/client.ts | 1 + .../migrations/0392_small_frightful_four.sql | 1 + ...{0387_snapshot.json => 0392_snapshot.json} | 108 ++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/approvals.ts | 2 + .../__tests__/approval-retention-work.test.ts | 423 ++++++++++++++++++ .../src/__tests__/approvals-service.test.ts | 14 +- .../src/__tests__/attention-service.test.ts | 9 +- server/src/__tests__/budgets-service.test.ts | 10 +- .../server-startup-feedback-export.test.ts | 18 + server/src/index.ts | 4 + server/src/lib/approval-records.ts | 8 +- server/src/services/approval-retention.ts | 67 +++ server/src/services/approval-visibility.ts | 18 + server/src/services/approvals.ts | 33 +- server/src/services/attention.ts | 137 +++--- server/src/services/budgets.ts | 5 + server/src/services/companies.ts | 3 + server/src/services/company-deletion.ts | 2 + server/src/services/decision-retention.ts | 154 ++++--- .../services/delivery-work-notifications.ts | 1 + server/src/services/issue-approvals.ts | 15 +- server/src/services/issues.ts | 5 + .../services/retention-work-notifications.ts | 14 + server/src/services/routines.ts | 2 + server/src/services/tool-gateway.ts | 12 +- 28 files changed, 1273 insertions(+), 188 deletions(-) create mode 100644 doc/idle-approval-retention.md create mode 100644 packages/db/src/migrations/0392_small_frightful_four.sql rename packages/db/src/migrations/meta/{0387_snapshot.json => 0392_snapshot.json} (99%) create mode 100644 server/src/__tests__/approval-retention-work.test.ts create mode 100644 server/src/services/approval-retention.ts create mode 100644 server/src/services/approval-visibility.ts create mode 100644 server/src/services/retention-work-notifications.ts diff --git a/doc/idle-approval-retention.md b/doc/idle-approval-retention.md new file mode 100644 index 00000000000..0675d22be09 --- /dev/null +++ b/doc/idle-approval-retention.md @@ -0,0 +1,164 @@ +# Approval retention without idle polling + +This slice removes approval retention from the recurring attention-feed sweep. +It does not migrate the other attention source kinds or establish host sleep. + +## Obligation and owner + +A pending approval in an active company is eligible for automatic archiving +90 days after its last approval activity, unless kept or already archived. +This is a presentation/attention archive: it does not approve, reject, cancel, +or delete the approval. The deadline and policy are unchanged. + +Approval rows are the source of truth, including those never viewed in the +attention UI. Scheduling only existing `decision_retention` rows would miss +unseen approvals because the old feed reader materializes those rows lazily. +`approval-retention.ts` discovers eligible approvals directly and registers +the earliest actual deadline with the existing delivery coordinator. + +The query retains the attention feed's first-linked-issue rule and consumes +`executionIssueCondition` from the issue visibility owner. Hidden, harness and +agent-conversation tasks remain excluded. A waiting external conversation with +a linked pending approval is active under that same shared visibility policy; +its chat state alone cannot hide this approval. Tests exercise the Slack case. + +Each pass visits at most 100 candidates in activity/id order. The cursor keeps +the database timestamp as text so sub-millisecond precision cannot repeat a +failed page. A failed row does +not prevent later pages from progressing. The failed durable record remains +eligible for recovery. Each candidate is re-read and locked before archiving, +so a resubmission committed while the worker waited cannot archive the new +activity using an old selection. Existing retention version, keep and archive +compare-and-set checks remain authoritative. This does not add cross-table +serializability for concurrent link/visibility/company policy changes. + +## Producer coverage + +| Supported producer | Notification boundary | +| --- | --- | +| Approval create, cancel, revision, resubmit | SQL write transaction in approval records | +| Approval resolution, including hire lifecycle | SQL write, retaining the caller's outer transaction | +| Hard budget approval and budget resolution | Existing accounting transaction | +| Formal tool approval and issue link | SQL write transaction; provider work is outside it | +| Link, unlink and bulk link | Issue approval write transaction | +| Issue visibility changes and deletion | Existing issue mutation transaction | +| Routine creation rollback cleanup | Existing transaction before cascading issue deletion | +| Company activation, pause, archive and deletion | Existing company/accounting transaction | +| Keep, manual archive, revive and accepted archive manifest | Existing retention mutation transaction | +| Retention materialization or source activity change | Signal only when inserting or changing actual state | + +Approval comments do not change approval activity under the existing policy. +Company portability imports do not write approvals; restored durable records +are reconstructed at startup. Signals belong to the application DB owner and +its dedicated child pools. Independent SQL writers or another active process +need explicit integration; this change does not claim to observe them. + +## Shared contracts and lifecycle + +The main maintenance feed excludes the approval source before collecting it. +Public attention reads keep their existing default behavior. Other attention +sources remain in the aggregate sweep until separately migrated. + +The retention service accepts the source fields it actually uses rather than +requiring fabricated full UI attention items. Its existing versioned state, +audit writes, authorization and archive-notification outbox are reused. No new +queue table is introduced. Transaction-aware notifications retain intent until +the outer transaction settles, including nested writes and rollback. + +The deadline worker uses the shared scheduler's generation and drain behavior. +Startup reconstructs pending work; idle drain retains requests without passive +SQL; resume reconciles them. Failed work retains a 30-second recovery spacing. +No eligible approval means no recurring deadline or query. Completed history, +kept/archived approvals, and inactive companies are quiet until a supported +mutation changes eligibility. Shutdown joins admitted work and stops between +candidates. Notification delivery remains owned by the separate archive worker. + +A pending-only `(updated_at, id)` index bounds deadline lookup independently of +resolved approval history. The migration uses concurrent, idempotent creation +and the existing migration runner's nontransactional execution list. + +## Verification and limits + +Real-Postgres lifecycle tests cover unseen sources, future deadlines, retained +history, keep/unkeep, archive/revive, revision/resubmission, company activity, +linked issue visibility/deletion/order, normal versus maintenance attention +reads, nested commits/rollback, startup and first-write drain, failed database +writes through drain, bounded progress past failure, shutdown, and a concurrent +resubmission. The waiting-Slack case also checks durable notification creation. + +Existing approval, attention, retention, budget, tool-gateway and startup suites +are relevant consumers. Track final results and any failures in the PR and the +[stack ledger](plans/2026-10-24-recurring-server-sleep-stack.md). + +The Cloud sleep guards are unchanged. An in-process deadline cannot wake stopped +compute, and this approval deadline has not been added to Cloud's durable wake +contract. This proves only the named worker's quiet/reactivation behavior. + + +## Initial review repair verification + +The feed and retention worker now share `approval-visibility.ts` for the first +linked issue and its execution visibility. The feed still applies actor-specific +read access before including a linked approval. Both link orders are covered: +hidden first/visible later stays excluded; visible first/hidden later stays +eligible. The worker's current-state recheck remains under the approval row lock. + +Manual keep, archive and revive materialize an unseen approval on demand inside +the existing retention write transaction. They lock the company-scoped approval +row in the same order as automatic archival, read its real activity timestamp, +and use the existing versioned `syncItems` operation. Missing or unauthorized +sources retain their existing rejection; other source kinds are unchanged. This +adds no eager startup materialization or recurring scan. + +Four real regressions failed before repair: unseen young keep/archive/revive +returned 404, and the atomic materialization test could not reach the keep write. +All pass after repair. A failing keep update rolls back the inserted retention +row and emits no audit; the next successful keep persists exactly one audit. +All 26 lifecycle cases pass, including the existing timestamp-precision failure +rotation. The complete eight-suite consumer selection passes 311 cases at +`063b61584d`: approval worker/service, attention, retention, budget, tool gateway, +startup and scoped deadlines. Server typecheck and production build pass. + +Normal parent reconciliation preserves every historical parent SQL file and all +390 parent journal entries. The approval index is regenerated through the normal +workflow as concurrent/idempotent `0392_small_frightful_four.sql`; five generated +snapshots remain (0388–0392). Schema drift, safety/numbering and actual fresh +migration apply/replay on five isolated PostgreSQL databases pass. Earlier +transaction-stub, budget-import, query-probe and CI failures remain recorded in +the stack ledger. Fresh current-head CI and both reviews are required. + + +## Wake amplification repair + +Ordinary creation, links, revision requests and resubmission signal retention +only. Tool-review queues are notified by the actual terminal transition owners: +approve/reject in approval resolution, and cancel in the approval service. These +are the same terminal transitions that notified tool reviews before this PR. +Cancellation must remain a producer because it settles parked tool actions as +unavailable. No extra wrapper or scope argument is threaded through those calls. + +The aggregate retention worker now coalesces sequential notifications for thirty +seconds after a successful reconciliation. This is a single instance-wide read +budget, stricter than granting each company an independent thirty-second budget, +and matches the former default maintenance cadence. Known business deadlines +still preempt the coalescing window. The window schedules no work on its own; +without a notification or saved deadline the worker remains quiet. Newly changed +eligibility can wait up to that window, as under the old default maintenance. + +Three new real regressions failed before repair: approve and reject cases +observed unwanted tool-review signals from ordinary mutations, and four +sequential tenant writes caused eight global reads rather than one two-read +reconciliation. Terminal approve/reject/cancel signals are now tested explicitly. +With another company's pending approval present, sequential writes cause no extra +SQL within the window and one reconciliation when the window ends. + +The first coalescing fixture incorrectly expected the pending-work hold to clear +before the requested reconciliation; it now lets the deferred wake remain owed +and measures SQL directly. The first broad repair passed 313 cases and failed +the existing parked-review cancellation case. Restoring the original cancel +producer fixed that regression without weakening its outcome assertion. All +315 cases in eight full suites now pass at `dd844f4e8c`, including 30 approval +worker cases; server types/build pass. The earlier security check at `5991f257bb` +was action-required and Greptile reported service error reference +`8fb1835c-e564-476b-b802-d5be164ea915`. Neither is claimed as passing; the repaired +head requires fresh security, CI and both bot reviews. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index 8d5a064676e..c2f604bfcc6 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -15,7 +15,7 @@ Do not mark an item done from a timer removal or a green check alone. | 01 | Custom-image setup expiry | Session writes, existing expiry, startup/retry/drain | [#17443](https://github.com/paperclipai/paperclip/pull/17443), merged after CI and passing reviews | | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | -| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | +| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Approval retention implemented in slice 14; final review/CI pending; other source retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) merged | @@ -32,7 +32,8 @@ Do not mark an item done from a timer removal or a green check alone. | 14 | Skill-version retention and native-harness backup expiration | Separate retention owners and existing retention guarantees | Pending | | 15 | Automatic database backups | Shared scheduling, actual backup deadlines, existing checkpoint contract | Pending | | 16 | Aggregate maintenance tick | Retire enabled and disabled branches after all constituent owners migrate | Pending | -| 17 | Grok routine dispatch | Committed reservations and feature changes; prepared-only recovery | Slice 13 implemented, local verification in progress | +| 17 | Grok routine dispatch | Committed reservations and feature changes; prepared-only recovery | [#17490](https://github.com/paperclipai/paperclip/pull/17490) merged after exact-head 5/5 reviews and passing CI | +| 18 | Abandoned import transfer spools | Creation/completion events, existing retention deadline, startup recovery | Pending; explicitly retained from the remaining-blocker checklist | ## Stack and verification contract @@ -770,6 +771,7 @@ files. All runtime and acceptance test sources are byte-identical to its tested `85ce6545bf` tree (248 cases, server types/build). Fresh final-head CI and reviews remain required after retargeting to master. + ## Slice 13: Grok routine dispatch The one-second idle scan is replaced with a shared-coordinator worker. The existing @@ -806,3 +808,221 @@ independent Runner E2E accounting files. All runtime and acceptance sources matc its tested/reviewed `679f523d78` tree; the 273-case, type/build and migration receipts retain their original revisions. Migration 0391 and all parent journal entries are unchanged. Fresh final-head CI and reviews required against master. + +## Next owner audit: decision retention + +Slice 13 is [#17490](https://github.com/paperclipai/paperclip/pull/17490), head +`706beb01f2`, based on the repaired target PR #17486 (`1bbfc69590`). It is +attached and review was requested. The target PR now has fresh Greptile 5/5 +with its mixed-hint finding resolved; its fresh CI is still running. Grok's CI +and review are running. No new readiness or merge claim is made. + +Branch `codex/sleep-14-decision-retention` starts from #17490. This section is +an audit, not an implemented retention migration. The original full inventory +remains the completion scope. + +The existing startup/tick retention function selects every active company, +builds `attentionService.list({includeDismissed:true, all:true, +allowUnscopedAll:true})`, then calls `autoArchive`. Notification delivery was +already separated in slice 08. Retention itself is not just an expiry scan of +`decision_retention`: `attentionService` materializes those records via +`syncItems` while building the feed. An item never viewed in the UI still needs +retention coverage. Removing the periodic feed build and looking only at existing +retention rows would silently miss such sources. + +The auto-archive policy uses 90 days since source activity, `keep = false`, and +no existing archive. The 30-day shelf default and per-queue shelf overrides are +UI classification, not the automatic archive deadline. Keep these concepts +separate. Preserve version CAS, actor columns, archiveVersion, notification +outbox creation, and audit in the owning transaction. Manual archive/revive, +keep/unkeep, and archive-proposal manifests are also producers of schedule +changes; they must not require another unrelated source write to take effect. + +Current feed sources and eligibility dependencies observed in attention.ts: + +| Source | Activity timestamp / eligibility dependencies to audit | +| --- | --- | +| approval | Approval updatedAt; pending status; issue approval links | +| issue_thread_interaction | Interaction updatedAt; pending status; issue and audience policy | +| decision | Decision updatedAt; open status; current feed selection has a default 500-row cap | +| join_request | Join updatedAt; pending_approval; matching company invite | +| recovery_action | Recovery updatedAt; active/escalated; human owner and visible source issue | +| blocker_attention | Prospective blockedTransitionAt or terminal blocker's updatedAt; dependency graph, liveness, monitor and unblock-owner state | +| review | Issue updatedAt; in_review; human participant/assignee, pending approval, or stalled review liveness | +| failed_run | finishedAt/updatedAt/createdAt; bounded-retry-exhaustion event; failed/timed_out; nonterminated agent; no newer run for that agent/task | +| budget_alert | Incident updatedAt; current budget overview/window; hard threshold or observed >=85% | +| agent_error_alert | Agent updatedAt; error status | + +`productivity_review` remains in the source-kind contract and retention origin +resolver, but this inspected feed does not directly emit that kind. Do not invent +a new producer or assume all enum values imply live rows. Failed-run attention +has no simple age cutoff in `listAttentionExhaustedRuns`; old rows cannot be +excluded merely because they are old. + +Review/blocker attention depends on more than issue status. The review liveness +input includes agents, queued/running runs, wake requests, pending interactions, +approvals, external-response waits, recovery paths, monitors, and `now`. Identify +any clock-driven eligibility transitions as well as source mutations before +removing the backstop. Company activation must discover old eligible sources, +while company deactivation must stop its passive work. + +A company-scoped dirty signal can coalesce a source change into a finite feed +reconciliation, but it is only valid after every supported source writer and +transaction owner is accounted for. A live-event callback or activity-log hook +alone is not a proven durable producer contract. Do not hide provider/domain +policy in the shared scheduler or add blanket database-write interception. + +Also avoid a notification feedback loop: syncItems currently attempts an +on-conflict insert for all items on every feed read, then updates changed activity +versions. Unconditionally signaling every such attempted insert would cause the +worker's own read/materialization to continually wake itself. Any notification +at this boundary must distinguish actual new/changed retention state and converge. + +Required evidence must include an unseen source aging into retention, all audited +source classes and activation paths, exact archive deadline, activity moving a +deadline later, keep/unkeep and manual archive/revive, no repeated feed rebuild +with unchanged pending or completed history, failure/restart/drain, and a source +change during archive selection. The current version CAS protects retention-row +changes; it does not by itself prove that an underlying source did not change +between the feed read and archive. Preserve or strengthen that boundary explicitly +without silently archiving freshly active work. No retention code or PR is yet +claimed for this branch. + + +Latest remote verification: #17486 has fresh Greptile 5/5 at `1bbfc69590`, +all review findings resolved, and one live build job still pending in run +40132295156. #17490 has fresh Greptile 5/5 at `706beb01f2`, no review threads, +and active CI run 40132581543 (24 completed jobs, 22 still pending, no failures +at this snapshot). These are verified live waits, not completed checks. Do not +rerun them merely because a later observation times out. Continue inspecting +these runs while implementing retention, and propagate any required parent +repair into the later branch. No stack PR was merged. + + +## Approval retention implementation in progress + +Remote verification on October 24: #17486 at `1bbfc69590` and #17490 at +`706beb01f2` each have 56 completed checks, all successful or accepted +neutral/skipped outcomes. Their PR bodies now record completion. Neither was +merged; this remains a stack. + +The retention audit is now split by source owner. Slice 14 migrates **approval +retention only**. Decisions, interactions, join requests, recovery actions, +blocker attention, review, failed runs, budget alerts and agent errors remain +explicitly pending. The original full recurring-work inventory remains scope. + +The implementation discovers pending approvals directly, including approvals +never materialized by an attention-feed read. It uses active-company status, +the feed's first linked issue, shared execution visibility, keep/archive state, +and the approval's updated timestamp. The next obligation is updated timestamp +plus 90 days. Each bounded pass rechecks and locks the approval before using +existing versioned retention, audit and notification-outbox writes. Other +retention sources still use the aggregate sweep; approval collection is excluded +from that maintenance feed before its query, while normal UI reads are unchanged. + +Signals cover approval create/cancel/revision/resubmit/resolve, hard budget +approvals, formal tool approvals, link/unlink/bulk linking, issue visibility and +cascading deletion, company activation/pause/archive/deletion, and keep/archive/ +revive/accepted archive manifests. Source comments do not alter approval activity +under the existing policy. Company portability imports no approval records; +offline restore is reconstructed at startup. Independent SQL writers still +require a wake integration. The Cloud sleep proof is unchanged. + +Initial real-Postgres proof: 14 lifecycle cases passed, then 19 including actual +archive-write faults, page progress past a failed row, startup drain, first-link +ordering and shutdown. The first consumer run passed retention and startup but +failed five approval unit cases because its transaction stub omitted nested +transactions; the stub was repaired without weakening production behavior. +The budget suite initially skipped due to embedded-Postgres availability. Budget +and attention suites now accept validated disposable external test databases. +Expanded consumer verification, final index/migration proof, build and PR review +remain required. No slice-14 PR has been published yet. + + +Slice 14 expanded proof: all 20 worker lifecycle cases pass. The shared execution +visibility predicate is used directly, and a waiting Slack conversation with a +pending approval remains eligible with its archive notification persisted. The +repaired approval stub and attention/retention/startup suites pass. Of 96 consumer +cases, one budget case initially timed out during lazy lifecycle module loading +and raced teardown; moving the import into setup preserves the original timeout +and all five budget cases pass. No assertion or production behavior was weakened. +The initial index diagnostic failed to construct a Drizzle client around the +scoped raw driver; supplying its existing driver options fixed the diagnostic. +The actual deadline query with 40,000 resolved approvals uses the pending index, +four shared buffer hits and 0.314 ms execution; test data rolled back. Migration +replay and checks pass. Workspace types, build and tool-gateway tests remain live +in the current verification run. This is not final acceptance. + + +Slice 14 final local verification passes 317 selected tests: 20 new lifecycle, +96 approval/attention/retention/startup/budget consumers (including the repaired +budget case), and 201 tool-gateway cases. Workspace typecheck, production build, +migration numbering/safety, fresh replay and idempotent index replay pass. The +implementation is committed as `ea3dd40cdd`; remote publication, CI and review +remain required. The earlier failures remain recorded above. + + +## Slice 14 publication and precision review repair + +Approval retention is [#17497](https://github.com/paperclipai/paperclip/pull/17497), +initial head `2f334c678b`, based on #17490. It is attached. Initial CI run +40135489716 completed without a failed CI job. Greptile gave 4/5 with one P2: +JavaScript Date truncation can repeat a full failed page whose source timestamp +contains microseconds (comment 4368633789, thread PRRT_kwDORclPyM6v79u7). + +A database-generated timestamp regression failed before repair: all 100 failed +rows repeated and blocked row 101. The cursor now preserves the database text +timestamp. All 21 lifecycle cases pass afterward, and server typecheck passes. +During the full regression run, the earlier quiet-window test crossed its real +deadline after settlement. It now measures an explicit window before that +deadline; it still asserts no SQL and eventual archival. Total selected coverage +is now 318 unique tests. Fresh remote checks/review are required after this fix. + +Related precision audit is required before final stack acceptance: the earlier +TTL worker (#17479) and effect-recovery worker (#17482) retain Date cursors too. +Add database-precision regressions and repair those owning PRs, then propagate +parent updates through the stack. The already-merged terminal-workspace sweep +also uses a Date keyset cursor; verify and address it as a focused follow-up if +confirmed. These are concrete remaining verification duties, not waived by the +earlier green checks. + +Branch `codex/sleep-15-join-retention` exists at the initial slice-14 head, with +no implementation yet. Its producer audit found request writes in access routes +(accept, replay, auto-approve, approve, reject, key claim), Slack identity access +creation, and company deletion. Eligibility is pending_approval plus matching +invite/company and active company. Invite expiry/revocation do not remove the +existing feed item. Preserve that policy. Retention controls and company status +changes must notify this owner. Decision retention remains separately pending; +its visibility dependencies differ, and normal decisions already have a +maximum 30-day TTL before the 90-day archive threshold. + + +Slice 14 current review repairs: share first-linked-issue/execution visibility in +one owner used by the feed and worker; keep actor read filtering. Materialize +unseen approvals inside existing manual retention transactions, locking the +source before state mutation. Four real failures (three 404s and atomic keep +rollback) are repaired; the reverse visibility-order regression also passes. +All 26 lifecycle cases and all 311 cases across eight complete consumer suites, +server types/build, schema drift, migration safety and fresh apply/replay pass. +Index 0392 is generated after all 390 unchanged parent journal entries; five +snapshots remain. Current parent is `679f523d78`; fresh head gates remain required. + + +Slice 14 wake-amplification follow-up: ordinary approval/link/revision/resubmit +writes no longer notify tool reviews. Approve/reject/cancel retain their existing +terminal wake contract. Retention uses a thirty-second aggregate notification +budget, with known deadlines preempting and no idle recurring scan. Three real +regressions failed before repair. A broad run also caught a removed cancellation +producer (313 pass/1 fail); it is restored. All 315 cases in eight complete suites, +including 30 worker cases, and server types/build pass at `dd844f4e8c`. Prior +coalescing-fixture, security and Greptile service failures remain in the owning +audit. Fresh exact-head checks and reviews are required on publication. + + +Slice 13 merged `15cddfcb0c` → `2adcde162e` after exact-head Greptile/Ponytail +5/5 and all 58 check contexts passed or were neutral/skipped. Slice 14 normally +integrates that landed dependency, preserving four independent Runner E2E +accounting files. Its entire runtime and acceptance sources match the tested +and reviewed `3efda33812` tree (315 cases, server types/build). Migration 0392 +and the parent journal prefix are unchanged. Fresh final-head CI/reviews are +required against master. The abandoned-import-spool inventory duty remains. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 0b743db6fce..2b987eb9720 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -510,6 +510,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0389_brave_hulk.sql", "0390_salty_killraven.sql", "0391_third_roulette.sql", + "0392_small_frightful_four.sql", "0384_funny_adam_warlock.sql", diff --git a/packages/db/src/migrations/0392_small_frightful_four.sql b/packages/db/src/migrations/0392_small_frightful_four.sql new file mode 100644 index 00000000000..b8933f13c11 --- /dev/null +++ b/packages/db/src/migrations/0392_small_frightful_four.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "approvals_pending_retention_idx" ON "approvals" USING btree ("updated_at","id") WHERE "approvals"."status" = 'pending'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0387_snapshot.json b/packages/db/src/migrations/meta/0392_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0387_snapshot.json rename to packages/db/src/migrations/meta/0392_snapshot.json index 7fd3f6cbe93..0f1a1921bbb 100644 --- a/packages/db/src/migrations/meta/0387_snapshot.json +++ b/packages/db/src/migrations/meta/0392_snapshot.json @@ -1,6 +1,6 @@ { - "id": "de27fdf8-430a-443a-9502-725f92c48892", - "prevId": "da3229b5-4eab-4d8b-ba6b-d756e6c42430", + "id": "6974dd25-b8e9-4143-b4b7-ee5a5c3ff54a", + "prevId": "e080fc1b-af8e-4012-8ce5-3b5065b65143", "version": "7", "dialect": "postgresql", "tables": { @@ -5721,6 +5721,28 @@ } }, "indexes": { + "approvals_pending_retention_idx": { + "name": "approvals_pending_retention_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"approvals\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, "approvals_company_status_type_idx": { "name": "approvals_company_status_type_idx", "columns": [ @@ -24058,6 +24080,66 @@ "method": "btree", "with": {} }, + "decisions_running_recovery_idx": { + "name": "decisions_running_recovery_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" = 'running'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_open_expiry_idx": { + "name": "decisions_open_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'open'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_pending_continuation_idx": { + "name": "decisions_pending_continuation_idx", + "columns": [ + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"continuation_policy\" = 'wake_origin_agent' AND \"decisions\".\"metadata\" ->> 'continuationPending' = 'true' AND (\"decisions\".\"status\" IN ('expired', 'cancelled') OR (\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" IN ('succeeded', 'partial', 'failed')))", + "concurrently": false, + "method": "btree", + "with": {} + }, "decisions_bundle_idx": { "name": "decisions_bundle_idx", "columns": [ @@ -32047,6 +32129,28 @@ "method": "btree", "with": {} }, + "grok_dispatch_prepared_idx": { + "name": "grok_dispatch_prepared_idx", + "columns": [ + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"grok_bot_dispatches\".\"status\" = 'prepared'", + "concurrently": false, + "method": "btree", + "with": {} + }, "grok_dispatch_company_status_idx": { "name": "grok_dispatch_company_status_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index c75fc030ee5..8536f384cb0 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2731,6 +2731,13 @@ "when": 1793142971791, "tag": "0391_third_roulette", "breakpoints": true + }, + { + "idx": 392, + "version": "7", + "when": 1793143372644, + "tag": "0392_small_frightful_four", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/approvals.ts b/packages/db/src/schema/approvals.ts index f4cd512c1fa..052e37b03a4 100644 --- a/packages/db/src/schema/approvals.ts +++ b/packages/db/src/schema/approvals.ts @@ -1,3 +1,4 @@ +import { sql } from "drizzle-orm"; import { pgTable, uuid, text, timestamp, jsonb, index } from "drizzle-orm/pg-core"; import { companies } from "./companies.js"; import { agents } from "./agents.js"; @@ -19,6 +20,7 @@ export const approvals = pgTable( updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), }, (table) => ({ + pendingRetentionIdx: index("approvals_pending_retention_idx").on(table.updatedAt, table.id).where(sql`${table.status} = 'pending'`), companyStatusTypeIdx: index("approvals_company_status_type_idx").on( table.companyId, table.status, diff --git a/server/src/__tests__/approval-retention-work.test.ts b/server/src/__tests__/approval-retention-work.test.ts new file mode 100644 index 00000000000..9315e401089 --- /dev/null +++ b/server/src/__tests__/approval-retention-work.test.ts @@ -0,0 +1,423 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, agents, approvals, chatConversations, chatEndpoints, companies, createDb, subscribeDatabaseWork, decisionArchiveNotificationOutbox, decisionRetention, issueApprovals, issues, toolApplications, toolConnections, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { approvalRecords } from "../services/approvals.js"; +import { approvalRetentionService, registerApprovalRetentionWork } from "../services/approval-retention.js"; +import { decisionRetentionService, DEFAULT_DECISION_ARCHIVE_DAYS } from "../services/decision-retention.js"; +import { attentionService } from "../services/attention.js"; +import { issueApprovalService } from "../services/issue-approvals.js"; +import { companyService } from "../services/companies.js"; +import { issueService } from "../services/issues.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_APPROVAL_RETENTION_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; +(support.supported ? describe : describe.skip)("approval retention worker", () => { + let temporary: Awaited> | undefined; + let db: ReturnType, companyId: string; + let coordinator: ReturnType | undefined; + let draining: boolean; + const errors: unknown[] = []; + beforeAll(async () => { + if (!externalUrl) temporary = await startEmbeddedPostgresTestDatabase("approval-retention-"); + db = createDb(externalUrl ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 30_000); + beforeEach(async () => { + companyId = randomUUID(); draining = false; errors.length = 0; + await db.insert(companies).values({ id: companyId, name: "Approval retention", issuePrefix: `A${companyId.slice(0, 6)}` }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.execute(sql`drop trigger if exists approval_retention_fail on decision_retention`); + await db.execute(sql`drop function if exists approval_retention_fail()`); + await db.delete(decisionArchiveNotificationOutbox); await db.delete(decisionRetention); + await db.delete(activityLog); await db.delete(issueApprovals); await db.delete(approvals); + await db.delete(chatConversations); await db.delete(chatEndpoints); + await db.delete(toolConnections); await db.delete(toolApplications); + await db.delete(issues); await db.delete(agents); await db.delete(companies); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + const create = (at = new Date(), database: Db = db) => approvalRecords(database).create(companyId, { + type: "request_board_approval", payload: {}, createdAt: at, updatedAt: at, + }); + const old = () => new Date(Date.now() - delay - 60_000); + const state = (id: string) => decisionRetentionService(db).getState(companyId, "approval", id); + const authActor = () => ({ type: "board" as const, source: "local_implicit" as const, userId: "board-user", companyIds: [companyId], isInstanceAdmin: true }); + const actor = { actorType: "user" as const, actorId: "board-user", agentId: null, userId: "board-user", runId: null, agentApiKeyId: null, responsibleUserId: "board-user" }; + const mutation = (id: string) => ({ companyId, sourceKind: "approval" as const, sourceId: id, authActor: authActor(), actor }); + async function start(service = approvalRetentionService(db)) { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => errors.push(error) }); + await registerApprovalRetentionWork(coordinator, service, () => !draining).ready; + return service; + } + async function settle() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + } + async function quiet() { + await settle(); + expect(coordinator!.nextWakeAt()).toBeNull(); + const reads = vi.spyOn(db, "select"), executes = vi.spyOn(db, "execute"), writes = vi.spyOn(db, "transaction"); + await vi.advanceTimersByTimeAsync(300_000); + expect(reads).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); executes.mockRestore(); writes.mockRestore(); + } + async function archived(id: string) { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(30_000); + expect((await state(id))?.archivedAt).toBeInstanceOf(Date); + }, { timeout: 5000 }); + } + async function issue(hiddenAt: Date | null = null) { + const [row] = await db.insert(issues).values({ companyId, title: "Approval target", status: "todo", hiddenAt }).returning(); + return row; + } + + it("is quiet with no approvals and archives unseen durable approvals at their actual deadline", async () => { + await start(); await quiet(); + const deadline = Date.now() + 600_000; + const row = await create(new Date(deadline - delay)); + await settle(); + expect(await state(row.id)).toBeNull(); + expect(coordinator!.nextWakeAt()).toBe(deadline); + const quietWindow = deadline - Date.now() - 30_000; + expect(quietWindow).toBeGreaterThan(0); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(quietWindow); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await vi.advanceTimersByTimeAsync(60_000); + await archived(row.id); await quiet(); + expect((await db.select().from(approvals).where(eq(approvals.id, row.id)))[0].status).toBe("pending"); + }); + + it.each(["approved", "rejected", "cancelled"] as const)("wakes parked tool reviews only on terminal %s transition", async outcome => { + const queues = [DELIVERY_QUEUES.toolReviewExpiry, DELIVERY_QUEUES.toolReviewRecovery, + DELIVERY_QUEUES.toolReviewInterruption, DELIVERY_QUEUES.toolReviewProjection]; + const events = queues.map(() => [] as string[]); + const subscriptions = queues.map((queue, index) => subscribeDatabaseWork(db, queue, event => { events[index].push(event); })); + try { + const row = await create(), target = await issue(); + for (let pass = 0; pass < 3; pass++) { + await issueApprovalService(db).link(target.id, row.id); + await issueApprovalService(db).unlink(target.id, row.id); + } + await approvalRecords(db).requestRevision(row.id, "board-user"); + await approvalRecords(db).resubmit(row.id); + expect(events).toEqual(queues.map(() => [])); + if (outcome === "cancelled") await approvalRecords(db).cancel(row.id, "cancelled by user"); + else await approvalRecords(db).resolveApproval(row.id, outcome, "board-user", null); + expect(events).toEqual(queues.map(() => ["intent", "settled"])); + } finally { for (const unsubscribe of subscriptions) unsubscribe(); } + }); + + it("bounds aggregate retention reads to one reconciliation per thirty seconds of sequential tenant writes", async () => { + const [otherCompany] = await db.insert(companies).values({ name: "Other tenant", issuePrefix: `B${companyId.slice(0, 6)}` }).returning(); + const other = await approvalRecords(db).create(otherCompany.id, { type: "request_board_approval", payload: {} }); + const service = await start(); await settle(); + const sweep = vi.spyOn(service, "sweep"), reads = vi.spyOn(db, "select"); + await create(); + await vi.waitFor(() => expect(sweep).toHaveBeenCalledTimes(1)); + await sweep.mock.results[0].value; + await vi.waitFor(() => expect(idleWorkSnapshot().active).toBe(0)); + const afterFirst = reads.mock.calls.length; + expect(afterFirst).toBe(2); + for (let pass = 0; pass < 3; pass++) { + await create(); + await vi.advanceTimersByTimeAsync(100); + } + await vi.advanceTimersByTimeAsync(29_000); + expect(reads).toHaveBeenCalledTimes(afterFirst); + await vi.advanceTimersByTimeAsync(2000); + await vi.waitFor(() => expect(reads).toHaveBeenCalledTimes(afterFirst + 2)); + reads.mockRestore(); + expect((await db.select().from(approvals).where(eq(approvals.id, other.id)))[0].status).toBe("pending"); + }); + + it("recovers old unseen approvals on startup and leaves completed history quiet", async () => { + const due = await create(old()), completed = await create(old()); + await approvalRecords(db).resolveApproval(completed.id, "approved", "board-user", null); + await start(); await archived(due.id); await quiet(); + expect(await state(completed.id)).toBeNull(); + }); + + it("disarms on revision and moves the deadline when resubmitted", async () => { + await start(); const row = await create(); await settle(); + await approvalRecords(db).requestRevision(row.id, "board-user"); await quiet(); + const resubmitted = await approvalRecords(db).resubmit(row.id); await settle(); + expect(coordinator!.nextWakeAt()).toBe(resubmitted.updatedAt.getTime() + delay); + await approvalRecords(db).cancel(row.id); await quiet(); + }); + + it.each(["keep", "archive", "revive"] as const)("applies %s to an unseen young approval without a feed read", async action => { + const row = await create(); + expect(await state(row.id)).toBeNull(); + const retention = decisionRetentionService(db); + const result = action === "keep" ? await retention.setKeep({ ...mutation(row.id), keep: true }) + : action === "archive" ? await retention.archive(mutation(row.id)) : await retention.revive(mutation(row.id)); + expect(result).toMatchObject({ sourceKind: "approval", sourceId: row.id, sourceActivityAt: row.updatedAt, keep: action === "keep" }); + expect(result.archivedAt).toEqual(action === "archive" ? expect.any(Date) : null); + expect(await state(row.id)).toMatchObject({ id: result.id, keep: action === "keep" }); + }); + + it("rolls back on-demand materialization when the manual keep write fails", async () => { + const row = await create(); + await db.execute(sql`create function approval_retention_fail() returns trigger language plpgsql as $$ begin + if NEW.keep then raise exception 'manual keep unavailable'; end if; return NEW; end $$`); + await db.execute(sql`create trigger approval_retention_fail before update on decision_retention for each row execute function approval_retention_fail()`); + await expect(decisionRetentionService(db).setKeep({ ...mutation(row.id), keep: true })) + .rejects.toMatchObject({ cause: { message: "manual keep unavailable" } }); + expect(await state(row.id)).toBeNull(); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision_retention.keep_updated"))).toHaveLength(0); + await db.execute(sql`drop trigger approval_retention_fail on decision_retention`); + expect(await decisionRetentionService(db).setKeep({ ...mutation(row.id), keep: true })).toMatchObject({ keep: true }); + expect(await db.select().from(activityLog).where(eq(activityLog.action, "decision_retention.keep_updated"))).toHaveLength(1); + }); + + it("honors keep and wakes when an old approval is unkept", async () => { + const row = await create(old()); + await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + await decisionRetentionService(db).setKeep({ ...mutation(row.id), keep: true }); + await start(); await quiet(); expect((await state(row.id))?.archivedAt).toBeNull(); + await decisionRetentionService(db).setKeep({ ...mutation(row.id), keep: false }); + await archived(row.id); await quiet(); + }); + + it("disarms on manual archive and reactivates on revive", async () => { + const row = await create(old()); + await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + await decisionRetentionService(db).archive(mutation(row.id)); + await start(); await quiet(); + await decisionRetentionService(db).revive(mutation(row.id)); + await archived(row.id); await quiet(); + expect((await state(row.id))?.archivedReason).toBe("idle_ttl"); + }); + + it("waits for inactive companies and wakes on company activation", async () => { + const row = await create(old()); + await companyService(db).update(companyId, { status: "paused" }); + await start(); await quiet(); expect(await state(row.id)).toBeNull(); + await companyService(db).update(companyId, { status: "active" }); + await archived(row.id); await quiet(); + }); + + it("excludes hidden linked targets and wakes after unlink", async () => { + const row = await create(old()), target = await issue(new Date()); + await issueApprovalService(db).link(target.id, row.id); + await start(); await quiet(); expect(await state(row.id)).toBeNull(); + await issueApprovalService(db).unlink(target.id, row.id); + await archived(row.id); await quiet(); + }); + + it("wakes when the linked target becomes visible", async () => { + const row = await create(old()), target = await issue(new Date()); + await issueApprovalService(db).link(target.id, row.id); + await start(); await quiet(); + await issueService(db).update(target.id, { hiddenAt: null }); + await archived(row.id); await quiet(); + }); + + it("wakes after deletion cascades the excluded target link", async () => { + const row = await create(old()), target = await issue(new Date()); + await issueApprovalService(db).link(target.id, row.id); + await start(); await quiet(); + await issueService(db).remove(target.id); + await archived(row.id); await quiet(); + }); + + it("preserves normal attention reads while excluding approval retention maintenance", async () => { + const row = await create(old()); + const excluded = await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true, excludeSources: ["approval"] }); + expect(excluded.items.some(item => item.sourceKind === "approval")).toBe(false); + expect(await state(row.id)).toBeNull(); + const visible = await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + expect(visible.items.some(item => item.subject.id === row.id)).toBe(true); + expect(await state(row.id)).not.toBeNull(); + }); + + it("does not publish work before an outer transaction commits", async () => { + await start(); await quiet(); + let id = ""; + await db.transaction(async tx => { + id = (await create(old(), tx as unknown as Db)).id; + await vi.advanceTimersByTimeAsync(60_000); + expect(await state(id)).toBeNull(); + }); + await archived(id); await quiet(); + }); + + it("discharges rolled back nested producers without archiving", async () => { + await start(); await quiet(); + await expect(db.transaction(async tx => { + await create(old(), tx as unknown as Db); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect(await db.select().from(approvals)).toEqual([]); + }); + + it("retains the first producer during drain and processes it after resume", async () => { + await start(); await quiet(); draining = true; + const row = await create(old()); + await vi.advanceTimersByTimeAsync(90_000); expect(await state(row.id)).toBeNull(); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(90_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; await archived(row.id); await quiet(); + }); + + it("rechecks current source activity after waiting on an approval writer", async () => { + const row = await create(old()); + let unlock!: () => void, locked!: () => void; + const barrier = new Promise(resolve => { unlock = resolve; }); + const admitted = new Promise(resolve => { locked = resolve; }); + const write = db.transaction(async tx => { + await tx.execute(sql`select id from approvals where id = ${row.id} for update`); + locked(); await barrier; + await approvalRecords(tx as unknown as Db).requestRevision(row.id, "board-user"); + await approvalRecords(tx as unknown as Db).resubmit(row.id); + }); + await admitted; + const sweep = approvalRetentionService(db).sweep(); + await vi.advanceTimersByTimeAsync(100); + unlock(); await write; await sweep; + expect(await state(row.id)).toBeNull(); + }); + + async function failArchives() { + await db.execute(sql`create function approval_retention_fail() returns trigger language plpgsql as $$ + begin + if new.archived_at is not null and exists ( + select 1 from approvals where id::text = new.source_id and payload->>'retentionTestFailure' = 'true' + ) then raise exception 'injected approval archive failure'; end if; + return new; + end $$`); + await db.execute(sql`create trigger approval_retention_fail before update on decision_retention + for each row execute function approval_retention_fail()`); + } + + it("retains failed durable work across drain and resumes recovery", async () => { + const row = await approvalRecords(db).create(companyId, { type: "request_board_approval", + payload: { retentionTestFailure: true }, updatedAt: old() }); + await failArchives(); + await start(); + expect(errors.length).toBeGreaterThan(0); errors.length = 0; + draining = true; + await vi.advanceTimersByTimeAsync(60_000); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await db.execute(sql`drop trigger approval_retention_fail on decision_retention`); + draining = false; await archived(row.id); await quiet(); + }); + + it("advances bounded pages past a failed row without losing its retry", async () => { + const at = old(); + await db.insert(approvals).values(Array.from({ length: 101 }, (_, i) => ({ companyId, + type: "request_board_approval", payload: { retentionTestFailure: i === 0 }, + createdAt: at, updatedAt: new Date(at.getTime() + i), + }))); + await failArchives(); + const service = approvalRetentionService(db); + await expect(service.sweep()).rejects.toThrow("Failed query"); + expect(await service.sweep()).toEqual({ archived: 1 }); + expect((await db.select().from(decisionRetention)).filter(row => row.archivedAt)).toHaveLength(100); + expect(await service.next()).not.toBeNull(); + await db.execute(sql`drop trigger approval_retention_fail on decision_retention`); + expect(await service.sweep()).toEqual({ archived: 1 }); expect(await service.next()).toBeNull(); + }); + + it("reconstructs startup obligations while paused and performs no passive SQL", async () => { + const row = await create(old()); draining = true; + await start(); + const reads = vi.spyOn(db, "select"); + await vi.advanceTimersByTimeAsync(120_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + expect(await state(row.id)).toBeNull(); + draining = false; await archived(row.id); await quiet(); + }); + + it("matches the feed's first linked issue rather than any visible link", async () => { + const row = await create(old()), one = await issue(), two = await issue(); + const [first, second] = [one, two].sort((a, b) => a.id.localeCompare(b.id)); + await db.update(issues).set({ hiddenAt: new Date() }).where(eq(issues.id, first.id)); + await issueApprovalService(db).linkManyForApproval(row.id, [second.id, first.id]); + expect((await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true })).items + .some(item => item.subject.id === row.id)).toBe(false); + await start(); await quiet(); + await issueApprovalService(db).unlink(first.id, row.id); + await archived(row.id); await quiet(); + }); + + it("keeps a visible first linked issue eligible when a later link is hidden", async () => { + const row = await create(old()), one = await issue(), two = await issue(); + const [first, second] = [one, two].sort((a, b) => a.id.localeCompare(b.id)); + await db.update(issues).set({ hiddenAt: new Date() }).where(eq(issues.id, second.id)); + await issueApprovalService(db).linkManyForApproval(row.id, [second.id, first.id]); + expect((await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true })).items + .some(item => item.subject.id === row.id)).toBe(true); + await start(); await archived(row.id); await quiet(); + }); + + it("joins an admitted sweep on shutdown and skips subsequent deadline queries", async () => { + const row = await create(old()); + const service = approvalRetentionService(db), original = service.sweep; + let release!: () => void, admitted!: () => void; + const barrier = new Promise(resolve => { release = resolve; }); + const admission = new Promise(resolve => { admitted = resolve; }); + vi.spyOn(service, "sweep").mockImplementation(async (now, signal) => { + admitted(); await barrier; return original(now, signal); + }); + const next = vi.spyOn(service, "next"); + const starting = start(service); await admission; + let stopped = false; + const stopping = coordinator!.stop().then(() => { stopped = true; }); + await vi.advanceTimersByTimeAsync(100); expect(stopped).toBe(false); + release(); await starting; await stopping; coordinator = undefined; + expect(next).not.toHaveBeenCalled(); expect(await state(row.id)).toBeNull(); + await start(); await archived(row.id); await quiet(); + }); + + + it("uses the shared feed eligibility for a waiting Slack conversation with a pending approval", async () => { + const [agent] = await db.insert(agents).values({ companyId, name: "Origin", role: "engineer", status: "active", adapterType: "codex_local" }).returning(); + const [app] = await db.insert(toolApplications).values({ companyId, name: "Slack", type: "chat" }).returning(); + const [connection] = await db.insert(toolConnections).values({ companyId, applicationId: app.id, name: "Slack", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel" }).returning(); + const [endpoint] = await db.insert(chatEndpoints).values({ companyId, connectionId: connection.id, provider: "slack", publicId: randomUUID(), assignedAgentId: agent.id, status: "active" }).returning(); + const target = await issue(); + await db.update(issues).set({ status: "in_review", assigneeAgentId: agent.id }).where(eq(issues.id, target.id)); + await db.insert(chatConversations).values({ companyId, endpointId: endpoint.id, issueId: target.id, externalConversationId: randomUUID(), externalLabel: "Waiting", state: "waiting" }); + const row = await approvalRecords(db).create(companyId, { type: "request_board_approval", payload: {}, requestedByAgentId: agent.id, updatedAt: old() }); + await issueApprovalService(db).link(target.id, row.id); + const feed = await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + expect(feed.items.some(item => item.subject.id === row.id)).toBe(true); + await start(); await archived(row.id); await quiet(); + expect(await db.select().from(decisionArchiveNotificationOutbox)).toMatchObject([ + { sourceId: row.id, originIssueId: target.id, originAgentId: agent.id, status: "pending" }, + ]); + }); + + + it("advances past a failed page with full database timestamp precision", async () => { + await db.execute(sql`insert into approvals(company_id, type, payload, updated_at) + select ${companyId}, 'request_board_approval', jsonb_build_object('retentionTestFailure', n <= 100), + date_trunc('milliseconds', now() - interval '91 days') + interval '456 microseconds' + + case when n <= 100 then interval '0 seconds' else interval '1 second' end + from generate_series(1, 101) n`); + await failArchives(); + const service = approvalRetentionService(db); + await expect(service.sweep()).rejects.toThrow("Approval retention failed"); + expect(await service.sweep()).toEqual({ archived: 1 }); + await db.execute(sql`drop trigger approval_retention_fail on decision_retention`); + expect(await service.sweep()).toEqual({ archived: 100 }); + expect(await service.sweep()).toEqual({ archived: 0 }); + expect(await service.next()).toBeNull(); + }); + +}); diff --git a/server/src/__tests__/approvals-service.test.ts b/server/src/__tests__/approvals-service.test.ts index 31755c1f65c..4049b3b0fe0 100644 --- a/server/src/__tests__/approvals-service.test.ts +++ b/server/src/__tests__/approvals-service.test.ts @@ -60,14 +60,12 @@ function createDbStub(selectResults: ApprovalRecord[][], updateResults: Approval const set = vi.fn(() => ({ where: updateWhere })); const update = vi.fn(() => ({ set })); - return { - db: { select, update, transaction: vi.fn(async (callback: (db: unknown) => Promise) => { - inTransaction = true; - try { return await callback({ select, update }); } finally { inTransaction = false; } - }) }, - selectWhere, - returning, - }; + const db = { select, update, transaction: vi.fn(async (callback: (db: unknown) => Promise) => { + const outerTransaction = inTransaction; + inTransaction = true; + try { return await callback(db); } finally { inTransaction = outerTransaction; } + }) }; + return { db, selectWhere, returning }; } describe("approvalService resolution idempotency", () => { diff --git a/server/src/__tests__/attention-service.test.ts b/server/src/__tests__/attention-service.test.ts index bf4879957cd..cf2001e426b 100644 --- a/server/src/__tests__/attention-service.test.ts +++ b/server/src/__tests__/attention-service.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { createAgentLifecycle } from "../services/agent-lifecycle.js"; import { randomUUID } from "node:crypto"; import express from "express"; @@ -48,7 +49,8 @@ import { listAttentionExhaustedRuns } from "../services/attention-exhausted-runs import { agentService } from "../services/agents.js"; import { ROUTABLE_BLOCKED_ROLLOUT_AT } from "../services/routable-blocked.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ATTENTION_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -62,8 +64,8 @@ describeEmbeddedPostgres("attention service", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-attention-service-"); - db = createDb(tempDb.connectionString); + if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-attention-service-"); + db = createDb(externalUrl ?? tempDb!.connectionString); }, 30_000); afterEach(async () => { @@ -99,6 +101,7 @@ describeEmbeddedPostgres("attention service", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/__tests__/budgets-service.test.ts b/server/src/__tests__/budgets-service.test.ts index 1dbf0631a90..0ea5a94dfc6 100644 --- a/server/src/__tests__/budgets-service.test.ts +++ b/server/src/__tests__/budgets-service.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import { eq } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; @@ -26,7 +27,8 @@ vi.mock("../services/activity-log.js", () => ({ // Budget behavior is exercised against PostgreSQL below and in // cost-accounting-reliability.test.ts, including transaction and retry paths. -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_BUDGETS_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; describeEmbeddedPostgres("budgetService release gate enforcement", () => { @@ -34,8 +36,9 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-budgets-service-"); - db = createDb(tempDb.connectionString); + await import("../services/agent-lifecycle.js"); + if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-budgets-service-"); + db = createDb(externalUrl ?? tempDb!.connectionString); }, 20_000); afterEach(async () => { @@ -50,6 +53,7 @@ describeEmbeddedPostgres("budgetService release gate enforcement", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index 12f46927634..e1d793142a1 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,3 +1,4 @@ +import { approvalRetentionService } from "../services/approval-retention.js"; import { createSecretProposalsService } from "../services/secret-proposals.js"; import { logger } from "../middleware/logger.js"; import { getStartupRecoveryState } from "../startup-recovery-state.js"; @@ -330,6 +331,11 @@ vi.mock("../realtime/live-events-ws.js", () => ({ setupLiveEventsWebSocketServer: vi.fn(), })); +vi.mock("../services/approval-retention.js", async importOriginal => ({ + ...await importOriginal(), + approvalRetentionService: vi.fn(() => ({ sweep: vi.fn(async () => ({ archived: 0 })), next: vi.fn(async () => null) })), +})); + vi.mock("../services/index.js", () => ({ backfillLegacyToolOAuthTokens: vi.fn(async () => ({ scannedConnections: 0, @@ -530,6 +536,18 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true])("registers approval retention only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.approvalRetention); + expect(Boolean(registration)).toBe(enabled); + if (enabled) { + const service = vi.mocked(approvalRetentionService).mock.results.at(-1)!.value; + await registration![1].run(new AbortController().signal); + expect(service.sweep).toHaveBeenCalled(); expect(service.next).toHaveBeenCalled(); + } + }); + it.each([false, true])("registers archive delivery separately from retention sweeps (scheduler %s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled, heartbeatSchedulerIntervalMs: 30000 })); let tick: (() => void) | undefined; diff --git a/server/src/index.ts b/server/src/index.ts index 4ea740f7113..d74ea2b848f 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { approvalRetentionService, registerApprovalRetentionWork } from "./services/approval-retention.js"; import { registerDecisionTargetWork } from "./services/decision-target-work.js"; import { registerDecisionEffectRecoveryWork } from "./services/decision-effect-recovery-work.js"; import { registerDecisionExpiryWork } from "./services/decision-expiry-work.js"; @@ -1811,6 +1812,7 @@ async function startServerWithDatabaseTeardown( // per company per tick. const page = await attentionService(db as any).list(company.id, { includeDismissed: true, + excludeSources: ["approval"], all: true, allowUnscopedAll: true, }); @@ -1858,6 +1860,8 @@ async function startServerWithDatabaseTeardown( await step("environment_lease_cleanup", startEnvironmentCleanup); await step("archive_notification_worker", () => registerDecisionArchiveDeliveryWork(deliveryWork, retentionExecutor, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("approval_retention", () => registerApprovalRetentionWork(deliveryWork, approvalRetentionService(db as any), + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("retention_sweep", runRetentionSweep); }; diff --git a/server/src/lib/approval-records.ts b/server/src/lib/approval-records.ts index 957dfac03ec..7bd14df166a 100644 --- a/server/src/lib/approval-records.ts +++ b/server/src/lib/approval-records.ts @@ -1,3 +1,5 @@ +import { notifyToolReviewChange } from "../services/tool-review-work-notifications.js"; +import { writeApprovalChange } from "../services/retention-work-notifications.js"; import { and, eq, inArray } from "drizzle-orm"; import { approvals, type Db } from "@paperclipai/db"; import { notFound, unprocessable } from "../errors.js"; @@ -37,8 +39,9 @@ export function approvalResolutionRecords(db: Db) { } const now = new Date(); - const updated = await database - .update(approvals) + const updated = await writeApprovalChange(database, async tx => { + await notifyToolReviewChange(tx, existing.companyId); + return tx.update(approvals) .set({ status: targetStatus, decidedByUserId, @@ -49,6 +52,7 @@ export function approvalResolutionRecords(db: Db) { .where(and(eq(approvals.id, id), inArray(approvals.status, resolvableStatuses))) .returning() .then((rows) => rows[0] ?? null); + }); if (updated) { return { approval: updated, applied: true }; diff --git a/server/src/services/approval-retention.ts b/server/src/services/approval-retention.ts new file mode 100644 index 00000000000..855c81e413d --- /dev/null +++ b/server/src/services/approval-retention.ts @@ -0,0 +1,67 @@ +import { approvalExecutionIssueCondition } from "./approval-visibility.js"; +import { and, asc, eq, isNull, lte, or, sql, type SQL } from "drizzle-orm"; +import { approvals, companies, decisionRetention, type Db } from "@paperclipai/db"; +import { decisionRetentionService, DEFAULT_DECISION_ARCHIVE_DAYS } from "./decision-retention.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; + +const archiveDelay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; + +export function approvalRetentionService(db: Db) { + let cursor: { at: string; id: string } | undefined; + function candidates(database: Db, condition?: SQL) { + return database.select({ id: approvals.id, companyId: approvals.companyId, at: approvals.updatedAt, cursorAt: sql`${approvals.updatedAt}::text` }).from(approvals) + .innerJoin(companies, eq(companies.id, approvals.companyId)) + .leftJoin(decisionRetention, and(eq(decisionRetention.companyId, approvals.companyId), + eq(decisionRetention.sourceKind, "approval"), sql`${decisionRetention.sourceId} = ${approvals.id}::text`)) + .where(and(sql`${approvals.status} = 'pending'`, eq(companies.status, "active"), approvalExecutionIssueCondition(), + or(isNull(decisionRetention.id), eq(decisionRetention.keep, false)), isNull(decisionRetention.archivedAt), condition)) + .orderBy(asc(approvals.updatedAt), asc(approvals.id)); + } + async function sweep(now = new Date(), signal?: AbortSignal) { + const cutoff = new Date(now.getTime() - archiveDelay); + const rows = await candidates(db, and(lte(approvals.updatedAt, cutoff), cursor + ? sql`(${approvals.updatedAt}, ${approvals.id}) > (${cursor.at}::timestamptz, ${cursor.id}::uuid)` : undefined)).limit(100); + let archived = 0; + const errors: unknown[] = []; + for (const candidate of rows) { + if (signal?.aborted) break; + try { + archived += await db.transaction(async tx => { + const database = tx as unknown as Db; + const [current] = await candidates(database, and(eq(approvals.id, candidate.id), lte(approvals.updatedAt, cutoff))) + .for("update", { of: approvals }); + if (!current) return 0; + const retention = decisionRetentionService(database); + const source = { sourceKind: "approval" as const, subject: { id: current.id }, activityAt: current.at.toISOString() }; + const states = await retention.syncItems(current.companyId, [source]); + const state = states.get(`approval:${current.id}`)!; + return retention.autoArchive({ companyId: current.companyId, now, items: [{ ...source, + keep: state.keep, archivedAt: state.archivedAt?.toISOString() ?? null, retentionVersion: state.version, + }] }); + }); + } catch (error) { errors.push(error); } + cursor = { at: candidate.cursorAt, id: candidate.id }; + } + if (rows.length < 100) cursor = undefined; + if (errors.length === 1) throw errors[0]; + if (errors.length) throw new AggregateError(errors, "Approval retention failed"); + return { archived }; + } + async function next() { + const [row] = await candidates(db).limit(1); + return row ? row.at.getTime() + archiveDelay : null; + } + return { sweep, next }; +} + +export function registerApprovalRetentionWork( + coordinator: ReturnType, + service: ReturnType, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.approvalRetention, { + run: signal => service.sweep(new Date(), signal), next: service.next, canRun, retryMs: 30_000, notificationCoalesceMs: 30_000, + }); +} diff --git a/server/src/services/approval-visibility.ts b/server/src/services/approval-visibility.ts new file mode 100644 index 00000000000..3ee21cf174c --- /dev/null +++ b/server/src/services/approval-visibility.ts @@ -0,0 +1,18 @@ +import { sql, type SQL } from "drizzle-orm"; +import { approvals, issues } from "@paperclipai/db"; +import { executionIssueCondition } from "./issue-visibility.js"; + +export function firstApprovalIssueId(): SQL { + return sql`(select first_link.issue_id from issue_approvals first_link + where first_link.company_id = ${approvals.companyId} and first_link.approval_id = ${approvals.id} + order by first_link.issue_id limit 1)`; +} + +export function approvalExecutionIssueCondition(): SQL { + return sql`not exists ( + select 1 from issue_approvals linked where linked.company_id = ${approvals.companyId} + and linked.approval_id = ${approvals.id} and linked.issue_id = ${firstApprovalIssueId()} + and not exists (select 1 from ${issues} where ${issues.id} = linked.issue_id + and ${issues.companyId} = ${approvals.companyId} and ${executionIssueCondition()}) + )`; +} diff --git a/server/src/services/approvals.ts b/server/src/services/approvals.ts index 4f4149f773d..1095766c3ef 100644 --- a/server/src/services/approvals.ts +++ b/server/src/services/approvals.ts @@ -1,4 +1,5 @@ -import { writeToolReviewChange } from "./tool-review-work-notifications.js"; +import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; +import { writeApprovalChange } from "./retention-work-notifications.js"; import { approvalResolutionRecords, resolvableApprovalStatuses as resolvableStatuses } from "../lib/approval-records.js"; import { notifyHireApproved } from "./hire-hook.js"; @@ -13,15 +14,7 @@ import { instanceSettingsService } from "./instance-settings.js"; export function approvalRecords(db: Db) { const instanceSettings = instanceSettingsService(db); - const { getExistingApproval, resolveApproval: resolveApprovalRecord } = approvalResolutionRecords(db); - - // Tool reviews waiting for a formal decision stay parked until this producer - // commits. The shared record owner retains approval validation and CAS rules. - async function resolveApproval(id: string, status: "approved" | "rejected", userId: string, - note: string | null | undefined, database: Db = db) { - const existing = await getExistingApproval(id, database); - return writeToolReviewChange(database, existing.companyId, tx => resolveApprovalRecord(id, status, userId, note, tx)); - } + const { getExistingApproval, resolveApproval } = approvalResolutionRecords(db); function redactApprovalComment(comment: T, censorUsernameInLogs: boolean): T { return { @@ -62,11 +55,11 @@ export function approvalRecords(db: Db) { }, create: (companyId: string, data: Omit) => - db + writeApprovalChange(db, tx => tx .insert(approvals) .values({ ...data, companyId }) .returning() - .then((rows) => rows[0]), + .then((rows) => rows[0])), // Cancel an open (pending/revision_requested) approval without a board // decision — e.g. when its paired agent is terminated during duplicate @@ -75,8 +68,9 @@ export function approvalRecords(db: Db) { const now = new Date(); const [existing] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, id)); if (!existing) return null; - const updated = await writeToolReviewChange(db, existing.companyId, tx => tx - .update(approvals) + const updated = await writeApprovalChange(db, async tx => { + await notifyToolReviewChange(tx, existing.companyId); + return tx.update(approvals) .set({ status: "cancelled", decisionNote: reason ?? null, @@ -85,7 +79,8 @@ export function approvalRecords(db: Db) { }) .where(and(eq(approvals.id, id), inArray(approvals.status, resolvableStatuses))) .returning() - .then((rows) => rows[0] ?? null)); + .then((rows) => rows[0] ?? null); + }); return updated; }, @@ -96,7 +91,7 @@ export function approvalRecords(db: Db) { } const now = new Date(); - return db + return writeApprovalChange(db, tx => tx .update(approvals) .set({ status: "revision_requested", @@ -107,7 +102,7 @@ export function approvalRecords(db: Db) { }) .where(eq(approvals.id, id)) .returning() - .then((rows) => rows[0]); + .then((rows) => rows[0])); }, resubmit: async (id: string, payload?: Record) => { @@ -117,7 +112,7 @@ export function approvalRecords(db: Db) { } const now = new Date(); - return db + return writeApprovalChange(db, tx => tx .update(approvals) .set({ status: "pending", @@ -129,7 +124,7 @@ export function approvalRecords(db: Db) { }) .where(eq(approvals.id, id)) .returning() - .then((rows) => rows[0]); + .then((rows) => rows[0])); }, listComments: async (approvalId: string) => { diff --git a/server/src/services/attention.ts b/server/src/services/attention.ts index 90ed24f135e..1e94cd26ec3 100644 --- a/server/src/services/attention.ts +++ b/server/src/services/attention.ts @@ -1,3 +1,4 @@ +import { approvalExecutionIssueCondition, firstApprovalIssueId } from "./approval-visibility.js"; import { and, asc, desc, eq, gt, inArray, isNotNull, isNull, notInArray, sql, type SQL } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { @@ -161,6 +162,7 @@ type AttentionListOptions = AttentionFeedQuery & { actor?: AuthorizationActor; /** Internal-only escape hatch for callers that need one stable, unpaginated feed snapshot. */ allowUnscopedAll?: boolean; + excludeSources?: readonly AttentionSourceKind[]; }; type AttentionServiceOptions = { @@ -1185,83 +1187,72 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions collected.push({ ...item, dismissal }); }; - const pendingApprovals = await db - .select({ - id: approvals.id, - type: approvals.type, - status: approvals.status, - requestedByAgentId: approvals.requestedByAgentId, - requestedByUserId: approvals.requestedByUserId, - payload: approvals.payload, - createdAt: approvals.createdAt, - updatedAt: approvals.updatedAt, - }) - .from(approvals) - .where(and(eq(approvals.companyId, companyId), eq(approvals.status, "pending"))) - .orderBy(desc(approvals.updatedAt), desc(approvals.id)); + if (!options.excludeSources?.includes("approval")) { + const pendingApprovals = await db + .select({ + id: approvals.id, + linkedIssueId: firstApprovalIssueId(), + type: approvals.type, + status: approvals.status, + requestedByAgentId: approvals.requestedByAgentId, + requestedByUserId: approvals.requestedByUserId, + payload: approvals.payload, + createdAt: approvals.createdAt, + updatedAt: approvals.updatedAt, + }) + .from(approvals) + .where(and(eq(approvals.companyId, companyId), eq(approvals.status, "pending"), approvalExecutionIssueCondition())) + .orderBy(desc(approvals.updatedAt), desc(approvals.id)); - const pendingApprovalIds = pendingApprovals.map((approval) => approval.id); - const approvalIssueRows = pendingApprovalIds.length > 0 - ? await db - .select({ approvalId: issueApprovals.approvalId, issueId: issueApprovals.issueId }) - .from(issueApprovals) - .where(and( - eq(issueApprovals.companyId, companyId), - inArray(issueApprovals.approvalId, pendingApprovalIds), - )) - .orderBy(asc(issueApprovals.approvalId), asc(issueApprovals.issueId)) - : []; - const approvalIssueMap = new Map(); - for (const row of approvalIssueRows) { - if (!approvalIssueMap.has(row.approvalId)) approvalIssueMap.set(row.approvalId, row.issueId); - } - const readableApprovalIssues = await issueSummaryMap( - db, - companyId, - approvalIssueRows.map((row) => row.issueId), - readCondition, - ); - - for (const approval of pendingApprovals) { - const linkedIssueId = approvalIssueMap.get(approval.id) ?? null; - if (linkedIssueId && !readableApprovalIssues.has(linkedIssueId)) continue; - const dedupKey = `approval:${approval.id}`; - const title = approvalTitle(approval.type, approval.payload); - add(createItem({ + const readableApprovalIssues = await issueSummaryMap( + db, companyId, - sourceKind: "approval", - subject: { - kind: "approval", - id: approval.id, + pendingApprovals.map((row) => row.linkedIssueId), + readCondition, + ); + + for (const approval of pendingApprovals) { + const linkedIssueId = approval.linkedIssueId ?? null; + if (linkedIssueId && !readableApprovalIssues.has(linkedIssueId)) continue; + const dedupKey = `approval:${approval.id}`; + const title = approvalTitle(approval.type, approval.payload); + add(createItem({ companyId, - title, - identifier: null, - status: approval.status, - href: `/${prefix}/approvals/${approval.id}`, - metadata: { - type: approval.type, - requestedByAgentId: approval.requestedByAgentId, - requestedByUserId: approval.requestedByUserId, - issueId: approvalIssueMap.get(approval.id) ?? null, + sourceKind: "approval", + subject: { + kind: "approval", + id: approval.id, + companyId, + title, + identifier: null, + status: approval.status, + href: `/${prefix}/approvals/${approval.id}`, + metadata: { + type: approval.type, + requestedByAgentId: approval.requestedByAgentId, + requestedByUserId: approval.requestedByUserId, + issueId: approval.linkedIssueId ?? null, + }, }, - }, - whyNow: "Approval is pending a board decision.", - decisionVerbs: decisionVerbs( - { id: "approve", label: "Approve", description: "Approve the request." }, - { id: "reject", label: "Reject", description: "Reject the request." }, - { id: "request_revision", label: "Request revision", description: "Send the request back for changes." }, - ), - inlineResolvable: approval.type !== "request_board_approval", - entryRule: "approvals.status = 'pending'", - exitRule: "Approval leaves pending status.", - dedupKey, - severity: "medium", - activityAt: toIso(approval.updatedAt), - createdAt: toIso(approval.createdAt), - updatedAt: toIso(approval.updatedAt), - relatedIssue: null, - detail: approvalDetail(approval.type, approval.payload), - })); + whyNow: "Approval is pending a board decision.", + decisionVerbs: decisionVerbs( + { id: "approve", label: "Approve", description: "Approve the request." }, + { id: "reject", label: "Reject", description: "Reject the request." }, + { id: "request_revision", label: "Request revision", description: "Send the request back for changes." }, + ), + inlineResolvable: approval.type !== "request_board_approval", + entryRule: "approvals.status = 'pending'", + exitRule: "Approval leaves pending status.", + dedupKey, + severity: "medium", + activityAt: toIso(approval.updatedAt), + createdAt: toIso(approval.createdAt), + updatedAt: toIso(approval.updatedAt), + relatedIssue: null, + detail: approvalDetail(approval.type, approval.payload), + })); + } + } const interactionRows = await db diff --git a/server/src/services/budgets.ts b/server/src/services/budgets.ts index bf02e510ca0..84e4ca1e6dc 100644 --- a/server/src/services/budgets.ts +++ b/server/src/services/budgets.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork } from "./retention-work-notifications.js"; import { addCents, compareCents, normalizeCents } from "@paperclipai/shared"; import { and, desc, eq, inArray, sql, or, isNull } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -258,6 +259,7 @@ async function markApprovalStatus( decidedByUserId: string, ) { if (!approvalId) return; + await notifyRetentionWork(db, "approval"); await db .update(approvals) .set({ @@ -295,6 +297,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.paused) return; + await notifyRetentionWork(db, "approval"); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync); await db @@ -351,6 +354,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync); const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.pauseReason !== "budget") return; + await notifyRetentionWork(db, "approval"); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId); await db .update(companies) @@ -453,6 +457,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica windowEnd: end, }); + if (thresholdType === "hard") await notifyRetentionWork(db, "approval"); const approval = thresholdType === "hard" ? await db .insert(approvals) diff --git a/server/src/services/companies.ts b/server/src/services/companies.ts index fa29b357f3b..f6fc7ed5de2 100644 --- a/server/src/services/companies.ts +++ b/server/src/services/companies.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork } from "./retention-work-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { isRunnerApiResponseAsset } from "./asset-content-access.js"; import { deleteCompany } from "./company-deletion.js"; @@ -330,6 +331,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (companyPatch.status !== undefined && companyPatch.status !== existing.status) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyRetentionWork(tx, "approval"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); } const updated = await tx @@ -481,6 +483,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (!wasAlreadyArchived) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); + await notifyRetentionWork(tx, "approval"); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); await tx .update(companies) diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index ab549ecbfb8..c3b3aaf3e13 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork } from "./retention-work-notifications.js"; import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { agentObservabilityService } from "./agent-observability.js"; @@ -96,6 +97,7 @@ export async function deleteCompany(db: Db, id: string) { await tx.delete(agentRuntimeState).where(eq(agentRuntimeState.companyId, id)); await tx.delete(issueComments).where(eq(issueComments.companyId, id)); await tx.delete(approvalComments).where(eq(approvalComments.companyId, id)); + await notifyRetentionWork(tx, "approval"); await tx.delete(approvals).where(eq(approvals.companyId, id)); await tx.delete(companySecrets).where(eq(companySecrets.companyId, id)); await tx.delete(joinRequests).where(eq(joinRequests.companyId, id)); diff --git a/server/src/services/decision-retention.ts b/server/src/services/decision-retention.ts index b65e6e3bb77..7b7a993cce4 100644 --- a/server/src/services/decision-retention.ts +++ b/server/src/services/decision-retention.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork } from "./retention-work-notifications.js"; import { decisionArchiveDeliveryService, type ArchiveNotificationBatch } from "./decision-archive-delivery.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { createHash } from "node:crypto"; @@ -32,6 +33,9 @@ export const DEFAULT_DECISION_SHELF_DAYS = 30; export const DEFAULT_DECISION_ARCHIVE_DAYS = 90; const DAY_MS = 86_400_000; +type RetentionSource = Pick & { subject: Pick }; +type ArchiveSource = RetentionSource & Pick; + export type DecisionRetentionState = typeof decisionRetention.$inferSelect; export type { ArchiveNotificationBatch } from "./decision-archive-delivery.js"; @@ -128,45 +132,49 @@ export function decisionRetentionService( db: Db, options: { notifyOriginAgent?: (batch: ArchiveNotificationBatch) => Promise } = {}, ) { - async function syncItems(companyId: string, items: readonly AttentionItem[]) { + async function syncItems(companyId: string, items: readonly RetentionSource[]) { if (items.length === 0) return new Map(); - const unique = [...new Map(items.map((item) => [itemSourceKey(item), item])).values()]; - await db.insert(decisionRetention).values(unique.map((item) => ({ - companyId, - sourceKind: item.sourceKind, - sourceId: item.subject.id, - sourceActivityAt: new Date(item.activityAt), - }))).onConflictDoNothing({ - target: [decisionRetention.companyId, decisionRetention.sourceKind, decisionRetention.sourceId], + const unique = [...new Map(items.map(item => [itemSourceKey(item), item])).values()]; + return db.transaction(async tx => { + const select = () => tx.select().from(decisionRetention).where(and( + eq(decisionRetention.companyId, companyId), + inArray(decisionRetention.sourceId, unique.map(item => item.subject.id)), + )); + let rows = await select(); + const existing = new Set(rows.map(row => sourceKey(row.sourceKind, row.sourceId))); + const missing = unique.filter(item => !existing.has(itemSourceKey(item))); + if (missing.length) { + for (const kind of new Set(missing.map(item => item.sourceKind))) await notifyRetentionWork(tx, kind); + await tx.insert(decisionRetention).values(missing.map(item => ({ + companyId, sourceKind: item.sourceKind, sourceId: item.subject.id, sourceActivityAt: new Date(item.activityAt), + }))).onConflictDoNothing({ target: [decisionRetention.companyId, decisionRetention.sourceKind, decisionRetention.sourceId] }); + rows = await select(); + } + const byKey = new Map(rows.map(row => [sourceKey(row.sourceKind, row.sourceId), row])); + for (const item of unique) { + const key = itemSourceKey(item), row = byKey.get(key), activityAt = new Date(item.activityAt); + if (!row || row.sourceActivityAt.getTime() === activityAt.getTime()) continue; + await notifyRetentionWork(tx, item.sourceKind); + const [updated] = await tx.update(decisionRetention).set({ + sourceActivityAt: activityAt, version: sql`${decisionRetention.version} + 1`, updatedAt: new Date(), + }).where(and(eq(decisionRetention.id, row.id), ne(decisionRetention.sourceActivityAt, activityAt))).returning(); + if (updated) byKey.set(key, updated); + } + return byKey; }); - let rows = await db.select().from(decisionRetention).where(and( - eq(decisionRetention.companyId, companyId), - inArray(decisionRetention.sourceId, unique.map((item) => item.subject.id)), - )); - const byKey = new Map(rows.map((row) => [sourceKey(row.sourceKind, row.sourceId), row])); - for (const item of unique) { - const key = itemSourceKey(item); - const row = byKey.get(key); - const activityAt = new Date(item.activityAt); - if (!row || row.sourceActivityAt.getTime() === activityAt.getTime()) continue; - const updated = await db.update(decisionRetention).set({ - sourceActivityAt: activityAt, - version: sql`${decisionRetention.version} + 1`, - updatedAt: new Date(), - }).where(and( - eq(decisionRetention.id, row.id), - ne(decisionRetention.sourceActivityAt, activityAt), - )).returning().then((values) => values[0] ?? null); - if (updated) byKey.set(key, updated); - } - rows = [...byKey.values()]; - return new Map(rows.map((row) => [sourceKey(row.sourceKind, row.sourceId), row])); } - function itemSourceKey(item: AttentionItem) { + function itemSourceKey(item: RetentionSource) { return sourceKey(item.sourceKind, item.subject.id); } + async function writeRetention(kind: AttentionSourceKind, write: (tx: Db) => Promise) { + return db.transaction(async tx => { + await notifyRetentionWork(tx, kind); + return write(tx as unknown as Db); + }); + } + async function getState(companyId: string, sourceKind: AttentionSourceKind, sourceId: string) { return db.select().from(decisionRetention).where(and( eq(decisionRetention.companyId, companyId), @@ -175,6 +183,16 @@ export function decisionRetentionService( )).then((rows) => rows[0] ?? null); } + async function materializeApprovalState(database: Db, input: { companyId: string; sourceKind: AttentionSourceKind; sourceId: string }) { + if (input.sourceKind !== "approval") return; + const [source] = await database.select({ updatedAt: approvals.updatedAt }).from(approvals) + .where(and(eq(approvals.companyId, input.companyId), eq(approvals.id, input.sourceId))).for("update"); + if (!source) throw notFound("Attention source not found"); + await decisionRetentionService(database).syncItems(input.companyId, [{ + sourceKind: "approval", subject: { id: input.sourceId }, activityAt: source.updatedAt.toISOString(), + }]); + } + async function setKeep(input: { companyId: string; sourceKind: AttentionSourceKind; @@ -186,15 +204,18 @@ export function decisionRetentionService( if (!(await canReadDecisionSource(db, input.authActor, input.companyId, input.sourceKind, input.sourceId))) { throw notFound("Attention source not found"); } - const updated = await db.update(decisionRetention).set({ - keep: input.keep, - version: sql`${decisionRetention.version} + 1`, - updatedAt: new Date(), - }).where(and( - eq(decisionRetention.companyId, input.companyId), - eq(decisionRetention.sourceKind, input.sourceKind), - eq(decisionRetention.sourceId, input.sourceId), - )).returning().then((rows) => rows[0] ?? null); + const updated = await writeRetention(input.sourceKind, async tx => { + await materializeApprovalState(tx, input); + return tx.update(decisionRetention).set({ + keep: input.keep, + version: sql`${decisionRetention.version} + 1`, + updatedAt: new Date(), + }).where(and( + eq(decisionRetention.companyId, input.companyId), + eq(decisionRetention.sourceKind, input.sourceKind), + eq(decisionRetention.sourceId, input.sourceId), + )).returning().then((rows) => rows[0] ?? null); + }); if (!updated) throw notFound("Attention source not found"); await logActivity(db, { companyId: input.companyId, @@ -224,28 +245,31 @@ export function decisionRetentionService( throw notFound("Attention source not found"); } const now = new Date(); - const updated = await db.update(decisionRetention).set(input.archived ? { - archivedAt: now, - archivedReason: "manual", - ...archiveActorColumns(input.actor), - archiveVersion: sql`${decisionRetention.archiveVersion} + 1`, - version: sql`${decisionRetention.version} + 1`, - updatedAt: now, - } : { - archivedAt: null, - archivedReason: null, - archivedByType: null, - archivedByAgentId: null, - archivedByUserId: null, - archivedByRunId: null, - version: sql`${decisionRetention.version} + 1`, - updatedAt: now, - }).where(and( - eq(decisionRetention.companyId, input.companyId), - eq(decisionRetention.sourceKind, input.sourceKind), - eq(decisionRetention.sourceId, input.sourceId), - input.archived ? isNull(decisionRetention.archivedAt) : isNotNull(decisionRetention.archivedAt), - )).returning().then((rows) => rows[0] ?? null); + const updated = await writeRetention(input.sourceKind, async tx => { + await materializeApprovalState(tx, input); + return tx.update(decisionRetention).set(input.archived ? { + archivedAt: now, + archivedReason: "manual", + ...archiveActorColumns(input.actor), + archiveVersion: sql`${decisionRetention.archiveVersion} + 1`, + version: sql`${decisionRetention.version} + 1`, + updatedAt: now, + } : { + archivedAt: null, + archivedReason: null, + archivedByType: null, + archivedByAgentId: null, + archivedByUserId: null, + archivedByRunId: null, + version: sql`${decisionRetention.version} + 1`, + updatedAt: now, + }).where(and( + eq(decisionRetention.companyId, input.companyId), + eq(decisionRetention.sourceKind, input.sourceKind), + eq(decisionRetention.sourceId, input.sourceId), + input.archived ? isNull(decisionRetention.archivedAt) : isNotNull(decisionRetention.archivedAt), + )).returning().then((rows) => rows[0] ?? null); + }); if (!updated) { const current = await getState(input.companyId, input.sourceKind, input.sourceId); if (!current) throw notFound("Attention source not found"); @@ -298,6 +322,7 @@ export function decisionRetentionService( } const now = new Date(); for (const entry of input.manifest) { + await notifyRetentionWork(txDb, entry.sourceKind as AttentionSourceKind); const archived = await txDb.update(decisionRetention).set({ archivedAt: now, archivedReason: "accepted_proposal", @@ -333,7 +358,7 @@ export function decisionRetentionService( }); } - async function autoArchive(input: { companyId: string; items: AttentionItem[]; now?: Date }) { + async function autoArchive(input: { companyId: string; items: readonly ArchiveSource[]; now?: Date }) { const now = input.now ?? new Date(); const cutoff = new Date(now.getTime() - DEFAULT_DECISION_ARCHIVE_DAYS * DAY_MS); const candidates = input.items.filter((item) => new Date(item.activityAt) <= cutoff && !item.keep && !item.archivedAt); @@ -341,6 +366,7 @@ export function decisionRetentionService( await db.transaction(async (tx) => { const txDb = tx as unknown as Db; for (const item of candidates) { + await notifyRetentionWork(txDb, item.sourceKind); const updated = await txDb.update(decisionRetention).set({ archivedAt: now, archivedReason: "idle_ttl", diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 59389821c89..55f7943d441 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + approvalRetention: "approval-retention", decisionTargets: "decision-targets", decisionEffectRecovery: "decision-effect-recovery", decisionExpiry: "decision-expiry", diff --git a/server/src/services/issue-approvals.ts b/server/src/services/issue-approvals.ts index 9d2d0a371ac..b00133898b0 100644 --- a/server/src/services/issue-approvals.ts +++ b/server/src/services/issue-approvals.ts @@ -1,3 +1,4 @@ +import { writeApprovalChange } from "./retention-work-notifications.js"; import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { approvals, issueApprovals, issues } from "@paperclipai/db"; @@ -107,7 +108,7 @@ export function issueApprovalService(db: Db) { link: async (issueId: string, approvalId: string, actor?: LinkActor) => { const { issue } = await assertIssueAndApprovalSameCompany(issueId, approvalId); - await db + await writeApprovalChange(db, tx => tx .insert(issueApprovals) .values({ companyId: issue.companyId, @@ -116,7 +117,7 @@ export function issueApprovalService(db: Db) { linkedByAgentId: actor?.agentId ?? null, linkedByUserId: actor?.userId ?? null, }) - .onConflictDoNothing(); + .onConflictDoNothing()); return db .select() @@ -126,10 +127,10 @@ export function issueApprovalService(db: Db) { }, unlink: async (issueId: string, approvalId: string) => { - await assertIssueAndApprovalSameCompany(issueId, approvalId); - await db + const { issue } = await assertIssueAndApprovalSameCompany(issueId, approvalId); + await writeApprovalChange(db, tx => tx .delete(issueApprovals) - .where(and(eq(issueApprovals.issueId, issueId), eq(issueApprovals.approvalId, approvalId))); + .where(and(eq(issueApprovals.issueId, issueId), eq(issueApprovals.approvalId, approvalId)))); }, linkManyForApproval: async (approvalId: string, issueIds: string[], actor?: LinkActor) => { @@ -157,7 +158,7 @@ export function issueApprovalService(db: Db) { } } - await db + await writeApprovalChange(db, tx => tx .insert(issueApprovals) .values( uniqueIssueIds.map((issueId) => ({ @@ -168,7 +169,7 @@ export function issueApprovalService(db: Db) { linkedByUserId: actor?.userId ?? null, })), ) - .onConflictDoNothing(); + .onConflictDoNothing()); }, }; } diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 1e77167e85d..53f3cff74a6 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork } from "./retention-work-notifications.js"; import { notifyDecisionTargetWork, notifyDecisionTargetIssueRemoval } from "./decision-targets.js"; import { assertOnboardingFirstTaskCreationAllowed } from "./onboarding-first-task-admission.js"; @@ -11389,6 +11390,9 @@ export function issueService(db: Db) { if (patch.status === "done" || patch.status === "cancelled") { await notifyDecisionTargetWork(tx, `issue:${id}`); } + if (patch.hiddenAt !== undefined || patch.harnessKind !== undefined || patch.conversationAgentId !== undefined) { + await notifyRetentionWork(tx, "approval"); + } const updated = await tx .update(issues) .set(patch) @@ -11796,6 +11800,7 @@ export function issueService(db: Db) { try { await notifyTerminalWorkspaceWork(tx, `issue:${id}`); await requestRestorePointDeletion(tx, { sourceIssueId: id }); + await notifyRetentionWork(tx, "approval"); await notifyDecisionTargetIssueRemoval(tx, id); removedIssue = await tx .delete(issues) diff --git a/server/src/services/retention-work-notifications.ts b/server/src/services/retention-work-notifications.ts new file mode 100644 index 00000000000..3464c624e89 --- /dev/null +++ b/server/src/services/retention-work-notifications.ts @@ -0,0 +1,14 @@ +import type { Db } from "@paperclipai/db"; +import type { AttentionSourceKind } from "@paperclipai/shared"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export async function notifyRetentionWork(transaction: object, kind: AttentionSourceKind) { + if (kind === "approval") await notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalRetention); +} + +export function writeApprovalChange(db: Db, write: (transaction: Db) => Promise): Promise { + return db.transaction(async tx => { + await notifyRetentionWork(tx, "approval"); + return write(tx as unknown as Db); + }); +} diff --git a/server/src/services/routines.ts b/server/src/services/routines.ts index 50c200490b7..372c8d4f45d 100644 --- a/server/src/services/routines.ts +++ b/server/src/services/routines.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork } from "./retention-work-notifications.js"; import { notifyDecisionTargetIssueRemoval } from "./decision-targets.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { verifyAppWebhook } from "./app-webhook.js"; @@ -1965,6 +1966,7 @@ export function routineService( return updated ?? createdRun; } catch (error) { if (createdIssue) { + await notifyRetentionWork(txDb, "approval"); await notifyDecisionTargetIssueRemoval(txDb, createdIssue.id); await txDb.delete(issues).where(eq(issues.id, createdIssue.id)); } diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index 4cacb30e47e..ae81998e947 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1,3 +1,4 @@ +import { writeApprovalChange } from "./retention-work-notifications.js"; import { toolReviewProjectionService } from "./tool-review-projection.js"; import { toolReviewInterruptionService } from "./tool-review-interruption.js"; import { bindVerifiedToolApprovalArguments, claimToolTestApproval, releaseToolApproval, persistToolTestOutcome, startToolInvocation, type ToolApprovalClaim } from "./tool-approval-execution.js"; @@ -2649,7 +2650,7 @@ export function createToolGatewayService( let formalApprovalId: string | null = null; if (toolRequiresFormalApproval(input.tool)) { - const [approval] = await db + const [approval] = await writeApprovalChange(db, tx => tx .insert(approvals) .values({ companyId: input.session.companyId, @@ -2672,17 +2673,18 @@ export function createToolGatewayService( argumentsHash: canonicalArgumentsHash, }, }) - .returning(); + .returning()); formalApprovalId = approval.id; - await db + const approvalIssueId = input.session.issueId; + await writeApprovalChange(db, tx => tx .insert(issueApprovals) .values({ companyId: input.session.companyId, - issueId: input.session.issueId, + issueId: approvalIssueId, approvalId: approval.id, linkedByAgentId: input.session.agentId, }) - .onConflictDoNothing(); + .onConflictDoNothing()); } const interaction = await interactions.create( From d23dd19750a646a69340f5075a69c1b1b638d679 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 19:42:58 -0500 Subject: [PATCH 33/41] fix(server): preserve terminal workspace cursors and retry deadlines (#17505) ## Thinking Path > - Paperclip manages agent work and its execution workspaces. > - Workspace recovery scans candidates and retains deadlines for unfinished work. > - Rounded database timestamps can prevent a scan from advancing. > - Separate clock reads can give one retry obligation different deadlines. > - Repeated inspection must preserve both scan progress and the saved retry time. > - This PR keeps full cursor precision and computes each retry deadline once. > - Recovery can finish its scan and wait until work is actually due. ## Linked Issues or Issue Description **What happened?** Terminal workspace recovery can repeatedly inspect the oldest skipped candidate when its timestamp contains microseconds. Eligible workspaces behind it never reach cleanup. Separately, nested-clone migration and isolated-repository rehome recovery use separate clock reads for their retry gate and advertised deadline. A later inspection can move the advertised deadline backward. **Expected behavior** Each bounded scan advances using the complete timestamp and ID. The retry gate and advertised deadline represent the same saved time, including when the clock advances between calls. **Steps to reproduce** For cursor starvation, place two skipped workspaces with microsecond timestamps before an eligible terminal workspace and use a one-row page. For deadline drift, inject a clock that advances one millisecond per read, run disabled migration or rehome recovery, then inspect it again before the retry time. Stacked on #17497. Related cursor fixes #17479 and #17482 have separate owners. This PR owns workspace recovery cursor and retry precision. ## What Changed - Preserve the database timestamp as text in the private timestamp/ID cursor. - Compute one retry time per migration or rehome outcome and reuse it in the existing maps. - Test the cursor with real PostgreSQL microsecond timestamps and the retry boundary with a deterministic advancing clock. - Support validated disposable external test databases in the affected fixtures, retaining embedded fallback. - Document the evidence in `doc/idle-terminal-workspace-pagination.md`. ## Verification Current test revision: `9d846c84e76c90893ab62a09548c11bbdb627771`, followed by docs-only head `a34dc50de3`, which received both 5/5 reviews and passing CI. This candidate normally integrates landed approval retention `553a004926`. Every runtime and acceptance-test source matches that tested/reviewed tree; only four independent Runner E2E accounting files and the ledger differ. Migration metadata is unchanged. Existing local checks are reused for their unchanged source; fresh final-head CI and reviews are required against master. This remains the terminal-precision duty. - **136 cases pass** in four complete suites: execution workspace service, terminal worker, isolated repository rehome and nested-clone migration. Server typecheck and production build pass with the current Runner vendor. - The preceding `f0f0de1f6f` run also passed all 136 on three newly created, unmigrated PostgreSQL databases. The suites themselves applied migrations. The current run reuses those clean fixtures after integrating the parent's approval wake repair; it is not presented as a fresh-database run. - The regressions retain real PostgreSQL microsecond timestamps and an advancing clock. Before repair, skipped rows starved eligible cleanup and early inspection moved a saved retry by one millisecond. Current assertions preserve exact progress/deadline behavior; no timeout or business policy is weakened. - Earlier external-fixture setup tests failed with missing company/restore-point schema. Their setup repair and red/green record remain in `doc/idle-terminal-workspace-pagination.md`. This PR adds no migration or new queue. - Earlier stack integration receipts and failures remain in the stack ledger and `doc/idle-approved-tool-recovery.md`; they are historical parent evidence, not this head's acceptance. No full local repository suite, deployed suspension, fleet cost or live provider qualification is claimed. Commands: server Vitest for `execution-workspaces-service`, `terminal-workspace-work`, `isolated-repository-rehome` and `nested-clone-migration`; server typecheck/build. Fresh exact-head CI and Greptile/Ponytail 5/5 remain required before merge in dependency order. ## Risks The cursor remains private query state. Retry intervals, eligibility, cleanup authority, and shutdown behavior are unchanged. No schema, queue, external alarm, or additional polling loop is introduced. The tests establish workspace recovery behavior, not host suspension or cost savings. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-terminal-workspace-pagination.md | 103 ++++++++++++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 59 +++++++++- .../execution-workspaces-service.test.ts | 20 +++- .../isolated-repository-rehome.test.ts | 25 +++-- .../__tests__/nested-clone-migration.test.ts | 21 ++-- server/src/services/execution-workspaces.ts | 33 +++--- 6 files changed, 224 insertions(+), 37 deletions(-) create mode 100644 doc/idle-terminal-workspace-pagination.md diff --git a/doc/idle-terminal-workspace-pagination.md b/doc/idle-terminal-workspace-pagination.md new file mode 100644 index 00000000000..eda475888d1 --- /dev/null +++ b/doc/idle-terminal-workspace-pagination.md @@ -0,0 +1,103 @@ +# Terminal workspace scan and retry deadline precision + +Terminal workspace maintenance already uses the shared delivery coordinator and +retains deadlines only for outstanding work. Its bounded scan must progress past +ineligible rows so that later eligible work can finish and the worker can park. + +The source query orders candidates by `(updated_at, id)`. PostgreSQL timestamps +can contain microseconds. The former cursor used a JavaScript Date, which loses +that precision. A skipped row could then compare greater than its own rounded +cursor. Repeated full pages kept recovery active and prevented later cleanup. + +The query now carries the database timestamp as text alongside each workspace. +Only the private scan cursor uses that value. Workspace data, business deadlines, +and the frozen rotation boundary keep their existing types and behavior. The +cursor comparison uses the same timestamp/ID tuple as the ordering. The source +service continues to own cleanup eligibility, authorization, concurrency fences, +page limits, abort handling, failed-page recovery, and rotation under churn. +No scheduler API, queue, migration, producer signal, or external alarm changes. + +## Verification + +The existing full-page starvation regression now gives its skipped rows actual +PostgreSQL microsecond timestamps. Before the repair, the eligible workspace was +never archived. After the repair, the full workspace service and terminal worker +selection passes all 87 tests, including bounded rotation under continuous churn, +changed rows behind the cursor, failed-page recovery, quiet workers, and shutdown. +The service suite accepts the repository's validated disposable external +PostgreSQL URL. Setup applies pending migrations before opening the test client, +then clears fixture tables before the first test as well as after each test. This +supports a fresh database and reuse after an interrupted test run. The database +connection closes during teardown. Server typechecking passes. + +This fixes recovery progress. It does not establish provider suspension or change +which active or unresolved resources prevent instance sleep. + +Workspace-wide typechecking and the production build also pass. Full sharded CI +and fresh review remain required before this PR is ready. + +Review follow-up reproduced the missing-schema failure against a newly created, +unmigrated disposable database. The setup repair migrates that same database +inside the suite; the full workspace-service selection then runs without manual +schema preparation. The production cursor implementation is unchanged. + +## Preserve one retry deadline through early inspection + +Nested-clone migration and isolated-source rehoming kept a retry guard and a +separate advertised scheduler deadline. Each value called the clock separately. +When the clock advanced between those calls, an early inspection copied the +earlier guard back into the advertised deadline. CI observed a one-millisecond +backward change in an otherwise unchanged retry. + +Each path now computes one retry timestamp and stores that value in all of its +retry and scheduling state. The cadence, cleanup eligibility, quiet conditions, +rotation, and wake contract do not change. No new timer, queue, or table is added. + +The existing nested-clone regression and the isolated-rehome dry-run case now +use a clock that advances one millisecond per read. Both failed before the fix +with the same one-millisecond drift. Both exact-deadline assertions remain. The +complete migration suites passed all 49 cases afterward, using real PostgreSQL +and Git, including preserved files/history and interrupted migration recovery. +The complete workspace-service and terminal-worker suites passed another 87 +cases; server typechecking passed. The two focused pre-fix failures used a name +filter intentionally; the reported final 136 cases are four complete suites +without skips. Validated external test database hooks keep each migration suite +on its own disposable database and retain the existing embedded fixture fallback. + +These are local correctness and scheduling results, not deployed host-sleep or +external-wake proof. Remote acceptance must be checked at the published head. + +## External fixture migrations (October 26 review repair) + +Review found that the two migration suites opened external test databases without +applying migrations. The previous verification used prepared databases, which +hid this setup gap. Both selected retry cases failed against new empty disposable +databases with missing `companies` and restore-point tables. Setup now applies +pending migrations before creating its test client, matching the workspace +service fixture. The embedded path is unchanged. + +All 49 cases in the two complete suites then passed against those same databases, +including Git/filesystem migration, recovery, and exact retry-deadline checks. +The fixture itself prepared the databases. This repair changes no production +recovery code. Evidence: `/tmp/sleep-workspace-fresh-red.log` and +`/tmp/sleep-workspace-fresh-green.log`. + + +## Current parent verification + +Approval-retention parent `3efda33812` is integrated with normal merges. All +workspace precision, retry and fixture implementations remain unchanged from the +preceding candidate. All 136 cases in the complete workspace-service, terminal +worker, isolated-rehome and nested-clone suites pass at `9d846c84e7`; server types +and production build pass with the current Runner vendor. + +The preceding `f0f0de1f6f` run also passed all 136 cases with three newly created, +unmigrated external databases: the suites themselves applied the current +migrations. The `9d846c84e7` run reused those clean fixtures after the parent's +approval wake repair. This retains actual proof of fixture self-preparation +without mislabeling the reused run as a fresh-database test. No new migration, +queue, wake contract or workspace policy change is added by this PR. + +Earlier microsecond starvation, one-millisecond retry drift and missing-schema +failures remain documented above. Fresh exact-head CI and both 5/5 reviews remain +required in stack order; the parent is not claimed merged. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index c2f604bfcc6..a5bc0de3881 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -15,7 +15,7 @@ Do not mark an item done from a timer removal or a green check alone. | 01 | Custom-image setup expiry | Session writes, existing expiry, startup/retry/drain | [#17443](https://github.com/paperclipai/paperclip/pull/17443), merged after CI and passing reviews | | 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending | | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | -| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Approval retention implemented in slice 14; final review/CI pending; other source retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI | +| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Approval retention [#17497](https://github.com/paperclipai/paperclip/pull/17497) and archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged; other source retention pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) merged | @@ -772,6 +772,7 @@ files. All runtime and acceptance test sources are byte-identical to its tested remain required after retargeting to master. + ## Slice 13: Grok routine dispatch The one-second idle scan is replaced with a shared-coordinator worker. The existing @@ -809,6 +810,7 @@ its tested/reviewed `679f523d78` tree; the 273-case, type/build and migration receipts retain their original revisions. Migration 0391 and all parent journal entries are unchanged. Fresh final-head CI and reviews required against master. + ## Next owner audit: decision retention Slice 13 is [#17490](https://github.com/paperclipai/paperclip/pull/17490), head @@ -1026,3 +1028,58 @@ accounting files. Its entire runtime and acceptance sources match the tested and reviewed `3efda33812` tree (315 cases, server types/build). Migration 0392 and the parent journal prefix are unchanged. Fresh final-head CI/reviews are required against master. The abandoned-import-spool inventory duty remains. + +Latest slice-14 head is `a29ce265b0c00678ada3f29f533d79f80fe25654`. +The precision finding was replied to (4368650634) and its thread resolved. +Fresh Greptile review was requested (6390004743). CI run **40136197148** is +confirmed in progress on this head; commit review run 40136193928 succeeded. +The previous-head CI run 40135489716 succeeded. Do not treat that as evidence +for the new head. No local test/build process remains active. + +Next actions: finish fresh #17497 CI/review; reproduce and repair precision +paging in the earlier TTL and effect-recovery owners, propagating parent changes +through later PRs; qualify the terminal-workspace cursor; then implement the +join-request retention audit on this branch. Every original inventory item and +remaining retention source remains required. No merge is authorized. + +## Decision and workspace precision follow-up + +Expiry #17479 now preserves the PostgreSQL timestamp in its cursor, head +`e89c682ba5`. Its regression failed before repair; all 15 expiry cases and server +typechecking pass. Fresh CI is green and Greptile is 5/5 at that head. +Effect recovery #17482 has the same verified repair at `cebcd9c427`: its invalid +legacy claim regression failed before repair, then all 18 cases and typechecking +passed. Both changes have been merged through #17486, #17490, and #17497. +The combined 33 expiry/effect cases pass on the latest stack. Fresh dependent +CI/review is running. Superseded runs on the Grok/retention heads were cancelled +when both base and head updated; replacement workflows exist. Do not rerun or +claim those heads green until their current runs finish. + +Approval retention previously passed all checks and fresh 5/5 at `a29ce265b0`. +Its new head is `2c5babcabc`, with only parent paging repairs added. Join retention +still has no implementation; its branch now includes the current parents. + +The terminal-workspace audit also reproduced starvation using real microsecond +source timestamps. The focused branch `codex/sleep-terminal-workspace-precision` +preserves cursor precision in the owning query. All 87 workspace service/worker +cases and server typechecking pass. See +[workspace pagination](../idle-terminal-workspace-pagination.md). Publication and +fresh CI/review are next. Join retention remains the next new migration duty. + + +Terminal workspace precision follow-up #17505 normally integrates approval wake +parent `3efda33812`. All 136 cases across four complete workspace/worker/rehome/ +nested-clone suites, server types and build pass at `9d846c84e7`. The preceding +`f0f0de1f6f` run passed the same cases on three freshly created unmigrated fixtures; +the suites prepared their own schemas. Current reused-fixture evidence and old +starvation/clock/setup failures are distinguished in its owning audit. No new +schema or wake mechanism. Fresh head CI and reviews required; parent unmerged. + + +Slice 14 merged `a26d865c99` → `553a004926` after exact-head Greptile/Ponytail +5/5 and all 58 checks passed or were neutral/skipped. Terminal precision #17505 +normally integrates that landed dependency and four independent Runner E2E +accounting files. Its entire runtime and acceptance sources match the tested/ +reviewed `a34dc50de3` tree (136 cases, server types/build); migration metadata is +unchanged. No local checks were repeated on unchanged source. Fresh final-head +CI/reviews are required against master; all remaining checklist duties persist. diff --git a/server/src/__tests__/execution-workspaces-service.test.ts b/server/src/__tests__/execution-workspaces-service.test.ts index faf0f68bff8..8a4a523fa26 100644 --- a/server/src/__tests__/execution-workspaces-service.test.ts +++ b/server/src/__tests__/execution-workspaces-service.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import * as localServiceSupervisor from "../services/local-service-supervisor.js"; import { EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db"; import { execFile } from "node:child_process"; @@ -11,6 +12,7 @@ import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest import { eq, inArray, sql } from "drizzle-orm"; import { activityLog, + applyPendingMigrations, agents, companies, createDb, @@ -160,7 +162,8 @@ describe("execution workspace config helpers", () => { }); }); -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EXECUTION_WORKSPACES_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -258,8 +261,10 @@ describeEmbeddedPostgres("executionWorkspaceService.getCloseReadiness", () => { }>(); beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-execution-workspaces-service-"); - db = createDb(tempDb.connectionString); + if (externalUrl) await applyPendingMigrations(externalUrl); + else tempDb = await startEmbeddedPostgresTestDatabase("paperclip-execution-workspaces-service-"); + db = createDb(externalUrl ?? tempDb!.connectionString); + await resetDatabase(); svc = executionWorkspaceService(db, { resolvePullRequestDetails: vi.fn(async (companyId, reference) => pullRequestDetailsByKey.get(`${companyId}:${reference.number}`) @@ -272,7 +277,7 @@ describeEmbeddedPostgres("executionWorkspaceService.getCloseReadiness", () => { }); }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS); - afterEach(async () => { + async function resetDatabase() { await db.delete(workspaceRuntimeServices); await db.delete(activityLog); await db.delete(issueRecoveryActions); @@ -286,6 +291,10 @@ describeEmbeddedPostgres("executionWorkspaceService.getCloseReadiness", () => { await db.delete(heartbeatRuns); await db.delete(agents); await db.delete(companies); + } + + afterEach(async () => { + await resetDatabase(); pullRequestDetailsByKey.clear(); for (const dir of tempDirs) { @@ -295,6 +304,7 @@ describeEmbeddedPostgres("executionWorkspaceService.getCloseReadiness", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); @@ -1124,6 +1134,8 @@ describeEmbeddedPostgres("executionWorkspaceService.getCloseReadiness", () => { .set({ sourceIssueId: openIssueId, updatedAt: new Date(Date.UTC(2020, 0, index + 1)) }) .where(eq(executionWorkspaces.id, workspaceId)); } + await db.execute(sql`update execution_workspaces set updated_at = updated_at + interval '456 microseconds' + where id in (${sql.join(skippedWorkspaceIds.map(id => sql`${id}::uuid`), sql`, `)})`); // The eligible workspace is newest, so it sorts after the whole skipped page. const eligible = await seedAncestryTerminalWorkspace({ updatedAt: new Date(Date.UTC(2020, 0, 9)) }); diff --git a/server/src/__tests__/isolated-repository-rehome.test.ts b/server/src/__tests__/isolated-repository-rehome.test.ts index 889b0ffa63f..1a3e4de47f1 100644 --- a/server/src/__tests__/isolated-repository-rehome.test.ts +++ b/server/src/__tests__/isolated-repository-rehome.test.ts @@ -6,7 +6,7 @@ import { randomUUID } from "node:crypto"; import { pathToFileURL } from "node:url"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { eq, sql } from "drizzle-orm"; -import { agents, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issues, projects } from "@paperclipai/db"; +import { agents, applyPendingMigrations, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issues, projects } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { resolveManagedProjectWorkspaceDir } from "../home-paths.js"; import { materializeManagedProjectWorkspace, ensureManagedRepositoriesIgnored } from "../services/managed-repository-checkout.js"; @@ -25,17 +25,21 @@ const git = (cwd: string, ...args: string[]) => execFileSync("git", args, { cwd, const exists = (target: string) => fs.lstat(target).then(() => true, () => false); const DAY_MS = 24 * 60 * 60 * 1000; -const support = await getEmbeddedPostgresTestSupport(); +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_ISOLATED_REHOME_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = support.supported ? describe : describe.skip; describeEmbeddedPostgres("isolated repository source re-home", () => { let db: ReturnType; - let database: Awaited>; + let database: Awaited> | undefined; let root: string; const previousHome = process.env.PAPERCLIP_HOME; const previousInstance = process.env.PAPERCLIP_INSTANCE_ID; beforeAll(async () => { - database = await startEmbeddedPostgresTestDatabase("isolated-repository-rehome-"); - db = createDb(database.connectionString); + if (external) await applyPendingMigrations(external); + else database = await startEmbeddedPostgresTestDatabase("isolated-repository-rehome-"); + db = createDb(external ?? database!.connectionString); root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-isolated-rehome-")); process.env.PAPERCLIP_HOME = path.join(root, "home"); process.env.PAPERCLIP_INSTANCE_ID = "default"; @@ -50,7 +54,7 @@ describeEmbeddedPostgres("isolated repository source re-home", () => { vi.unstubAllEnvs(); if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance; - await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true }); + await db.$client.end({ timeout: 1 }); await database?.cleanup(); await fs.rm(root, { recursive: true, force: true }); }); async function fixture(options: { sharedBase?: boolean; issueStatus?: "in_progress" | "done" } = {}) { @@ -409,11 +413,16 @@ describeEmbeddedPostgres("isolated repository source re-home", () => { await executionWorkspaceService(db, { gitCacheExpirationEnabled: true }).sweepTerminalWorkspaces(); await executionWorkspaceService(db, { isolatedSourceRehomeEnabled: true }).sweepTerminalWorkspaces(); await expectUntouched(f, before); - const dryRun = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true, isolatedSourceRehomeDryRun: true }); + let tick = Date.now(); + const dryRun = executionWorkspaceService(db, { gitCacheExpirationEnabled: true, isolatedSourceRehomeEnabled: true, + isolatedSourceRehomeDryRun: true, now: () => new Date(tick++) }); await dryRun.sweepTerminalWorkspaces(); await expectUntouched(f, before); expect(await exists(f.clone)).toBe(true); - expect(await dryRun.nextTerminalWorkspaceSweepAt()).toBeGreaterThan(Date.now()); + const retryAt = await dryRun.nextTerminalWorkspaceSweepAt(); + expect(retryAt).toBeGreaterThan(Date.now()); + await dryRun.sweepTerminalWorkspaces(); + expect(await dryRun.nextTerminalWorkspaceSweepAt()).toBe(retryAt); expect(await rehomeIsolatedRepositoryWorkspace(db, await f.row(), { dryRun: true })).toMatchObject({ outcome: "dry_run", rehomed: false, to: f.target }); await expectUntouched(f, before); }); diff --git a/server/src/__tests__/nested-clone-migration.test.ts b/server/src/__tests__/nested-clone-migration.test.ts index b417a4ce897..1d57db13401 100644 --- a/server/src/__tests__/nested-clone-migration.test.ts +++ b/server/src/__tests__/nested-clone-migration.test.ts @@ -6,7 +6,7 @@ import { randomUUID } from "node:crypto"; import { pathToFileURL } from "node:url"; import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { eq, sql } from "drizzle-orm"; -import { agents, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces } from "@paperclipai/db"; +import { agents, applyPendingMigrations, companies, createDb, executionWorkspaceRestorePoints, executionWorkspaces, heartbeatRuns, issues, projects, projectWorkspaces } from "@paperclipai/db"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js"; import { materializeIsolatedRemoteWorkspaceSource } from "../services/isolated-remote-workspace-source.js"; @@ -44,17 +44,21 @@ describe("nested clone migration batches", () => { }); }); -const support = await getEmbeddedPostgresTestSupport(); +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NESTED_CLONE_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = support.supported ? describe : describe.skip; describeEmbeddedPostgres("nested repository clone migration", () => { let db: ReturnType; - let database: Awaited>; + let database: Awaited> | undefined; let root: string; const previousHome = process.env.PAPERCLIP_HOME; const previousInstance = process.env.PAPERCLIP_INSTANCE_ID; beforeAll(async () => { - database = await startEmbeddedPostgresTestDatabase("nested-clone-migration-"); - db = createDb(database.connectionString); + if (external) await applyPendingMigrations(external); + else database = await startEmbeddedPostgresTestDatabase("nested-clone-migration-"); + db = createDb(external ?? database!.connectionString); root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-nested-migration-")); process.env.PAPERCLIP_HOME = path.join(root, "home"); process.env.PAPERCLIP_INSTANCE_ID = "default"; @@ -68,7 +72,7 @@ describeEmbeddedPostgres("nested repository clone migration", () => { vi.unstubAllEnvs(); if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; else process.env.PAPERCLIP_HOME = previousHome; if (previousInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID; else process.env.PAPERCLIP_INSTANCE_ID = previousInstance; - await db.$client.end({ timeout: 1 }); await database.cleanup(); await fs.rm(root, { recursive: true, force: true }); + await db.$client.end({ timeout: 1 }); await database?.cleanup(); await fs.rm(root, { recursive: true, force: true }); }); async function repository(location: string, name: string, files: Record) { @@ -504,7 +508,10 @@ describeEmbeddedPostgres("nested repository clone migration", () => { const agentId = randomUUID(); await db.insert(agents).values({ id: agentId, companyId: f.companyId, name: "Fixture", role: "engineer", adapterType: "process" }); const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId, issueId: f.issueId, status: "running" }).returning(); - const busy = executionWorkspaceService(db, { nestedCloneMigrationEnabled: true }); + // Make a second clock read observably different without changing the + // scheduler's saved business deadline during an early inspection. + let tick = Date.now(); + const busy = executionWorkspaceService(db, { nestedCloneMigrationEnabled: true, now: () => new Date(tick++) }); await busy.sweepTerminalWorkspaces(); await expectUntouched(f, before); const retryAt = await busy.nextTerminalWorkspaceSweepAt(); diff --git a/server/src/services/execution-workspaces.ts b/server/src/services/execution-workspaces.ts index 4d83e2c8a9c..ef4c2da940f 100644 --- a/server/src/services/execution-workspaces.ts +++ b/server/src/services/execution-workspaces.ts @@ -45,7 +45,7 @@ import { createHash } from "node:crypto"; import fs from "node:fs/promises"; import path from "node:path"; import { isDeepStrictEqual, promisify } from "node:util"; -import { and, asc, desc, eq, gt, inArray, isNull, lte, ne, or, sql } from "drizzle-orm"; +import { and, asc, desc, eq, inArray, isNull, lte, ne, or, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { executionWorkspaces, @@ -1757,7 +1757,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic // required: without it the query re-selects the oldest ineligible rows every // sweep and starves eligible rows behind them. The cursor resets to the start // when a sweep reaches the end of the candidate set. - let terminalSweepCursor: { updatedAt: Date; id: string } | null = null; + let terminalSweepCursor: { updatedAt: string; id: string } | null = null; const terminalInspections = new Map; nextAt: number }>(); const terminalWorkspaceDeadlines = new Map(); const isolatedRehomeDeadlines = new Map(); @@ -3399,17 +3399,11 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic ? lte(executionWorkspaces.updatedAt, boundary) : undefined; const cursorFilter = cursor - ? or( - gt(executionWorkspaces.updatedAt, cursor.updatedAt), - and( - eq(executionWorkspaces.updatedAt, cursor.updatedAt), - gt(executionWorkspaces.id, cursor.id), - ), - ) + ? sql`(${executionWorkspaces.updatedAt}, ${executionWorkspaces.id}) > (${cursor.updatedAt}::timestamptz, ${cursor.id}::uuid)` : undefined; const scanFilter = and(baseCandidateFilter, boundaryFilter, cursorFilter); const candidates = await db - .select() + .select({ workspace: executionWorkspaces, cursorAt: sql`${executionWorkspaces.updatedAt}::text` }) .from(executionWorkspaces) .where(scanFilter) .orderBy(asc(executionWorkspaces.updatedAt), asc(executionWorkspaces.id)) @@ -3422,7 +3416,7 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic terminalSweepBoundary = null; } else { const lastCandidate = candidates[candidates.length - 1]!; - terminalSweepCursor = { updatedAt: lastCandidate.updatedAt, id: lastCandidate.id }; + terminalSweepCursor = { updatedAt: lastCandidate.cursorAt, id: lastCandidate.workspace.id }; } const result = { checked: candidates.length, @@ -3470,7 +3464,8 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic "Isolated task directory retention inspected"); }; - for (let workspace of candidates) { + for (const candidate of candidates) { + let workspace = candidate.workspace; signal?.throwIfAborted(); // Trash deletion is part of compaction, so a sweep that skips compaction // also leaves leftover trash for a later sweep. @@ -3488,8 +3483,9 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic const [after] = await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)); if (after && (isIsolatedRepositoryRehomeCandidate(after) || hasPendingIsolatedRepositoryRehomeCleanup(after))) { if (isolatedRehomeNotBefore.size >= 4096) isolatedRehomeNotBefore.delete(isolatedRehomeNotBefore.keys().next().value!); - isolatedRehomeNotBefore.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS); - isolatedRehomeDeadlines.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS); + const retryAt = now().getTime() + ISOLATED_REHOME_RETRY_MS; + isolatedRehomeNotBefore.set(workspace.id, retryAt); + isolatedRehomeDeadlines.set(workspace.id, retryAt); } else { isolatedRehomeNotBefore.delete(workspace.id); isolatedRehomeDeadlines.delete(workspace.id); @@ -3526,14 +3522,17 @@ export function executionWorkspaceService(db: Db, opts: ExecutionWorkspaceServic } const [after] = names.length > 0 ? await db.select().from(executionWorkspaces).where(eq(executionWorkspaces.id, workspace.id)) : [workspace]; if (!dryRun && after && hasPendingNestedCloneMigration(after)) retry = true; + // The retry gate and advertised deadline describe one obligation. + // An early inspection must not move it by reading the clock again. + const retryAt = now().getTime() + ISOLATED_REHOME_RETRY_MS; if (remaining) { if (nestedCloneMigrationNotBefore.size >= 4096) nestedCloneMigrationNotBefore.delete(nestedCloneMigrationNotBefore.keys().next().value!); - nestedCloneMigrationNotBefore.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS); + nestedCloneMigrationNotBefore.set(workspace.id, retryAt); } else nestedCloneMigrationNotBefore.delete(workspace.id); if (retry) { if (nestedCloneMigrationRetryAt.size >= 4096) nestedCloneMigrationRetryAt.delete(nestedCloneMigrationRetryAt.keys().next().value!); - nestedCloneMigrationRetryAt.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS); - nestedCloneMigrationDeadlines.set(workspace.id, now().getTime() + ISOLATED_REHOME_RETRY_MS); + nestedCloneMigrationRetryAt.set(workspace.id, retryAt); + nestedCloneMigrationDeadlines.set(workspace.id, retryAt); } else { nestedCloneMigrationRetryAt.delete(workspace.id); nestedCloneMigrationDeadlines.delete(workspace.id); From 8406047b4e9fdf473f90a19573c6f4575731bb5a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:03:16 -0500 Subject: [PATCH 34/41] fix(server): schedule join request retention from durable deadlines (#17508) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages agent companies and their access requests. > - Pending join requests appear in the attention feed and can age into automatic archival. > - The maintenance tick reads those requests even when no archive is due. > - Existing request rows already store the activity timestamp needed for retention. > - Committed changes can wake a worker that registers the next actual deadline. > - This PR gives join-request retention its own bounded worker and removes its recurring feed scan. > - Idle and resolved history stop producing retention traffic without changing access approval. ## Linked Issues or Issue Description **What happened?** Every retention maintenance pass reads pending join requests through the attention feed. Empty, resolved, kept, and future-due states still participate in recurring discovery. **Expected behavior** Committed request and retention changes wake the owner. It waits until the next actual archive deadline and stops querying when no eligible request remains. Existing approval and visibility behavior is preserved. **Steps to reproduce** Start an idle company with no pending join requests, or with requests younger than the 90-day archive threshold. Observe the maintenance feed repeatedly select join_requests. No state change is needed to cause those reads. This PR targets master after #17505 merged. Approval retention is #17497. Other attention source kinds remain separate migration duties. No duplicate open join-retention PR was found. ## What Changed - Add a join-request retention owner that preserves pending status, active-company visibility, matching invite ownership, and keep/archive controls. Invite expiry or revocation retains the existing feed policy. - Reuse one `createDecisionRetentionSweep` for approval and join paging, atomic retention and finite scan cycles that revisit failures despite continuing arrivals. Eligibility and source-row locks stay with each owner. - Register real saved deadlines, rounded upward in SQL to millisecond precision, with the existing coordinator. Preserve 100-row pages and thirty-second notification coalescing without idle polling. - Materialize unseen join state inside the existing authorized keep/archive/revive transaction, preserving archive choices and source locking. - Notify before request inserts and updates in the existing acceptance and Slack transactions. Use a small SQL-only helper for replay and approval/rejection updates. Company status, budget pause/resume, deletion, and retention controls notify this owner too. - Preserve the approved-only API-key consumption path without a new wrapper; it cannot produce pending retention work. - Exclude join requests from maintenance feed collection while retaining normal UI reads. - Add an idempotent concurrent partial pending index and prune old schema snapshots through the existing generator. - Document producer coverage, boundaries, failure evidence, and remaining work. ## Verification Test revision: `df1815fbba`. Normal integration of landed workspace precision `d23dd19750` has an identical complete source tree. This remains only the join-retention duty and its shared approval-retention mechanics, targeting master. - **55 worker cases pass**, including two new PostgreSQL regressions that failed before repair: continuing overdue arrivals no longer postpone a recovered failure indefinitely, while an entire failed page still allows later healthy rows to progress. - The nine complete suites passed 341 cases but failed an attention cleanup hook after its 16,400-run case. After vacuum/analyze of only that disposable fixture, the full 31-case attention suite passes unchanged. Together with the original eight passing suites (311 cases), all 342 distinct cases pass across complete-suite executions. Recovery is established; the original cleanup slowdown is not claimed explained. Server typecheck and production build pass. - Earlier CI at `df1d4d0` failed the unchanged decision-expiry restart deadline assertion (a thirty-second retry appeared before the saved expiry). All twenty unchanged expiry cases pass locally, but the CI timing cause remains unproven. No retry, timeout relaxation or assertion change is used to dismiss it; fresh final-head CI remains required. - **Five regressions failed before repair:** three unseen join controls returned 404, and both owners advertised a microsecond deadline one millisecond early. All pass. Two old real-producer expectations failed after the correct rounding change; they now derive exact expected milliseconds independently from database integer microseconds instead of truncating a JavaScript Date. - The initial timing-only concurrency test still passed a missing-lock mutation. It now observes the actual PostgreSQL wait before releasing the writer. Production passes; removing the lock fails with zero waiters instead of one. Source is restored before commit. The archive-first committed-order case confirms that later activity stays archived until explicit revival; no broader race guarantee is claimed. - Generated concurrent/idempotent `0393_open_grandmaster.sql` adds only the pending join index. All 391 parent journal entries and historical SQL files remain unchanged. Five generated snapshots remain (0389–0393). Migration safety/numbering, schema drift and actual apply/replay on six isolated PostgreSQL fixtures pass. - Existing lifecycle cases retain real HTTP invite acceptance/approval/rejection, Slack identity linking, rollback/outer commit, source visibility, failure rotation, drain/resume and shutdown coverage. Earlier stack setup and CI failures remain in the owning audit and stack ledger; they are historical evidence, not this head's acceptance. Commands: server Vitest for `join-request-retention-work`, `approval-retention-work`, `approvals-service`, `attention-service`, `decision-retention-service`, `budgets-service`, `tool-gateway-service`, `server-startup-feedback-export`, `scoped-deadline-work`; server typecheck/build; DB migration checks, snapshot drift and actual apply/replay. Fresh exact-head CI, security and Greptile/Ponytail 5/5 remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - A future supported producer must signal its transaction. Independent SQL writers and another active server process are outside the process-local notification contract. - A writer that wins the source lock first is rechecked before archival. If archival commits first, later activity preserves the archived state until explicit revival. Existing invite replay expires before the retention cutoff and Slack replay deduplicates. Future producers needing renewed visibility must request revival. Cross-table visibility retains the existing contract. - The worker registers only under the existing maintenance enablement switch. The internal deadline does not add Cloud alarm coverage or relax conservative sleep blockers. - New eligibility can wait within the thirty-second aggregate notification window; known deadlines preempt it. - Archival preserves pending approval status. Other attention sources and aggregate maintenance remain separate work. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-join-request-retention.md | 183 +++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 59 ++- packages/db/src/client.ts | 1 + .../src/migrations/0393_open_grandmaster.sql | 1 + ...{0388_snapshot.json => 0393_snapshot.json} | 114 +++++- packages/db/src/migrations/meta/_journal.json | 7 + packages/db/src/schema/join_requests.ts | 2 + .../__tests__/approval-retention-work.test.ts | 31 ++ .../join-request-retention-work.test.ts | 377 ++++++++++++++++++ .../server-startup-feedback-export.test.ts | 18 + server/src/index.ts | 5 +- server/src/routes/access.ts | 18 +- server/src/services/approval-retention.ts | 66 ++- server/src/services/attention.ts | 3 + server/src/services/budgets.ts | 6 +- .../chat-channels/slack/identity-links.ts | 2 + server/src/services/companies.ts | 6 +- server/src/services/company-deletion.ts | 4 +- .../src/services/decision-retention-sweep.ts | 50 +++ server/src/services/decision-retention.ts | 16 +- .../services/delivery-work-notifications.ts | 1 + server/src/services/join-request-retention.ts | 52 +++ .../services/retention-work-notifications.ts | 13 + 23 files changed, 963 insertions(+), 72 deletions(-) create mode 100644 doc/idle-join-request-retention.md create mode 100644 packages/db/src/migrations/0393_open_grandmaster.sql rename packages/db/src/migrations/meta/{0388_snapshot.json => 0393_snapshot.json} (99%) create mode 100644 server/src/__tests__/join-request-retention-work.test.ts create mode 100644 server/src/services/decision-retention-sweep.ts create mode 100644 server/src/services/join-request-retention.ts diff --git a/doc/idle-join-request-retention.md b/doc/idle-join-request-retention.md new file mode 100644 index 00000000000..14f46f25bb8 --- /dev/null +++ b/doc/idle-join-request-retention.md @@ -0,0 +1,183 @@ +# Join request retention while idle + +Pending join requests already have durable activity timestamps. Retention now +uses their next 90-day archive deadline instead of including them in the main +maintenance tick's attention-feed scan. The regular attention UI still includes +join requests. This changes automatic retention discovery, not access approval. + +## Owner and eligibility + +`join-request-retention.ts` owns candidate eligibility, its locked recheck and +deadline. `decision-retention-sweep.ts` shares bounded paging, per-row retention +transactions and failure handling with the approval-retention owner. The shared +sweep consumes normalized candidates; each owner retains its table/query policy. +A candidate is a pending_approval request belonging to an active company, with +an invite whose company matches the request. Kept and already archived items do +not schedule work. Invite expiration or revocation does not remove the existing +attention item and therefore does not disable its retention. No provider policy +is interpreted by the worker or shared scheduler. + +The next deadline is the earliest eligible updatedAt plus 90 days, rounded up +in SQL to the scheduler's millisecond resolution. It must not wake a fraction of +a millisecond early and then wait thirty seconds for recovery. The partial +pending index orders updated_at and id. Resolved request history is excluded by +the index predicate. Each pass attempts at most 100 due rows. The private cursor +preserves database timestamp precision so a failed page cannot starve later work. +Errors keep the shared 30-second retry; successful full pages continue until +work is exhausted. No eligible rows means no scheduled SQL or network scan. + +Each candidate is re-read under its request row lock before retention changes. +A writer that wins that lock first is rechecked before archival. If archival +commits first, a later activity write leaves the item archived until explicit +revival; updating activity does not clear existing retention choices. The tests +cover the writer-first race and the archive-first committed outcome. Supported +invite replay expires before the 90-day cutoff and Slack access replay deduplicates; +a future producer that needs to reappear after archival must request revival. +The shared retention owner still controls keep, archive/revive, version checks, +activity logging, and notification creation. Archiving does not approve or reject a join. +Join requests have no origin agent, so their archival creates no agent delivery. +Company and invite joins preserve existing visibility policy; this does not add +cross-table serializability beyond the existing retention contract. + +## Producer audit + +| Producer | Notification boundary | +| --- | --- | +| Invite acceptance creates a pending request | Existing acceptance transaction, before the request insert | +| Supported gateway invite replay changes activity | Small writeJoinRequestChange transaction around the existing SQL update | +| Automatic human approval after acceptance | Same helper around the status update | +| Explicit approval and rejection | Same helper around the status update | +| Slack self-service access request | Existing platform transaction, before the request insert | +| Company pause, resume, archive, or deletion | Company-level retention notification inside the existing transaction | +| Budget-driven company pause or resume | Same company notification in the accounting transaction | +| Keep/unkeep, manual archive/revive, accepted archive proposal | Existing typed retention notification now routes join_request to its worker | + +Provider calls, membership grants, and unrelated route behavior stay outside the +small SQL mutation helper. Nested callers retain the existing outer transaction +settlement and uncertain-outcome protocol. The access router itself requires a +root database connection; it is not wrapped in a caller transaction. + +API-key consumption only permits an already-approved request. It cannot create +pending retention work and does not acquire a new wrapper or signal. Invite +acceptance that reuses a human request does not update that request. Revocation, +expiry, and acceptedAt changes do not change the feed predicate. Invite/company +identity is immutable in supported routes; company deletion is the invite delete +owner. A future producer that changes these facts must join the same contract. +Process-local signals do not cover independent SQL writers or another instance. + +## Lifecycle and limits + +Startup reconstructs pending deadlines. The worker is registered only when the +existing heartbeat maintenance switch is enabled. Writes during drain retain a +wake without passive SQL. Failed work retains its retry across drain. Shutdown +joins an admitted archive and stops before subsequent candidates; restart picks +up the durable remainder. There is no new queue table or permanent DB listener. +The in-process deadline is not a Cloud alarm. Existing unsupported-future-work +sleep blockers remain in place. + +## Earlier verification + +Seventeen new real-PostgreSQL cases pass. They cover actual invite acceptance, +rejection, human approval, gateway replay, and Slack access requests; empty and +resolved-history quiet; saved deadlines; startup reconstruction; keep and manual +archival controls; company pause/resume; unchanged expired/revoked invite policy; +normal versus maintenance feeds; nested commit/rollback; first writes and failed +work during drain; concurrent source updates; 101-row microsecond paging with a +full failed page; and shutdown/restart. + +Another 139 consumer cases pass: 71 invitation/startup cases, 21 approval worker +cases, and 47 shared retention/attention/budget cases. This is 156 selected tests, +not a full local suite claim. The first consumer command used a nonexistent +retention test filename and ran only the approval suite; the corrected selection +ran the actual retention suite against its dedicated external database. + +Initial test failures exposed fixture assumptions: the access router cannot run +on a transaction handle; unresolved recovery cannot be expected to become idle; +Slack links require actual endpoint/principal ownership rows; and shutdown tests +must establish admission and deterministic candidate order. Those fixtures were +corrected without weakening assertions or production checks. The first replay +fixture lacked required gateway credentials. The final fixture supplies test-only +credentials. A reused transaction spy retained admission calls during the quiet +assertion; restoring it before the quiet measurement fixed the measurement. + +Migration 0387 applies, the full migration set replays, and the concurrent index +statement replays idempotently. Migration safety and numbering checks pass; the +five newest schema snapshots are retained. The actual next-deadline query with +40,000 resolved requests and one pending request uses the pending index and +executes in 0.017 ms locally. The probe rolls back its data. This is query-plan +evidence, not production latency or measured savings. Workspace typechecking and the production build pass. Fresh PR CI/review remain +required before readiness. + + +## Current review repair and verification + +The duplicate approval/join archive loop is now one `createDecisionRetentionSweep` +implementation. Each owner supplies its candidate ordering boundary, page, locked current-state +lookup and next deadline. This keeps eligibility, company/invite joins and +source-row locks with their owner while sharing cursor/error/retention behavior. +The existing 100-row budget, PostgreSQL text cursor and failure rotation remain. +Both workers coalesce notification-driven global reads under a thirty-second +aggregate budget; saved deadlines preempt it and no idle scan is scheduled. + +Manual keep/archive/revive can materialize an unseen join request in the existing +retention write transaction, as approvals already can. The company-scoped source +is locked, its real activity is read, and existing versioned state is synchronized. +Authorization, archive choices and other source kinds retain their contracts. +No route bookkeeping, queue table, permanent listener or Cloud exemption is added. + +Five real regressions failed before repair: three unseen-request controls returned +404 and both owners rounded a saved microsecond deadline down. All pass. The first +two-owner run passed 50 cases but failed two old producer expectations that used +a truncated JavaScript Date. They now compute the expected scheduler millisecond +independently from integer database microseconds; exact assertions remain. + +The first missing-lock mutation exposed a weak test: its timing-only writer gate +released before proving the sweep had reached the lock, so it still passed without +`FOR UPDATE`. The strengthened test observes the actual PostgreSQL lock wait before +releasing the writer. It passes with the production lock and fails with zero waiters +when that lock is removed. The production source was restored in `finally` before +commit. A separate archive-first committed-order case preserves the existing +archived-stays-archived behavior and verifies explicit revival. + +All 340 cases in nine complete suites pass at `405c7dcdef`, including all 31 +approval and 22 join worker cases. Server typecheck and production build pass. +Normal integration of parent `0b0e844b77` changes only four independently merged +Runner E2E accounting files and the ledger; runtime and acceptance sources remain +byte-identical. Existing results retain their tested revision. + +Generated concurrent/idempotent migration `0393_open_grandmaster.sql` adds only the +pending-request index. All 391 parent journal entries and historical SQL files are +unchanged; five generated snapshots remain (0389–0393). Migration safety/numbering, +schema drift and actual apply/replay on six isolated PostgreSQL fixtures pass. +Earlier setup, timing, mutation and CI failures remain recorded here and in the +stack ledger. Fresh final-head CI, security and both 5/5 reviews remain required. + + +## Retry progress under continuing arrivals + +Each scan cycle saves the final due `(updated_at, id)` tuple, at full database +precision. Its pages stop at that boundary. Later due rows cannot keep extending +the cycle and postpone failed rows forever. The cursor still advances past an +individual failure, so one bad request—or an entire failed page—does not block +later healthy rows. Once that finite cycle ends, the next cycle revisits failures. +There is no new durable queue or per-row retry registry. Restart reconstructs +eligibility from existing rows. + +Two real PostgreSQL regressions failed before this repair: continued newer overdue +arrivals left a recovered approval and join request behind the cursor. Both now +archive in the next cycle. All 55 worker cases pass, including the existing +100-failure-page progress and microsecond-cursor cases. The nine-suite run passed +341 cases but failed the attention suite's cleanup hook after its 16,400-run case: +`delete from heartbeat_runs` exceeded thirty seconds. After vacuum/analyze of only +the disposable attention fixture, all 31 attention cases pass without code or +timeout changes. The other eight suites passed 311 cases in the original run; +342 distinct cases pass across those complete-suite executions. This demonstrates +recovery, not a proven explanation of the cleanup slowdown. Server typecheck and +production build pass at `df1815fbba`; the subsequent master integration has an +identical complete source tree. + +Earlier CI at `df1d4d0` also failed the unchanged decision-expiry restart test: +its earliest wake was a thirty-second retry rather than the saved expiry. The +complete unchanged expiry suite passes all twenty cases locally. Its CI timing +cause remains unproven; it is not labeled infrastructure or silently discarded. +Fresh final-head CI and reviews must pass before merging. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index a5bc0de3881..b032fcbb481 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -17,12 +17,8 @@ Do not mark an item done from a timer removal or a green check alone. | 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending | | 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Approval retention [#17497](https://github.com/paperclipai/paperclip/pull/17497) and archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged; other source retention pending | | 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) merged | - - - | 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending | | 07 | Secret proposals, tool reviews, gateway tokens | Separate expiry/scrub/recovery owners | Slices 02–07 merged: [secret proposals](https://github.com/paperclipai/paperclip/pull/17446), [credentials](https://github.com/paperclipai/paperclip/pull/17450), [review expiry](https://github.com/paperclipai/paperclip/pull/17453), [approved recovery](https://github.com/paperclipai/paperclip/pull/17454), [interrupted claims](https://github.com/paperclipai/paperclip/pull/17458), [projection](https://github.com/paperclipai/paperclip/pull/17468) | - | 08 | GitHub connection events and continuity | Event discovery/replay and credential continuity separately | Pending | | 09 | Merged-PR confirmations | Candidate writes, webhook activation where supported, outstanding-candidate fallback | Pending | | 10 | External-object refresh | Configuration changes, refresh deadlines, source events | Pending | @@ -41,7 +37,8 @@ The first PR targets master. Each following PR targets its predecessor branch so its diff shows only its own reason. Parent changes must be propagated before final verification. Record the exact checked head and base for every PR. All PRs must finish with passing checks, fresh 5/5 Greptile review, and no unresolved -findings. This request does not authorize merging the stack. +findings. The later user authorization permits merges after current-head Greptile +5/5, a passing Ponytail review when present, and passing CI, in dependency order. Reuse the delivery coordinator, transaction notifications, and shared scheduler. Existing durable rows own work; no table per loop. Keep domain/provider behavior @@ -1083,3 +1080,55 @@ accounting files. Its entire runtime and acceptance sources match the tested/ reviewed `a34dc50de3` tree (136 cases, server types/build); migration metadata is unchanged. No local checks were repeated on unchanged source. Fresh final-head CI/reviews are required against master; all remaining checklist duties persist. + +## Join request retention implementation + +Terminal workspace pagination is [#17505](https://github.com/paperclipai/paperclip/pull/17505), +head `28a197cdfd`, stacked on #17497. It has fresh 5/5 review without unresolved +findings. The full CI matrix is running. Workspace typecheck/build and all 87 +selected service/worker tests passed locally. + +Join retention is implemented on `codex/sleep-15-join-retention`, based on #17505. +The worker preserves pending request/invite/company eligibility, manual controls, +transaction ownership, microsecond paging, and bounded recovery. All supported +writers are audited; approved-only API-key consumption is deliberately exempt. +See [join retention](../idle-join-request-retention.md). Seventeen new real-DB +lifecycle cases and 139 consumers pass (156 total). Initial fixture and command +failures are recorded in that document. Index generation, apply, replay, safety, +numbering, and an actual-query plan with 40,000 resolved requests pass. Workspace +typecheck/build and remote publication/CI/review remain in progress. No slice is +claimed ready based only on local tests, and no merge is authorized. + +After join retention, migrate the remaining attention source owners separately: +decisions, interactions, recovery actions, reviews, and other source kinds. Keep +the full original server inventory and later aggregate-tick retirement in scope. + + +Slice 15 review repair: approval and join retention share the bounded archive +sweep; owners retain eligibility and source-row locks. SQL deadlines round up +from database precision. Unseen join controls materialize state atomically in +the existing retention boundary. Five real regressions failed then passed. The +initial timing-only concurrency test failed to detect a missing lock; it now +observes the actual database wait, passes normally and rejects lock removal. +Archive-first committed activity remains archived until explicit revival. + +All 340 cases across nine complete suites (31 approval/22 join worker cases), +server types/build, migration checks and schema drift pass at `405c7dcdef`. +All six isolated database fixtures pass actual migration apply/replay. Index 0393 +is generated after 391 unchanged parent entries; five snapshots remain. Parent +`0b0e844b77` adds only four Runner E2E accounting files and ledger changes; runtime +and acceptance sources are unchanged. Fresh publication checks/reviews required. + + +Slice 15 final master integration: #17505 merged as `d23dd19750` after both +final-head 5/5 reviews and all 58 accepted checks. Join retention now targets +master and integrates that landed parent without changing the tested source tree. +The shared retention scan saves a finite upper tuple so continuing arrivals cannot +starve failed rows; both real arrival regressions failed then passed. All 55 worker +cases pass. Eight consumer suites passed 311 cases; the ninth attention suite had +a cleanup-hook timeout and then passed all 31 after disposable-fixture vacuuming. +The unchanged expiry suite passes twenty local cases after a separate CI deadline +assertion failure; neither passing rerun is presented as a proven root cause. +Server types/build pass. See the join-retention document for the full audit. +Fresh exact-head reviews and CI remain required. All original duties remain open +until their own accepted merge, including native-harness backup expiration. diff --git a/packages/db/src/client.ts b/packages/db/src/client.ts index 2b987eb9720..4823515f5fd 100644 --- a/packages/db/src/client.ts +++ b/packages/db/src/client.ts @@ -511,6 +511,7 @@ const MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS = new Set([ "0390_salty_killraven.sql", "0391_third_roulette.sql", "0392_small_frightful_four.sql", + "0393_open_grandmaster.sql", "0384_funny_adam_warlock.sql", diff --git a/packages/db/src/migrations/0393_open_grandmaster.sql b/packages/db/src/migrations/0393_open_grandmaster.sql new file mode 100644 index 00000000000..dd3fa35bc3c --- /dev/null +++ b/packages/db/src/migrations/0393_open_grandmaster.sql @@ -0,0 +1 @@ +CREATE INDEX CONCURRENTLY IF NOT EXISTS "join_requests_pending_retention_idx" ON "join_requests" USING btree ("updated_at","id") WHERE "join_requests"."status" = 'pending_approval'; \ No newline at end of file diff --git a/packages/db/src/migrations/meta/0388_snapshot.json b/packages/db/src/migrations/meta/0393_snapshot.json similarity index 99% rename from packages/db/src/migrations/meta/0388_snapshot.json rename to packages/db/src/migrations/meta/0393_snapshot.json index c7cd58d2a5a..70b260ad9aa 100644 --- a/packages/db/src/migrations/meta/0388_snapshot.json +++ b/packages/db/src/migrations/meta/0393_snapshot.json @@ -1,6 +1,6 @@ { - "id": "4f21a636-735f-4dce-9ab8-0583f3bc7b79", - "prevId": "de27fdf8-430a-443a-9502-725f92c48892", + "id": "e1ddc6f7-6335-4b3d-aeec-a5df27793874", + "prevId": "6974dd25-b8e9-4143-b4b7-ee5a5c3ff54a", "version": "7", "dialect": "postgresql", "tables": { @@ -5721,6 +5721,28 @@ } }, "indexes": { + "approvals_pending_retention_idx": { + "name": "approvals_pending_retention_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"approvals\".\"status\" = 'pending'", + "concurrently": false, + "method": "btree", + "with": {} + }, "approvals_company_status_type_idx": { "name": "approvals_company_status_type_idx", "columns": [ @@ -24058,6 +24080,50 @@ "method": "btree", "with": {} }, + "decisions_running_recovery_idx": { + "name": "decisions_running_recovery_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'decided' AND \"decisions\".\"execution_status\" = 'running'", + "concurrently": false, + "method": "btree", + "with": {} + }, + "decisions_open_expiry_idx": { + "name": "decisions_open_expiry_idx", + "columns": [ + { + "expression": "expires_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"decisions\".\"status\" = 'open'", + "concurrently": false, + "method": "btree", + "with": {} + }, "decisions_pending_continuation_idx": { "name": "decisions_pending_continuation_idx", "columns": [ @@ -32063,6 +32129,28 @@ "method": "btree", "with": {} }, + "grok_dispatch_prepared_idx": { + "name": "grok_dispatch_prepared_idx", + "columns": [ + { + "expression": "created_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"grok_bot_dispatches\".\"status\" = 'prepared'", + "concurrently": false, + "method": "btree", + "with": {} + }, "grok_dispatch_company_status_idx": { "name": "grok_dispatch_company_status_idx", "columns": [ @@ -41857,6 +41945,28 @@ } }, "indexes": { + "join_requests_pending_retention_idx": { + "name": "join_requests_pending_retention_idx", + "columns": [ + { + "expression": "updated_at", + "isExpression": false, + "asc": true, + "nulls": "last" + }, + { + "expression": "id", + "isExpression": false, + "asc": true, + "nulls": "last" + } + ], + "isUnique": false, + "where": "\"join_requests\".\"status\" = 'pending_approval'", + "concurrently": false, + "method": "btree", + "with": {} + }, "join_requests_invite_unique_idx": { "name": "join_requests_invite_unique_idx", "columns": [ diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 8536f384cb0..49d5be4f242 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -2738,6 +2738,13 @@ "when": 1793143372644, "tag": "0392_small_frightful_four", "breakpoints": true + }, + { + "idx": 393, + "version": "7", + "when": 1793145214502, + "tag": "0393_open_grandmaster", + "breakpoints": true } ] } \ No newline at end of file diff --git a/packages/db/src/schema/join_requests.ts b/packages/db/src/schema/join_requests.ts index 3813c27db78..43a58726d45 100644 --- a/packages/db/src/schema/join_requests.ts +++ b/packages/db/src/schema/join_requests.ts @@ -31,6 +31,8 @@ export const joinRequests = pgTable( updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), }, (table) => ({ + pendingRetentionIdx: index("join_requests_pending_retention_idx").on(table.updatedAt, table.id) + .where(sql`${table.status} = 'pending_approval'`), inviteUniqueIdx: uniqueIndex("join_requests_invite_unique_idx").on(table.inviteId), companyStatusTypeCreatedIdx: index("join_requests_company_status_type_created_idx").on( table.companyId, diff --git a/server/src/__tests__/approval-retention-work.test.ts b/server/src/__tests__/approval-retention-work.test.ts index 9315e401089..97724e6c1bf 100644 --- a/server/src/__tests__/approval-retention-work.test.ts +++ b/server/src/__tests__/approval-retention-work.test.ts @@ -145,6 +145,17 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect((await db.select().from(approvals).where(eq(approvals.id, other.id)))[0].status).toBe("pending"); }); + it("rounds a submillisecond approval deadline up to the next scheduler millisecond", async () => { + const row = await create(), floor = Date.now() + 120_000; + const activity = new Date(floor - delay).toISOString(); + await db.execute(sql`update approvals set updated_at = ${activity}::timestamptz + interval '456 microseconds' where id = ${row.id}`); + const service = approvalRetentionService(db); + expect(await service.next()).toBe(floor + 1); + expect(await service.sweep(new Date(floor))).toEqual({ archived: 0 }); + expect(await state(row.id)).toBeNull(); + expect(await service.sweep(new Date(floor + 1))).toEqual({ archived: 1 }); + }); + it("recovers old unseen approvals on startup and leaves completed history quiet", async () => { const due = await create(old()), completed = await create(old()); await approvalRecords(db).resolveApproval(completed.id, "approved", "board-user", null); @@ -334,6 +345,26 @@ const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; expect(await service.sweep()).toEqual({ archived: 1 }); expect(await service.next()).toBeNull(); }); + it("retries a failed approval while newer overdue pages keep arriving", async () => { + const at = old(); + const [failed] = await db.insert(approvals).values({ companyId, type: "request_board_approval", + payload: { retentionTestFailure: true }, createdAt: at, updatedAt: at }).returning(); + const append = async (offset: number) => db.insert(approvals).values(Array.from({ length: 100 }, (_, i) => ({ + companyId, type: "request_board_approval", payload: {}, createdAt: at, + updatedAt: new Date(at.getTime() + offset + i), + }))); + await append(1); await failArchives(); + const service = approvalRetentionService(db); + await expect(service.sweep()).rejects.toThrow("Failed query"); + await append(1000); await service.sweep(); + await append(2000); + await db.execute(sql`drop trigger approval_retention_fail on decision_retention`); + await service.sweep(); + expect((await state(failed.id))?.archivedAt).toBeInstanceOf(Date); + while (await service.next() !== null) await service.sweep(); + expect(await service.next()).toBeNull(); + }); + it("reconstructs startup obligations while paused and performs no passive SQL", async () => { const row = await create(old()); draining = true; await start(); diff --git a/server/src/__tests__/join-request-retention-work.test.ts b/server/src/__tests__/join-request-retention-work.test.ts new file mode 100644 index 00000000000..17950576342 --- /dev/null +++ b/server/src/__tests__/join-request-retention-work.test.ts @@ -0,0 +1,377 @@ +import { createSlackIdentityLinks } from "../services/chat-channels/slack/identity-links.js"; +import { createHash, randomUUID } from "node:crypto"; +import express from "express"; +import request from "supertest"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, agents, authUsers, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, companies, companyMemberships, companySecretProviderConfigs, companySecrets, companySecretVersions, createDb, decisionArchiveNotificationOutbox, + decisionRetention, invites, joinRequests, principalPermissionGrants, toolApplications, toolConnections, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { joinRequestRetentionService, registerJoinRequestRetentionWork } from "../services/join-request-retention.js"; +import { decisionRetentionService, DEFAULT_DECISION_ARCHIVE_DAYS } from "../services/decision-retention.js"; +import { attentionService } from "../services/attention.js"; +import { companyService } from "../services/companies.js"; +import { writeJoinRequestChange } from "../services/retention-work-notifications.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_JOIN_RETENTION_TEST_DATABASE_URL"); +const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport(); +const delay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; +(support.supported ? describe : describe.skip)("join request retention worker", () => { + let temporary: Awaited> | undefined; + let db: ReturnType, companyId: string, userId: string; + let coordinator: ReturnType | undefined; + let draining: boolean; + let accessRoutes: typeof import("../routes/access.js").accessRoutes; + const errors: unknown[] = []; + beforeAll(async () => { + ({ accessRoutes } = await import("../routes/access.js")); + if (!externalUrl) temporary = await startEmbeddedPostgresTestDatabase("join-retention-"); + db = createDb(externalUrl ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 30_000); + beforeEach(async () => { + vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "0123456789abcdef0123456789abcdef"); + companyId = randomUUID(); userId = randomUUID(); draining = false; errors.length = 0; + await db.insert(companies).values({ id: companyId, name: "Join retention", issuePrefix: `J${companyId.slice(0, 6)}` }); + await db.insert(authUsers).values({ id: userId, name: "Owner", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, membershipRole: "owner", status: "active" }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs(); + await db.execute(sql`drop trigger if exists join_retention_fail on decision_retention`); + await db.execute(sql`drop function if exists join_retention_fail()`); + await db.delete(decisionArchiveNotificationOutbox); await db.delete(decisionRetention); await db.delete(activityLog); + await db.delete(chatIdentityLinks); await db.delete(chatExternalPrincipals); await db.delete(chatEndpoints); + await db.delete(toolConnections); await db.delete(toolApplications); await db.delete(agents); + await db.delete(joinRequests); await db.delete(invites); await db.delete(principalPermissionGrants); + await db.delete(companySecretVersions); await db.delete(companySecrets); await db.delete(companySecretProviderConfigs); + await db.delete(companyMemberships); await db.delete(authUsers); await db.delete(companies); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + const old = () => new Date(Date.now() - delay - 60_000); + const state = (id: string) => decisionRetentionService(db).getState(companyId, "join_request", id); + async function exactDeadline(id: string) { + const [row] = await db.select({ micros: sql`(extract(epoch from ${joinRequests.updatedAt}) * 1000000)::bigint::text` }) + .from(joinRequests).where(eq(joinRequests.id, id)); + return Number((BigInt(row!.micros) + 999n) / 1000n) + delay; + } + const authActor = () => ({ type: "board" as const, source: "local_implicit" as const, userId, companyIds: [companyId], isInstanceAdmin: true, + memberships: [{ companyId, membershipRole: "owner", status: "active" }] }); + const mutation = (id: string) => ({ companyId, sourceKind: "join_request" as const, sourceId: id, authActor: authActor(), + actor: { actorType: "user" as const, actorId: userId, agentId: null, userId, runId: null, agentApiKeyId: null, responsibleUserId: userId } }); + async function invite(database: Db = db) { + const token = randomUUID(); + const [row] = await database.insert(invites).values({ companyId, tokenHash: createHash("sha256").update(token).digest("hex"), + allowedJoinTypes: "both", expiresAt: new Date(Date.now() + 86_400_000), invitedByUserId: userId }).returning(); + return { ...row, token }; + } + async function create(at = new Date(), database: Db = db) { + const invitation = await invite(database); + const [row] = await writeJoinRequestChange(database, tx => tx.insert(joinRequests).values({ companyId, inviteId: invitation.id, + requestType: "agent", requestIp: "127.0.0.1", agentName: "Pending agent", createdAt: at, updatedAt: at }).returning()); + return row; + } + function app(database: Db = db) { + const server = express(); server.use(express.json()); + server.use((req, _res, next) => { req.actor = authActor(); next(); }); + server.use("/api", accessRoutes(database, { deploymentMode: "authenticated", deploymentExposure: "private", + bindHost: "127.0.0.1", allowedHostnames: [] })); + server.use((error: { status?: number; message?: string }, _req: express.Request, res: express.Response, _next: express.NextFunction) => { + res.status(error.status ?? 500).json({ error: error.message }); + }); + return server; + } + async function accept(token: string, database: Db = db) { + const response = await request(app(database)).post(`/api/invites/${token}/accept`) + .send({ requestType: "agent", agentName: "New agent", adapterType: "claude_local" }); + expect(response.status, JSON.stringify(response.body)).toBe(202); + const [row] = await database.select().from(joinRequests).where(eq(joinRequests.companyId, companyId)); + return row; + } + async function start(service = joinRequestRetentionService(db)) { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => errors.push(error) }); + await registerJoinRequestRetentionWork(coordinator, service, () => !draining).ready; + return service; + } + async function settle() { + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(idleWorkSnapshot().active).toBe(0); }, { timeout: 5000 }); + } + async function quiet() { + await settle(); expect(coordinator!.nextWakeAt()).toBeNull(); + const reads = vi.spyOn(db, "select"), executes = vi.spyOn(db, "execute"), writes = vi.spyOn(db, "transaction"); + await vi.advanceTimersByTimeAsync(300_000); + expect(reads).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); executes.mockRestore(); writes.mockRestore(); + } + async function archived(id: string) { + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect((await state(id))?.archivedAt).toBeInstanceOf(Date); }, { timeout: 5000 }); + } + + it("wakes from real invite acceptance, waits for the saved deadline, and disarms on rejection", async () => { + await start(); await quiet(); + const invitation = await invite(); const row = await accept(invitation.token); + await settle(); expect(coordinator!.nextWakeAt()).toBe(await exactDeadline(row.id)); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(300_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + const response = await request(app()).post(`/api/companies/${companyId}/join-requests/${row.id}/reject`).send({}); + expect(response.status, JSON.stringify(response.body)).toBe(200); await quiet(); + expect(await state(row.id)).toBeNull(); + }); + + it("archives unseen pending requests at the actual deadline without changing approval status", async () => { + await start(); await quiet(); const deadline = Date.now() + 600_000; + const row = await create(new Date(deadline - delay)); await settle(); + expect(coordinator!.nextWakeAt()).toBe(deadline); expect(await state(row.id)).toBeNull(); + const window = deadline - Date.now() - 30_000; expect(window).toBeGreaterThan(0); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(window); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await vi.advanceTimersByTimeAsync(60_000); await archived(row.id); await quiet(); + expect((await db.select().from(joinRequests))[0].status).toBe("pending_approval"); + expect(await db.select().from(decisionArchiveNotificationOutbox)).toHaveLength(0); + }); + + it("rounds a submillisecond join deadline up to the next scheduler millisecond", async () => { + const row = await create(), floor = Date.now() + 120_000; + const activity = new Date(floor - delay).toISOString(); + await db.execute(sql`update join_requests set updated_at = ${activity}::timestamptz + interval '456 microseconds' where id = ${row.id}`); + const service = joinRequestRetentionService(db); + expect(await service.next()).toBe(floor + 1); + expect(await service.sweep(new Date(floor))).toEqual({ archived: 0 }); + expect(await state(row.id)).toBeNull(); + expect(await service.sweep(new Date(floor + 1))).toEqual({ archived: 1 }); + }); + + it.each(["keep", "archive", "revive"] as const)("applies %s to an unseen young request without reading the feed", async action => { + const row = await create(); expect(await state(row.id)).toBeNull(); + const retention = decisionRetentionService(db); + const result = action === "keep" ? await retention.setKeep({ ...mutation(row.id), keep: true }) + : action === "archive" ? await retention.archive(mutation(row.id)) : await retention.revive(mutation(row.id)); + expect(result).toMatchObject({ sourceKind: "join_request", sourceId: row.id, sourceActivityAt: row.updatedAt, keep: action === "keep" }); + expect(result.archivedAt).toEqual(action === "archive" ? expect.any(Date) : null); + expect(await state(row.id)).toMatchObject({ id: result.id }); + }); + + it("reconstructs pending requests at startup and ignores resolved history", async () => { + const due = await create(old()), resolved = await create(old()); + await db.update(joinRequests).set({ status: "approved" }).where(eq(joinRequests.id, resolved.id)); + await start(); await archived(due.id); await quiet(); expect(await state(resolved.id)).toBeNull(); + }); + + it("keeps revoked and expired invite requests visible under the existing retention policy", async () => { + const row = await create(old()); await db.update(invites).set({ expiresAt: old(), revokedAt: old() }); + const feed = await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + expect(feed.items.some(item => item.subject.id === row.id)).toBe(true); + await start(); await archived(row.id); await quiet(); + }); + + it("waits for company activation and disarms when the company pauses", async () => { + const row = await create(old()); await companyService(db).update(companyId, { status: "paused" }); + await start(); await quiet(); expect(await state(row.id)).toBeNull(); + await companyService(db).update(companyId, { status: "active" }); await archived(row.id); await quiet(); + await create(); await settle(); expect(coordinator!.nextWakeAt()).not.toBeNull(); + await companyService(db).update(companyId, { status: "paused" }); await quiet(); + }); + + it("honors keep and schedules an old request again when unkept", async () => { + const row = await create(old()); await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + await decisionRetentionService(db).setKeep({ ...mutation(row.id), keep: true }); + await start(); await quiet(); + await decisionRetentionService(db).setKeep({ ...mutation(row.id), keep: false }); await archived(row.id); await quiet(); + }); + + it("honors manual archival and reactivates on revive", async () => { + const row = await create(old()); await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + await decisionRetentionService(db).archive(mutation(row.id)); await start(); await quiet(); + await decisionRetentionService(db).revive(mutation(row.id)); await archived(row.id); await quiet(); + }); + + it("preserves normal attention reads but omits join requests from maintenance", async () => { + const row = await create(old()); + const excluded = await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true, excludeSources: ["join_request"] }); + expect(excluded.items.some(item => item.sourceKind === "join_request")).toBe(false); expect(await state(row.id)).toBeNull(); + const feed = await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); + expect(feed.items.some(item => item.subject.id === row.id)).toBe(true); + }); + + it("defers nested request writes until outer commit and releases rolled-back writes", async () => { + await start(); await quiet(); + await expect(db.transaction(async tx => { + await create(new Date(), tx as unknown as Db); + await vi.advanceTimersByTimeAsync(120_000); expect(coordinator!.nextWakeAt()).toBeNull(); throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); expect(await db.select().from(joinRequests)).toHaveLength(0); + let row: typeof joinRequests.$inferSelect; + await db.transaction(async tx => { + row = await create(new Date(), tx as unknown as Db); + await vi.advanceTimersByTimeAsync(120_000); expect(coordinator!.nextWakeAt()).toBeNull(); + }); + await settle(); expect(coordinator!.nextWakeAt()).toBe(row!.updatedAt.getTime() + delay); + }); + + it("retains a first write during drain without passive SQL", async () => { + await start(); await quiet(); draining = true; const row = await create(old()); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + draining = false; await archived(row.id); await quiet(); + }); + + it("retains a failed archival through drain until the database recovers", async () => { + const row = await create(old()); + await db.execute(sql`create function join_retention_fail() returns trigger language plpgsql as $$ begin + raise exception 'retention unavailable'; end $$`); + await db.execute(sql`create trigger join_retention_fail before insert or update on decision_retention for each row execute function join_retention_fail()`); + const service = joinRequestRetentionService(db); const sweep = vi.spyOn(service, "sweep"); + await start(service); await vi.waitFor(() => expect(errors.length).toBeGreaterThan(0)); draining = true; + await Promise.allSettled(sweep.mock.results.map(result => result.value)); + await vi.advanceTimersByTimeAsync(30_000); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000); + expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await db.execute(sql`drop trigger join_retention_fail on decision_retention`); draining = false; + await archived(row.id); await quiet(); + for (const error of errors) expect(error).toMatchObject({ cause: { message: "retention unavailable" } }); errors.length = 0; + }); + + it("wakes from the Slack access-request producer and deduplicates repeated requests", async () => { + await start(); await quiet(); const token = randomUUID(), endpointId = randomUUID(); + const [agent] = await db.insert(agents).values({ companyId, name: "Chat agent", role: "engineer", status: "active", adapterType: "codex_local" }).returning(); + const [application] = await db.insert(toolApplications).values({ companyId, name: "Slack", type: "chat" }).returning(); + const [connection] = await db.insert(toolConnections).values({ companyId, applicationId: application.id, name: "Slack", uid: randomUUID(), transport: "chat_sdk", connectionPurpose: "channel" }).returning(); + await db.insert(chatEndpoints).values({ id: endpointId, companyId, connectionId: connection.id, provider: "slack", publicId: randomUUID(), assignedAgentId: agent.id, status: "active" }); + const [principal] = await db.insert(chatExternalPrincipals).values({ companyId, provider: "slack", providerAccountId: "team", externalId: randomUUID() }).returning(); + await db.insert(chatIdentityLinks).values({ companyId, endpointId, principalId: principal.id, + confirmationTokenHash: createHash("sha256").update(token).digest("hex"), expiresAt: new Date(Date.now() + 600_000) }); + const links = createSlackIdentityLinks(db, { + previewIdentityLink: async () => ({ selfService: true, canConfirm: false, companyId, endpointId }), + stageProviderEffect: vi.fn(async () => null), + }); + expect(await links.requestIdentityAccess(token, userId, "127.0.0.1")).toEqual({ status: "pending_approval" }); + await settle(); const [row] = await db.select().from(joinRequests); + expect(coordinator!.nextWakeAt()).toBe(await exactDeadline(row.id)); + await links.requestIdentityAccess(token, userId, "127.0.0.1"); await settle(); + expect(await db.select().from(joinRequests)).toHaveLength(1); + }); + + it("disarms after the real approval route resolves a human request", async () => { + const requester = randomUUID(); + await db.insert(authUsers).values({ id: requester, name: "Joiner", email: `${requester}@example.test`, createdAt: new Date(), updatedAt: new Date() }); + const row = await create(); + await db.update(joinRequests).set({ requestType: "human", requestingUserId: requester }).where(eq(joinRequests.id, row.id)); + await start(); await settle(); expect(coordinator!.nextWakeAt()).not.toBeNull(); + const response = await request(app()).post(`/api/companies/${companyId}/join-requests/${row.id}/approve`).send({}); + expect(response.status, JSON.stringify(response.body)).toBe(200); await quiet(); + expect((await db.select().from(joinRequests))[0].status).toBe("approved"); + }); + + it("rechecks activity after waiting for a concurrent request writer", async () => { + const row = await create(old()); let release!: () => void, locked!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const admitted = new Promise(resolve => { locked = resolve; }); + const write = db.transaction(async tx => { + await tx.execute(sql`select id from join_requests where id = ${row.id} for update`); locked(); await gate; + await writeJoinRequestChange(tx as unknown as Db, owner => owner.update(joinRequests).set({ updatedAt: new Date() }).where(eq(joinRequests.id, row.id))); + }); + await admitted; const sweep = joinRequestRetentionService(db).sweep(); + try { + await vi.waitFor(async () => { + const [row] = await db.execute(sql`select count(*)::int as waiting from pg_stat_activity + where datname = current_database() and pid <> pg_backend_pid() and wait_event_type = 'Lock' + and query like 'select%' and query like '%from "join_requests"%'`); + expect(row.waiting).toBe(1); + }); + } finally { release(); await write; await sweep; } + expect(await state(row.id)).toBeNull(); + }); + + it("keeps an archived request archived when activity commits after the sweep until explicitly revived", async () => { + const row = await create(old()), service = joinRequestRetentionService(db); + expect(await service.sweep()).toEqual({ archived: 1 }); + const archivedAt = (await state(row.id))!.archivedAt; + await writeJoinRequestChange(db, tx => tx.update(joinRequests).set({ updatedAt: new Date() }).where(eq(joinRequests.id, row.id))); + expect(await state(row.id)).toMatchObject({ archivedAt }); + expect(await service.next()).toBeNull(); + expect((await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true })).items + .some(item => item.subject.id === row.id)).toBe(false); + await decisionRetentionService(db).revive(mutation(row.id)); + expect((await state(row.id))!.archivedAt).toBeNull(); + expect(await service.next()).toBe(await exactDeadline(row.id)); + expect(await service.sweep()).toEqual({ archived: 0 }); + }); + + it("pages past failed requests with database timestamp precision and retries them later", async () => { + const ids: string[] = []; + for (let i = 0; i < 101; i++) ids.push((await create(old())).id); + ids.sort(); const last = ids.at(-1)!; + await db.execute(sql`update join_requests set updated_at = date_trunc('milliseconds', now() - interval '91 days') + interval '456 microseconds'`); + await db.execute(sql`create function join_retention_fail() returns trigger language plpgsql as $$ begin + if new.archived_at is not null and new.source_id <> TG_ARGV[0] then raise exception 'retention unavailable'; end if; + return new; end $$`); + await db.execute(sql`create trigger join_retention_fail before update on decision_retention for each row execute function join_retention_fail(${sql.raw("'" + last + "'")})`); + const service = joinRequestRetentionService(db); + await expect(service.sweep()).rejects.toThrow("Join request retention failed"); + expect(await service.sweep()).toEqual({ archived: 1 }); expect((await state(last))?.archivedAt).toBeInstanceOf(Date); + await db.execute(sql`drop trigger join_retention_fail on decision_retention`); + expect(await service.sweep()).toEqual({ archived: 100 }); expect(await service.sweep()).toEqual({ archived: 0 }); + expect(await service.next()).toBeNull(); + }); + + it("retries a failed request while newer overdue pages keep arriving", async () => { + const at = old(); const failed = await create(at); + const append = async (offset: number) => { + for (let i = 0; i < 100; i++) await create(new Date(at.getTime() + offset + i)); + }; + await append(1); + await db.execute(sql`create function join_retention_fail() returns trigger language plpgsql as $$ begin + if new.archived_at is not null and new.source_id = TG_ARGV[0] then raise exception 'retention unavailable'; end if; + return new; end $$`); + await db.execute(sql`create trigger join_retention_fail before update on decision_retention for each row execute function join_retention_fail(${sql.raw("'" + failed.id + "'")})`); + const service = joinRequestRetentionService(db); + await expect(service.sweep()).rejects.toThrow("Failed query"); + await append(1000); await service.sweep(); + await append(2000); + await db.execute(sql`drop trigger join_retention_fail on decision_retention`); + await service.sweep(); + expect((await state(failed.id))?.archivedAt).toBeInstanceOf(Date); + while (await service.next() !== null) await service.sweep(); + expect(await service.next()).toBeNull(); + }); + + it("joins an admitted archive on shutdown and leaves later candidates for restart", async () => { + const first = await create(new Date(old().getTime() - 1000)), second = await create(old()); + let release!: () => void, locked!: () => void; + const gate = new Promise(resolve => { release = resolve; }); + const acquired = new Promise(resolve => { locked = resolve; }); + const transaction = db.transaction.bind(db); + const transactionSpy = vi.spyOn(db, "transaction").mockImplementationOnce((write, options) => transaction(async tx => { + locked(); await gate; return write(tx); + }, options)); + const starting = start(); await acquired; + let stopped = false; const stopping = coordinator!.stop().then(() => { stopped = true; }); + try { await vi.advanceTimersByTimeAsync(500); expect(stopped).toBe(false); } finally { release(); } + await starting; await stopping; coordinator = undefined; transactionSpy.mockRestore(); + expect((await state(first.id))?.archivedAt).toBeInstanceOf(Date); expect(await state(second.id)).toBeNull(); + await start(); await archived(second.id); await quiet(); + }); + + + it("moves the saved deadline after a supported gateway invite replay", async () => { + const invitation = await invite(); await start(); await quiet(); + const body = { requestType: "agent", agentName: "Gateway", adapterType: "openclaw_gateway", + agentDefaultsPayload: { url: "ws://127.0.0.1:18789", disableDeviceAuth: true, headers: { "x-openclaw-token": "fixture-gateway-token-1234567890" } } }; + const first = await request(app()).post(`/api/invites/${invitation.token}/accept`).send(body); + expect(first.status, JSON.stringify(first.body)).toBe(202); await settle(); + const initial = coordinator!.nextWakeAt(); + await vi.advanceTimersByTimeAsync(60_000); + const replay = await request(app()).post(`/api/invites/${invitation.token}/accept`).send({ ...body, agentName: "Renamed gateway" }); + expect(replay.status, JSON.stringify(replay.body)).toBe(202); await settle(); + const rows = await db.select().from(joinRequests); expect(rows).toHaveLength(1); + expect(coordinator!.nextWakeAt()).toBe(rows[0].updatedAt.getTime() + delay); + expect(coordinator!.nextWakeAt()).toBeGreaterThan(initial!); + }); + +}); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index e1d793142a1..b08a2ed13e1 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,3 +1,4 @@ +import { joinRequestRetentionService } from "../services/join-request-retention.js"; import { approvalRetentionService } from "../services/approval-retention.js"; import { createSecretProposalsService } from "../services/secret-proposals.js"; import { logger } from "../middleware/logger.js"; @@ -331,6 +332,11 @@ vi.mock("../realtime/live-events-ws.js", () => ({ setupLiveEventsWebSocketServer: vi.fn(), })); +vi.mock("../services/join-request-retention.js", async importOriginal => ({ + ...await importOriginal(), + joinRequestRetentionService: vi.fn(() => ({ sweep: vi.fn(async () => ({ archived: 0 })), next: vi.fn(async () => null) })), +})); + vi.mock("../services/approval-retention.js", async importOriginal => ({ ...await importOriginal(), approvalRetentionService: vi.fn(() => ({ sweep: vi.fn(async () => ({ archived: 0 })), next: vi.fn(async () => null) })), @@ -548,6 +554,18 @@ describe("startServer feedback export wiring", () => { } }); + it.each([false, true])("registers join request retention only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.joinRequestRetention); + expect(Boolean(registration)).toBe(enabled); + if (enabled) { + const service = vi.mocked(joinRequestRetentionService).mock.results.at(-1)!.value; + await registration![1].run(new AbortController().signal); + expect(service.sweep).toHaveBeenCalled(); expect(service.next).toHaveBeenCalled(); + } + }); + it.each([false, true])("registers archive delivery separately from retention sweeps (scheduler %s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled, heartbeatSchedulerIntervalMs: 30000 })); let tick: (() => void) | undefined; diff --git a/server/src/index.ts b/server/src/index.ts index d74ea2b848f..a69a95b526d 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { joinRequestRetentionService, registerJoinRequestRetentionWork } from "./services/join-request-retention.js"; import { approvalRetentionService, registerApprovalRetentionWork } from "./services/approval-retention.js"; import { registerDecisionTargetWork } from "./services/decision-target-work.js"; import { registerDecisionEffectRecoveryWork } from "./services/decision-effect-recovery-work.js"; @@ -1812,7 +1813,7 @@ async function startServerWithDatabaseTeardown( // per company per tick. const page = await attentionService(db as any).list(company.id, { includeDismissed: true, - excludeSources: ["approval"], + excludeSources: ["approval", "join_request"], all: true, allowUnscopedAll: true, }); @@ -1862,6 +1863,8 @@ async function startServerWithDatabaseTeardown( () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("approval_retention", () => registerApprovalRetentionWork(deliveryWork, approvalRetentionService(db as any), () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("join_request_retention", () => registerJoinRequestRetentionWork(deliveryWork, joinRequestRetentionService(db as any), + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("retention_sweep", runRetentionSweep); }; diff --git a/server/src/routes/access.ts b/server/src/routes/access.ts index 526bd1a3b42..e1fe2566c3f 100644 --- a/server/src/routes/access.ts +++ b/server/src/routes/access.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork, writeJoinRequestChange } from "../services/retention-work-notifications.js"; import { createAgentLifecycle } from "../services/agent-lifecycle.js"; import { createHash, @@ -3136,7 +3137,7 @@ export function accessRoutes( const approvedAt = new Date(); const approvedByUserId = input.invite.invitedByUserId ?? (isLocalImplicit(input.req) ? "local-board" : null); - const approved = await db + const approved = await writeJoinRequestChange(db, tx => tx .update(joinRequests) .set({ status: "approved", @@ -3146,7 +3147,7 @@ export function accessRoutes( }) .where(eq(joinRequests.id, input.joinRequest.id)) .returning() - .then((rows) => rows[0] ?? null); + .then((rows) => rows[0] ?? null)); await logActivity(db, { companyId: input.companyId, @@ -3882,6 +3883,7 @@ export function accessRoutes( ) ); + await notifyRetentionWork(tx, "join_request"); const row = await tx .insert(joinRequests) .values({ @@ -3912,7 +3914,7 @@ export function accessRoutes( .then((rows) => rows[0]); return row; }) - : await db + : await writeJoinRequestChange(db, tx => tx .update(joinRequests) .set({ requestIp: requestIp(req), @@ -3935,7 +3937,7 @@ export function accessRoutes( }) .where(eq(joinRequests.id, replayJoinRequestId as string)) .returning() - .then((rows) => rows[0]); + .then((rows) => rows[0])); if (!created) { throw conflict("Join request not found"); @@ -4279,7 +4281,7 @@ export function accessRoutes( ); } - const approved = await db + const approved = await writeJoinRequestChange(db, tx => tx .update(joinRequests) .set({ status: "approved", @@ -4291,7 +4293,7 @@ export function accessRoutes( }) .where(eq(joinRequests.id, requestId)) .returning() - .then((rows) => rows[0]); + .then((rows) => rows[0])); await logActivity(db, { companyId, @@ -4338,7 +4340,7 @@ export function accessRoutes( if (existing.status !== "pending_approval") throw conflict("Join request is not pending"); - const rejected = await db + const rejected = await writeJoinRequestChange(db, tx => tx .update(joinRequests) .set({ status: "rejected", @@ -4349,7 +4351,7 @@ export function accessRoutes( }) .where(eq(joinRequests.id, requestId)) .returning() - .then((rows) => rows[0]); + .then((rows) => rows[0])); await logActivity(db, { companyId, diff --git a/server/src/services/approval-retention.ts b/server/src/services/approval-retention.ts index 855c81e413d..38e76012f85 100644 --- a/server/src/services/approval-retention.ts +++ b/server/src/services/approval-retention.ts @@ -1,7 +1,8 @@ import { approvalExecutionIssueCondition } from "./approval-visibility.js"; -import { and, asc, eq, isNull, lte, or, sql, type SQL } from "drizzle-orm"; +import { and, asc, desc, eq, isNull, lte, or, sql, type SQL } from "drizzle-orm"; import { approvals, companies, decisionRetention, type Db } from "@paperclipai/db"; -import { decisionRetentionService, DEFAULT_DECISION_ARCHIVE_DAYS } from "./decision-retention.js"; +import { DEFAULT_DECISION_ARCHIVE_DAYS } from "./decision-retention.js"; +import { createDecisionRetentionSweep } from "./decision-retention-sweep.js"; import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; import { registerDeadlineWork } from "./deadline-work.js"; import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; @@ -9,51 +10,34 @@ import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; const archiveDelay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; export function approvalRetentionService(db: Db) { - let cursor: { at: string; id: string } | undefined; - function candidates(database: Db, condition?: SQL) { - return database.select({ id: approvals.id, companyId: approvals.companyId, at: approvals.updatedAt, cursorAt: sql`${approvals.updatedAt}::text` }).from(approvals) + function candidates(database: Db, condition?: SQL, order = asc) { + return database.select({ id: approvals.id, companyId: approvals.companyId, at: approvals.updatedAt, cursorAt: sql`${approvals.updatedAt}::text`, deadlineAt: sql`ceil(extract(epoch from ${approvals.updatedAt}) * 1000 + ${archiveDelay})`.mapWith(Number) }).from(approvals) .innerJoin(companies, eq(companies.id, approvals.companyId)) .leftJoin(decisionRetention, and(eq(decisionRetention.companyId, approvals.companyId), eq(decisionRetention.sourceKind, "approval"), sql`${decisionRetention.sourceId} = ${approvals.id}::text`)) .where(and(sql`${approvals.status} = 'pending'`, eq(companies.status, "active"), approvalExecutionIssueCondition(), or(isNull(decisionRetention.id), eq(decisionRetention.keep, false)), isNull(decisionRetention.archivedAt), condition)) - .orderBy(asc(approvals.updatedAt), asc(approvals.id)); + .orderBy(order(approvals.updatedAt), order(approvals.id)); } - async function sweep(now = new Date(), signal?: AbortSignal) { - const cutoff = new Date(now.getTime() - archiveDelay); - const rows = await candidates(db, and(lte(approvals.updatedAt, cutoff), cursor - ? sql`(${approvals.updatedAt}, ${approvals.id}) > (${cursor.at}::timestamptz, ${cursor.id}::uuid)` : undefined)).limit(100); - let archived = 0; - const errors: unknown[] = []; - for (const candidate of rows) { - if (signal?.aborted) break; - try { - archived += await db.transaction(async tx => { - const database = tx as unknown as Db; - const [current] = await candidates(database, and(eq(approvals.id, candidate.id), lte(approvals.updatedAt, cutoff))) - .for("update", { of: approvals }); - if (!current) return 0; - const retention = decisionRetentionService(database); - const source = { sourceKind: "approval" as const, subject: { id: current.id }, activityAt: current.at.toISOString() }; - const states = await retention.syncItems(current.companyId, [source]); - const state = states.get(`approval:${current.id}`)!; - return retention.autoArchive({ companyId: current.companyId, now, items: [{ ...source, - keep: state.keep, archivedAt: state.archivedAt?.toISOString() ?? null, retentionVersion: state.version, - }] }); - }); - } catch (error) { errors.push(error); } - cursor = { at: candidate.cursorAt, id: candidate.id }; - } - if (rows.length < 100) cursor = undefined; - if (errors.length === 1) throw errors[0]; - if (errors.length) throw new AggregateError(errors, "Approval retention failed"); - return { archived }; - } - async function next() { - const [row] = await candidates(db).limit(1); - return row ? row.at.getTime() + archiveDelay : null; - } - return { sweep, next }; + return createDecisionRetentionSweep(db, { + sourceKind: "approval", failureMessage: "Approval retention failed", + end: async cutoff => { + const [row] = await candidates(db, lte(approvals.updatedAt, cutoff), desc).limit(1); + return row ? { at: row.cursorAt, id: row.id } : undefined; + }, + page: async (cutoff, cursor, end, limit) => candidates(db, and(lte(approvals.updatedAt, cutoff), + sql`(${approvals.updatedAt}, ${approvals.id}) <= (${end.at}::timestamptz, ${end.id}::uuid)`, cursor + ? sql`(${approvals.updatedAt}, ${approvals.id}) > (${cursor.at}::timestamptz, ${cursor.id}::uuid)` : undefined)).limit(limit), + lock: async (database, id, cutoff) => { + const [row] = await candidates(database, and(eq(approvals.id, id), lte(approvals.updatedAt, cutoff))) + .for("update", { of: approvals }); + return row; + }, + next: async () => { + const [row] = await candidates(db).limit(1); + return row?.deadlineAt ?? null; + }, + }); } export function registerApprovalRetentionWork( diff --git a/server/src/services/attention.ts b/server/src/services/attention.ts index 1e94cd26ec3..8eca680a4c1 100644 --- a/server/src/services/attention.ts +++ b/server/src/services/attention.ts @@ -1421,6 +1421,7 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions })); } + if (!options.excludeSources?.includes("join_request")) { const pendingJoins = await db .select({ id: joinRequests.id, @@ -1482,6 +1483,8 @@ export function attentionService(db: Db, serviceOptions: AttentionServiceOptions })); } + } + const recoveryRows = await db .select() .from(issueRecoveryActions) diff --git a/server/src/services/budgets.ts b/server/src/services/budgets.ts index 84e4ca1e6dc..aba74d6e0c6 100644 --- a/server/src/services/budgets.ts +++ b/server/src/services/budgets.ts @@ -1,4 +1,4 @@ -import { notifyRetentionWork } from "./retention-work-notifications.js"; +import { notifyCompanyRetentionWork, notifyRetentionWork } from "./retention-work-notifications.js"; import { addCents, compareCents, normalizeCents } from "@paperclipai/shared"; import { and, desc, eq, inArray, sql, or, isNull } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -297,7 +297,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.paused) return; - await notifyRetentionWork(db, "approval"); + await notifyCompanyRetentionWork(db); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync); await db @@ -354,7 +354,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync); const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.pauseReason !== "budget") return; - await notifyRetentionWork(db, "approval"); + await notifyCompanyRetentionWork(db); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId); await db .update(companies) diff --git a/server/src/services/chat-channels/slack/identity-links.ts b/server/src/services/chat-channels/slack/identity-links.ts index 1a37771fa01..6ce66825fb1 100644 --- a/server/src/services/chat-channels/slack/identity-links.ts +++ b/server/src/services/chat-channels/slack/identity-links.ts @@ -1,3 +1,4 @@ +import { notifyRetentionWork } from "../../retention-work-notifications.js"; import { createHash, randomBytes } from "node:crypto"; import { and, desc, eq, gt, or, sql } from "drizzle-orm"; @@ -97,6 +98,7 @@ export function createSlackIdentityLinks(db: Db, { previewIdentityLink, stagePro inviteType: "company_join", allowedJoinTypes: "human", acceptedAt: now, expiresAt: now, defaultsPayload: { human: { role: "operator" }, source: "slack_identity_link" }, }).returning({ id: invites.id }); + await notifyRetentionWork(tx, "join_request"); const [request] = await tx.insert(joinRequests).values({ inviteId: invite.id, companyId: preview.companyId, requestType: "human", status: "pending_approval", requestIp, requestingUserId: userId, requestEmailSnapshot: user.email, diff --git a/server/src/services/companies.ts b/server/src/services/companies.ts index f6fc7ed5de2..4f17b854339 100644 --- a/server/src/services/companies.ts +++ b/server/src/services/companies.ts @@ -1,4 +1,4 @@ -import { notifyRetentionWork } from "./retention-work-notifications.js"; +import { notifyCompanyRetentionWork } from "./retention-work-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { isRunnerApiResponseAsset } from "./asset-content-access.js"; import { deleteCompany } from "./company-deletion.js"; @@ -331,7 +331,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (companyPatch.status !== undefined && companyPatch.status !== existing.status) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); - await notifyRetentionWork(tx, "approval"); + await notifyCompanyRetentionWork(tx); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); } const updated = await tx @@ -483,7 +483,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (!wasAlreadyArchived) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); - await notifyRetentionWork(tx, "approval"); + await notifyCompanyRetentionWork(tx); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); await tx .update(companies) diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index c3b3aaf3e13..e6f716faca3 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -1,4 +1,4 @@ -import { notifyRetentionWork } from "./retention-work-notifications.js"; +import { notifyCompanyRetentionWork } from "./retention-work-notifications.js"; import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { agentObservabilityService } from "./agent-observability.js"; @@ -97,7 +97,7 @@ export async function deleteCompany(db: Db, id: string) { await tx.delete(agentRuntimeState).where(eq(agentRuntimeState.companyId, id)); await tx.delete(issueComments).where(eq(issueComments.companyId, id)); await tx.delete(approvalComments).where(eq(approvalComments.companyId, id)); - await notifyRetentionWork(tx, "approval"); + await notifyCompanyRetentionWork(tx); await tx.delete(approvals).where(eq(approvals.companyId, id)); await tx.delete(companySecrets).where(eq(companySecrets.companyId, id)); await tx.delete(joinRequests).where(eq(joinRequests.companyId, id)); diff --git a/server/src/services/decision-retention-sweep.ts b/server/src/services/decision-retention-sweep.ts new file mode 100644 index 00000000000..444fb91c2bf --- /dev/null +++ b/server/src/services/decision-retention-sweep.ts @@ -0,0 +1,50 @@ +import type { Db } from "@paperclipai/db"; +import type { AttentionSourceKind } from "@paperclipai/shared"; +import { decisionRetentionService, DEFAULT_DECISION_ARCHIVE_DAYS } from "./decision-retention.js"; + +type Candidate = { id: string; companyId: string; at: Date; cursorAt: string }; +type Cursor = { at: string; id: string }; + +export function createDecisionRetentionSweep(db: Db, options: { + sourceKind: AttentionSourceKind; + failureMessage: string; + end(cutoff: Date): Promise; + page(cutoff: Date, cursor: Cursor | undefined, end: Cursor, limit: number): Promise; + lock(database: Db, id: string, cutoff: Date): Promise; + next(): Promise; +}) { + let cursor: Cursor | undefined; + let end: Cursor | undefined; + const batchSize = 100; + async function sweep(now = new Date(), signal?: AbortSignal) { + const cutoff = new Date(now.getTime() - DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000); + end ??= await options.end(cutoff); + if (!end) return { archived: 0 }; + const rows = await options.page(cutoff, cursor, end, batchSize); + let archived = 0; + const errors: unknown[] = []; + for (const candidate of rows) { + if (signal?.aborted) break; + try { + archived += await db.transaction(async tx => { + const database = tx as unknown as Db; + const current = await options.lock(database, candidate.id, cutoff); + if (!current) return 0; + const retention = decisionRetentionService(database); + const source = { sourceKind: options.sourceKind, subject: { id: current.id }, activityAt: current.at.toISOString() }; + const states = await retention.syncItems(current.companyId, [source]); + const state = states.get(`${options.sourceKind}:${current.id}`)!; + return retention.autoArchive({ companyId: current.companyId, now, items: [{ ...source, + keep: state.keep, archivedAt: state.archivedAt?.toISOString() ?? null, retentionVersion: state.version, + }] }); + }); + } catch (error) { errors.push(error); } + cursor = { at: candidate.cursorAt, id: candidate.id }; + } + if (rows.length < batchSize) { cursor = undefined; end = undefined; } + if (errors.length === 1) throw errors[0]; + if (errors.length) throw new AggregateError(errors, options.failureMessage); + return { archived }; + } + return { sweep, next: options.next }; +} diff --git a/server/src/services/decision-retention.ts b/server/src/services/decision-retention.ts index 7b7a993cce4..fb22df2a583 100644 --- a/server/src/services/decision-retention.ts +++ b/server/src/services/decision-retention.ts @@ -15,6 +15,7 @@ import { issueRecoveryActions, issueThreadInteractions, issues, + joinRequests, } from "@paperclipai/db"; import type { AttentionArchiveManifestEntry, @@ -183,13 +184,14 @@ export function decisionRetentionService( )).then((rows) => rows[0] ?? null); } - async function materializeApprovalState(database: Db, input: { companyId: string; sourceKind: AttentionSourceKind; sourceId: string }) { - if (input.sourceKind !== "approval") return; - const [source] = await database.select({ updatedAt: approvals.updatedAt }).from(approvals) - .where(and(eq(approvals.companyId, input.companyId), eq(approvals.id, input.sourceId))).for("update"); + async function materializeSourceState(database: Db, input: { companyId: string; sourceKind: AttentionSourceKind; sourceId: string }) { + const table = input.sourceKind === "approval" ? approvals : input.sourceKind === "join_request" ? joinRequests : null; + if (!table) return; + const [source] = await database.select({ updatedAt: table.updatedAt }).from(table) + .where(and(eq(table.companyId, input.companyId), eq(table.id, input.sourceId))).for("update"); if (!source) throw notFound("Attention source not found"); await decisionRetentionService(database).syncItems(input.companyId, [{ - sourceKind: "approval", subject: { id: input.sourceId }, activityAt: source.updatedAt.toISOString(), + sourceKind: input.sourceKind, subject: { id: input.sourceId }, activityAt: source.updatedAt.toISOString(), }]); } @@ -205,7 +207,7 @@ export function decisionRetentionService( throw notFound("Attention source not found"); } const updated = await writeRetention(input.sourceKind, async tx => { - await materializeApprovalState(tx, input); + await materializeSourceState(tx, input); return tx.update(decisionRetention).set({ keep: input.keep, version: sql`${decisionRetention.version} + 1`, @@ -246,7 +248,7 @@ export function decisionRetentionService( } const now = new Date(); const updated = await writeRetention(input.sourceKind, async tx => { - await materializeApprovalState(tx, input); + await materializeSourceState(tx, input); return tx.update(decisionRetention).set(input.archived ? { archivedAt: now, archivedReason: "manual", diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index 55f7943d441..f6d1d6bb5f7 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -10,6 +10,7 @@ export const DELIVERY_QUEUES = { executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", approvalRetention: "approval-retention", + joinRequestRetention: "join-request-retention", decisionTargets: "decision-targets", decisionEffectRecovery: "decision-effect-recovery", decisionExpiry: "decision-expiry", diff --git a/server/src/services/join-request-retention.ts b/server/src/services/join-request-retention.ts new file mode 100644 index 00000000000..06a74fa814b --- /dev/null +++ b/server/src/services/join-request-retention.ts @@ -0,0 +1,52 @@ +import { and, asc, desc, eq, isNull, lte, or, sql, type SQL } from "drizzle-orm"; +import { companies, decisionRetention, invites, joinRequests, type Db } from "@paperclipai/db"; +import { DEFAULT_DECISION_ARCHIVE_DAYS } from "./decision-retention.js"; +import { createDecisionRetentionSweep } from "./decision-retention-sweep.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { registerDeadlineWork } from "./deadline-work.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; + +const archiveDelay = DEFAULT_DECISION_ARCHIVE_DAYS * 86_400_000; + +export function joinRequestRetentionService(db: Db) { + function candidates(database: Db, condition?: SQL, order = asc) { + return database.select({ id: joinRequests.id, companyId: joinRequests.companyId, + at: joinRequests.updatedAt, cursorAt: sql`${joinRequests.updatedAt}::text`, deadlineAt: sql`ceil(extract(epoch from ${joinRequests.updatedAt}) * 1000 + ${archiveDelay})`.mapWith(Number) }).from(joinRequests) + .innerJoin(companies, eq(companies.id, joinRequests.companyId)) + .innerJoin(invites, and(eq(invites.id, joinRequests.inviteId), eq(invites.companyId, joinRequests.companyId))) + .leftJoin(decisionRetention, and(eq(decisionRetention.companyId, joinRequests.companyId), + eq(decisionRetention.sourceKind, "join_request"), sql`${decisionRetention.sourceId} = ${joinRequests.id}::text`)) + .where(and(sql`${joinRequests.status} = 'pending_approval'`, eq(companies.status, "active"), + or(isNull(decisionRetention.id), eq(decisionRetention.keep, false)), isNull(decisionRetention.archivedAt), condition)) + .orderBy(order(joinRequests.updatedAt), order(joinRequests.id)); + } + return createDecisionRetentionSweep(db, { + sourceKind: "join_request", failureMessage: "Join request retention failed", + end: async cutoff => { + const [row] = await candidates(db, lte(joinRequests.updatedAt, cutoff), desc).limit(1); + return row ? { at: row.cursorAt, id: row.id } : undefined; + }, + page: async (cutoff, cursor, end, limit) => candidates(db, and(lte(joinRequests.updatedAt, cutoff), + sql`(${joinRequests.updatedAt}, ${joinRequests.id}) <= (${end.at}::timestamptz, ${end.id}::uuid)`, cursor + ? sql`(${joinRequests.updatedAt}, ${joinRequests.id}) > (${cursor.at}::timestamptz, ${cursor.id}::uuid)` : undefined)).limit(limit), + lock: async (database, id, cutoff) => { + const [row] = await candidates(database, and(eq(joinRequests.id, id), lte(joinRequests.updatedAt, cutoff))) + .for("update", { of: joinRequests }); + return row; + }, + next: async () => { + const [row] = await candidates(db).limit(1); + return row?.deadlineAt ?? null; + }, + }); +} + +export function registerJoinRequestRetentionWork( + coordinator: ReturnType, + service: ReturnType, + canRun: () => boolean, +) { + return registerDeadlineWork(coordinator, DELIVERY_QUEUES.joinRequestRetention, { + run: signal => service.sweep(new Date(), signal), next: service.next, canRun, retryMs: 30_000, notificationCoalesceMs: 30_000, + }); +} diff --git a/server/src/services/retention-work-notifications.ts b/server/src/services/retention-work-notifications.ts index 3464c624e89..a52ce2de883 100644 --- a/server/src/services/retention-work-notifications.ts +++ b/server/src/services/retention-work-notifications.ts @@ -4,6 +4,7 @@ import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notificatio export async function notifyRetentionWork(transaction: object, kind: AttentionSourceKind) { if (kind === "approval") await notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalRetention); + if (kind === "join_request") await notifyDeliveryWork(transaction, DELIVERY_QUEUES.joinRequestRetention); } export function writeApprovalChange(db: Db, write: (transaction: Db) => Promise): Promise { @@ -12,3 +13,15 @@ export function writeApprovalChange(db: Db, write: (transaction: Db) => Promi return write(tx as unknown as Db); }); } + +export async function notifyCompanyRetentionWork(transaction: object) { + await notifyRetentionWork(transaction, "approval"); + await notifyRetentionWork(transaction, "join_request"); +} + +export function writeJoinRequestChange(db: Db, write: (transaction: Db) => Promise): Promise { + return db.transaction(async tx => { + await notifyRetentionWork(tx, "join_request"); + return write(tx as unknown as Db); + }); +} From 4e27a82a4265c01f3ab4f05aaecc010a0026de75 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:33:01 -0500 Subject: [PATCH 35/41] fix(server): wake approval queue seeding from committed changes (#17510) ## Thinking Path > - Paperclip manages work and human decisions for AI-agent companies. > - Automatic decision queues store membership for pending attention sources. > - The old attention maintenance scan created that membership as a side effect. > - Moving approval retention out of the scan also removed approval queue discovery. > - Queue API consumers do not build the attention feed first, so they can miss pending PR approvals. > - This pull request gives approval queue discovery its own worker and commit signals. > - The worker becomes quiet after it saves every missing membership. ## Linked Issues or Issue Description Refs #17497. Stacked on #17508, following #17505. This is a correctness dependency for the approval retention migration in #17497. Do not treat that migration as complete without this queue-population path. A real PostgreSQL regression found no PR queue when a pending approval had a linked PR work product and the maintenance feed excluded approvals. Related: #13364 proposes pruning stale seeded memberships. This PR preserves existing membership retention and does not implement pruning. #15149 documents the decision-queue APIs. ## What Changed - Add an approval queue worker that selects missing PR-queue memberships from existing durable records. Bound each pass to 100 rows. Continue past failed rows and revisit them on retry. - Reuse the queue service's seed writer through a normalized source contract. Keep queue rules, uniqueness, and audit writes in that service. Keep feed readers compatible. - Wake from committed approval and link changes, issue/company visibility, PR work-product changes and imports, and queue-rule changes. Reuse the existing transaction boundaries. - Recheck approval status under a row lock. Serialize queue-rule changes with the seed writer. Preserve required recovery during drain and shutdown. - Register startup reconstruction under the existing maintenance switch. Leave other attention-source seeding with its current owner until its own migration. - Document the complete producer audit and stack dependency. Add a validated external PostgreSQL option to the existing queue route suite so it runs on hosts without embedded database support. ## Verification Current tested revision: `7948136ef8`. This PR targets master after #17508 merged as `8406047b4e`. Normal ancestry reconciliation preserved the complete tested integration tree, before the review repairs below. Only the original 25 seeding paths differ from master, and database schema/migration metadata is identical. - **56 cases pass in the three complete affected suites:** 28 approval seeding, 6 queue routes and 22 task-watchdog cases. Server typecheck and production build pass. - Three real PostgreSQL retry regressions failed then passed: new matching arrivals cannot postpone a recovered failure indefinitely; successful partial progress resets old failure penalties; an empty cursor rollover does not reset outage backoff. Exact one- and two-second retry assertions, eventual complete recovery and existing capped-backoff/company isolation tests pass. - Three genuinely empty external databases failed before setup repair because tables were missing. Each suite now applies migrations itself. All 55 cases passed on a second set of fresh databases; adding the final rollover guard then passed all 56 cases using those now-migrated fixtures. No production migration or table is added. - The initial expanded fresh run passed 54 cases but failed a cleanup expectation that omitted a required cursor rollover pass. The fixture now permits that explicit continuation while preserving exact deadline and complete-membership assertions. All failure records remain in `doc/idle-approval-queue-seeding.md`. - The prior integrated revision `4e725e2055` passed **522 cases in fifteen complete producer/consumer suites**, plus types/build. Its unchanged consumer code and historical deadlock, cross-company amplification, backoff, gateway integration and setup evidence remain documented; those counts are not relabeled as current acceptance. Commands: server Vitest for approval queue seeding, queue routes and task watchdogs; server typecheck/build. Fresh exact-head Greptile/Ponytail reviews and all CI checks remain required. No full local repository-suite, deployed suspension or cost claim is made. ## Risks - Signals cover the supported in-process service writers. Independent SQL writers and multiple server replicas need a separate notification contract. - The source predicate must stay aligned with unscoped attention maintenance. The first linked issue, company scope, pending status, and execution visibility are tested. A pending approval itself keeps its linked external conversation active. - Existing enabled seed rules can re-add manually removed matching memberships. This preserves the old scan's behavior. Completed or archived membership is not pruned. - Queue seeding and retention are separate owners. Other attention sources still need their own event-driven discovery before the aggregate scan can be retired. - This PR does not extend Cloud wake coverage, relax sleep admission, or claim measured host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/idle-approval-queue-seeding.md | 231 ++++++++ ...2026-10-24-recurring-server-sleep-stack.md | 91 +++ .../approval-queue-seeding-work.test.ts | 534 ++++++++++++++++++ .../__tests__/decision-queues-routes.test.ts | 12 +- .../environment-cleanup-work.test.ts | 7 +- .../issue-comment-reopen-routes.test.ts | 23 +- .../server-startup-feedback-export.test.ts | 21 + .../task-watchdogs-scheduler.test.ts | 19 +- server/src/index.ts | 3 + server/src/lib/approval-records.ts | 4 +- .../services/approval-queue-seeding-work.ts | 106 ++++ .../services/approval-work-notifications.ts | 16 + server/src/services/approvals.ts | 10 +- server/src/services/budgets.ts | 13 +- server/src/services/companies.ts | 4 +- server/src/services/company-deletion.ts | 2 +- .../src/services/decision-queue-seed-rules.ts | 6 + server/src/services/decision-queues.ts | 222 ++++---- .../services/delivery-work-notifications.ts | 1 + server/src/services/issue-approvals.ts | 8 +- server/src/services/issues.ts | 7 +- .../services/retention-work-notifications.ts | 12 +- server/src/services/routines.ts | 4 +- server/src/services/tool-gateway.ts | 6 +- server/src/services/work-products.ts | 21 +- 25 files changed, 1218 insertions(+), 165 deletions(-) create mode 100644 doc/idle-approval-queue-seeding.md create mode 100644 server/src/__tests__/approval-queue-seeding-work.test.ts create mode 100644 server/src/services/approval-queue-seeding-work.ts create mode 100644 server/src/services/approval-work-notifications.ts create mode 100644 server/src/services/decision-queue-seed-rules.ts diff --git a/doc/idle-approval-queue-seeding.md b/doc/idle-approval-queue-seeding.md new file mode 100644 index 00000000000..a741064ca00 --- /dev/null +++ b/doc/idle-approval-queue-seeding.md @@ -0,0 +1,231 @@ +# Approval queue seeding without retention polling + +## Obligation and stack dependency + +The attention maintenance scan does two different jobs: retention and automatic +queue membership. Its call to `materializeSeededQueues` adds pending approvals to +the PR queue when the first linked issue has a pull-request work product. Queue +API readers use saved membership; they do not first build the attention feed. +Removing approvals from maintenance therefore lost automatic queue population. +A real PostgreSQL regression reproduced an empty queue with a pending linked +approval and PR work product, without an attention reader. + +This slice owns automatic PR-queue membership for approvals. It depends on the +approval producer boundaries introduced in #17497 and is required with that +retention migration before treating the combined behavior as complete. Join +requests have no issue association or matching seed rule. Other attention-source +seed discovery remains in the existing feed scan and must migrate with an +explicit owner before those sources leave that scan. + +## Source of truth and ownership + +Existing approvals, issue approvals, issue work products, queues, and queue items +are the only durable state. No new table, outbox, external API, or provider branch +is introduced. `approvalQueueSeedingService` selects missing memberships for: + +- A pending approval in an active company. +- Its first linked issue, ordered by issue ID, in the same company and eligible + under the attention feed's execution-issue visibility predicate. +- A pull-request work product belonging to that issue and company. +- A missing PR queue or an enabled seed rule without this approval's membership. + +Disabled rules, nonmatching approvals, completed approvals, and existing seeded +history are passive. A pending approval itself prevents its linked Slack/Speko +conversation from projecting as idle, so this source does not depend on every +other conversation-state producer. Hidden/harness/conversation-container issue +changes retain their explicit signals. + +The queue service owns seed matching, creation, membership uniqueness, and audit +writes. The feed and worker use the same writer through a small normalized source +contract. The candidate lookup and writer share the pull-request product SQL +predicate, so the matching policy has one owner. A worker does not construct a presentation-layer AttentionItem. Queue +reads keep their existing source authorization checks. A queue-rule update and +seeding serialize on the existing queue row. Existing memberships are not pruned +when an approval resolves. This preserves current behavior; pruning is a separate +change, including the proposal in #13364. + +## Producer audit + +| Producer | Commit boundary and effect | +| --- | --- | +| Approval create, cancel, revision, resubmit, resolve | `writeApprovalChange` notifies retention and company-scoped queue seeding before writes in the owning transaction. Approve/reject/cancel retain their explicit tool-review notification; ordinary mutations do not wake those queues. | +| Shared approval records, budget approvals, formal tool approvals | The same approval notification covers their existing transactions. | +| Issue approval link, unlink, link-many, tool-created links | Existing approval transaction helper. Link removal can expose a different first issue. | +| Issue visibility changes and issue deletion | Existing approval notification before the update or cascading delete. Routine failure cleanup retains this signal. | +| Company activation, pause, archive, deletion and budget pause/resume | Existing company notification wakes both approval consumers. | +| Work-product create, type/issue update, import batch, delete | The owning work-product transaction notifies seeding when a pull-request row is involved. | +| PR queue creation, rule update and manual approval membership removal | The queue mutation transaction notifies seeding. An enabled rule can re-add a still-matching manually removed item, as the old scan did. | + +Other direct work-product writes only create artifacts or update title, metadata, +or trust. They cannot create a pull-request match. Supported writes use these +in-process service boundaries. Independent SQL writers and multiple server +replicas are not covered by local transaction signals. + +Retention-only keep/archive/revive changes do not create queue projection debt. +The old seed writer ran before retention filtering, so archived or kept approvals +still follow the same seeding rule. User-specific dismissals do not filter the +unscoped maintenance feed. No new deadline is inferred from those display states. + +## Worker lifecycle + +Startup reconstructs missing memberships across the instance. Each later committed +producer carries its company ID and wakes only that company's reconciliation. +Pending scopes coalesce in memory; one company's ordinary mutations cannot restart +a global scan. Scope rotation prevents one failing company from starving another. Each sweep processes at most 100 approvals, with an ID cursor and a fixed upper ID per scan cycle. The cursor advances past +failures and finishes that finite cycle before revisiting them; new arrivals cannot +extend it indefinitely. Each row is rechecked under +an approval lock before the queue writer runs. One bad row cannot permanently +hide later candidates. Atomic existing membership uniqueness prevents duplicate +rows and duplicate insertion audit events. + +After a sweep, another lookup decides whether actual projection debt remains. +Unprocessed pages and newer committed signals retain a one-second continuation. +Failed scopes back off exponentially from one second to one minute. A successful +nonempty batch resets consecutive failures; an empty cursor rollover while debt +remains preserves outage backoff. New signals +do not reset a failing scope's backoff; other companies can proceed independently. +The next retry is registered with the shared scheduler, with no SQL during the +wait. A failed scope remains tracked until its owed memberships are reconciled. The generation check follows the final awaited lookup, so a commit arriving +during an empty read cannot be erased. Once reconciled, there is no deadline and +no recurring worker SQL. The shared coordinator preserves requested work during +drain. Passive pending checks inspect memory only; startup reconstructs from rows. Shutdown aborts between rows +and waits for an admitted transaction; unprocessed rows remain recoverable. +Registration follows the existing heartbeat-maintenance enablement switch. + +This slice does not extend Cloud alarm coverage or loosen host sleep admission. +It proves a worker's quiet/reactivation behavior, not deployed host or database +suspension. The remaining attention maintenance interval is still present for +other source owners. + +## Verification + +- Before the fix, the PostgreSQL regression found no PR queue after the normal + maintenance path excluded approvals. After the fix, the queue API finds it + without a full attention read. +- Twenty-one real-database worker tests cover empty-to-active-to-quiet behavior, + startup and completed history, actual approval/link/work-product producers, + batch import, first-link selection, company isolation, hidden targets, queue + rule enablement, company activation, resubmission, outer commit and rollback, + first write during drain, database failure through drain, multiple pages, + failed-page progress and retry, concurrent resolution, archived/kept history, shutdown/restart, and an admitted transaction at shutdown. +- 115 existing tests pass across queue routes, attention, approval retention, + approval service, work products, and startup wiring. The shared writer still + passes existing plan/question/PR seed tests and authorization tests. +- Workspace typechecks and the production build pass. Fresh PR checks are + tracked in the PR and must pass before readiness is claimed. +- The actual pending lookup on 40,000 completed approvals plus one pending PR + approval uses the existing partial pending-retention index. The probe runs in + a rolled-back transaction; execution took 0.094 ms on this local fixture. No new index or migration is needed for this case. + +The first expanded test run caught two fixture errors: a hand-inserted queue +omitted required creator attribution, and a failure test expected no idle hold +while committed projection work was still unresolved. The corrected test waits +for the active attempt to finish and checks passive SQL directly while preserving +that required hold. Neither required weakening the worker's recovery contract. + +Review found a queue/item lock inversion between seeded insertion and manual +membership changes. A two-connection PostgreSQL regression reproduced `deadlock +detected`. Manual add/remove now lock the queue before touching its items, matching +the seed writer. The regression and the existing queue route suite pass (27 cases). +Server typecheck and build pass after the repair. Fresh-head CI/review is required. + +Security review found that normal producer wakes could restart a cross-company +scan. The regression reproduced another company's queue being populated by an +unrelated company write. Producers now include company hints, including batch +imports and nested transactions. A separate controlled PostgreSQL test holds the +final empty read while a real producer commits and verifies the retained wake. + +After company scoping, all 169 selected cases pass across approval queue seeding, +approval retention, queue routes, approval services, work products, and issue +comment routes. Server typechecking passes. The preceding run suffered timeouts +and a consequent cleanup foreign-key error; it is not counted as passing. The +route suite separately exposed cold imports inside timed tests. Moving those +imports to suite loading preserves assertions and prevents timed-out requests +from leaking into later mock state. The watchdog fixture collision from CI uses +monotonic identifiers instead; its 19 real-PostgreSQL cases pass. + +The next CI run passed the security gate and the two originally failing shards. +Its remaining server shard caught a historical cleanup test checking 33 provider +inspections after only 32 had completed. The test advanced fake time before the +previous asynchronous page had necessarily published its next deadline. It now +waits for all 33 inspections, then retains the zero-further-work assertions. All +14 environment cleanup cases pass on real PostgreSQL; production code is unchanged. + + +Further review reproduced 120 failed transactions in two simulated minutes for a +persistently failing projection. Per-scope backoff now bounds that test to at most +eight attempts, even with new signals every second. Removing the fault recovers +the membership. A real coordinator/producer case proves thirty seconds without +SQL during backoff, another company's successful queue seeding during that wait, +and eventual recovery without losing the original work. All 25 worker cases pass. +The first consumer command omitted the external DB setting for queue routes and +skipped those six cases; the corrected command executes and passes all six. +The shared PR predicate is exercised by both those writer tests and worker tests. +Server typecheck and build pass; fresh remote review/checks remain required. + +The final startup, approval-retention, and attention consumer run passes all 93 +cases. With the 25 worker and six queue cases, 124 affected tests pass. + + +## Current retention-parent integration + +The seeding worker reuses `firstApprovalIssueId` from the approval-visibility +owner rather than retaining its own first-link SQL. Company IDs are part of the +approval notification contract because projection reconciliation is scoped. +Generic approval writes notify retention and seeding only; the existing terminal +approve/reject/cancel owners alone notify tool-review consumers. Current precise +issue-deletion target notifications, unseen retention controls, finite retention +cycles and the real source-row lock regressions are preserved. + +All 522 cases in fifteen complete suites pass at `4e725e2055`, with server +typecheck and production build. This includes seeding, queue routes, work products, +issue comments, both retention workers, approvals, attention, retention controls, +budgets, tool gateway, startup, scoped deadlines, environment cleanup and task +watchdogs. Four new disposable fixtures were cloned from the already migrated +parent fixture; this is not a fresh-empty-schema claim. The other suites used +separate existing disposable databases. No database was shared with another +concurrent local test run. The subsequent parent merge changes only two documents; +all tested runtime and acceptance sources remain byte-identical. + +The first integration typecheck caught two incorrect gateway company references. +They now use the existing session company, and typecheck plus the complete gateway +suite pass. Earlier deadlock, cross-company wake, backoff, skipped-fixture and +teardown failures remain in the historical sections above. The parent's attention +cleanup timeout and expiry-test CI assertion are recorded in its document; this +larger current run passes without changing their assertions or timeouts. Fresh +exact-head reviews and CI remain required before this dependent PR can merge. + + +## Final master and review repairs + +Join retention merged as `8406047b4e`. The normal master reconciliation first +proved that landed tree equals the existing parent. Conflict resolution preserved +the complete tested seeding tree; it also removed a duplicate helper introduced +by automatic merge. The PR again differs from master in only its original 25 +seeding paths. Database schema and migration metadata are identical to master. + +Three retry regressions failed before repair: ongoing arrivals skipped a recovered +membership, a partial successful batch retained the old sixty-second failure +penalty, and resetting that counter on an empty cursor rollover weakened outage +backoff. The worker now fixes the upper ID per cycle and resets consecutive +failures only on real successful progress or complete reconciliation. Exact +one-second recovery and two-second consecutive-failure assertions pass; repeated +signals still cannot reset a failing scope's delay. No new table or retry wrapper +is introduced. + +The external seeding, queue-route and watchdog fixtures now apply pending +migrations before creating their client. Each of three empty-database regressions +failed with missing tables before this setup repair. All 55 cases then passed on +a second set of genuinely empty databases, with the suites applying their own +migrations. The final additional rollover guard brings the three complete suites +to 56 passing cases (28 seeding, 6 queue-route, 22 watchdog), using those now +migrated fixtures. Server types/build pass at `7948136ef8`. The earlier 522-case +integration run remains evidence for its stated revision and unchanged consumers; +it is not relabeled as a current full-suite run. + +The first expanded fresh run passed 54 cases and failed the new backoff test's +cleanup expectation: after an entirely failed full page, a separate empty pass +ends the cycle before the next pass revisits rows. The corrected fixture permits +that explicit continuation and still checks complete recovery and the exact retry +deadline. Earlier failures remain recorded. Fresh final-head CI and reviews are +required before merge. diff --git a/doc/plans/2026-10-24-recurring-server-sleep-stack.md b/doc/plans/2026-10-24-recurring-server-sleep-stack.md index b032fcbb481..56ecd467c84 100644 --- a/doc/plans/2026-10-24-recurring-server-sleep-stack.md +++ b/doc/plans/2026-10-24-recurring-server-sleep-stack.md @@ -1132,3 +1132,94 @@ assertion failure; neither passing rerun is presented as a proven root cause. Server types/build pass. See the join-retention document for the full audit. Fresh exact-head reviews and CI remain required. All original duties remain open until their own accepted merge, including native-harness backup expiration. + +## Slice 16: approval PR-queue membership + +The source audit found automatic seeded-queue materialization hidden inside the +attention retention scan. A PostgreSQL regression failed after approval exclusion: +queue API readers saw no PR queue despite a pending linked approval and PR work +product. This is a separate obligation from retention, not grounds to leave a +new permanent scan or to put queue policy into retention. + +The new approval queue worker uses existing durable records and the queue service's +normalized seed writer. It gets committed signals from all approval/link producers, +issue/company visibility changes, work-product writes, and queue-rule changes. +Missing membership is recovery debt; existing membership and disabled rules are +quiet. Twenty real-database lifecycle cases and 115 existing consumer/startup +tests pass, as do workspace typechecks and the production build. See +[approval queue seeding](../idle-approval-queue-seeding.md). Fresh CI/review +remain required before readiness. This slice is a correctness dependency of +approval retention #17497; do not accept that migration as functionally complete +without preserving queue population. Other source seeding remains to be migrated. + +#17505 now has all latest-head checks accepted at `28a197cdfdab7977eecf591df39612009d725e17` +(Paperclip Review skipped), with Greptile 5/5. #17508 at +`9c5c4fbf65628589b466ce5650e8969388213716` has Greptile 5/5 and no unresolved +threads. Its unchanged login-cleanup consume test failed in CI; all 16 local +login-cleanup tests pass. One justified retry of failed run 40139410144 was +requested. Do not mark that PR green until the retry completes successfully. + + +Slice 16 is [#17510](https://github.com/paperclipai/paperclip/pull/17510). +Greptile identified a queue/item lock-order inversion. A real PostgreSQL test +reproduced a deadlock; the common queue-before-item order repairs manual add/remove +and seeding. The worker and queue route suites pass 27 cases after the fix, with +server typecheck/build passing. Other selected consumers passed on the initial +head (136 unique local cases including the new regression). Fresh-head checks are +required; the initial head also hit an unchanged environment cleanup paging test. +Embedded PostgreSQL is unsupported on this host, so its first local run skipped; +this is not counted as a pass. + +#17508 now has all checks accepted at `9c5c4fbf65628589b466ce5650e8969388213716` +after its single justified retry. The duplicate cancelled workflows for #17490 +and #17497 were each rerun once to clear stale cancelled contexts. The #17490 +retry has a new E2E shard 3 failure under investigation; do not call it green or +rerun again without a repair. #17497's retry was still running at this observation. + + +Slice 16 security review reproduced cross-company work amplification from normal +producer writes. The repair carries company hints through all supported producers +and maintains separate pending scopes. Startup recovery remains instance-wide. +A generation comparison after the final awaited read preserves concurrent commits. +All 169 affected cases and server typechecking pass. CI fixture repairs move cold +route imports outside timed tests and replace random watchdog identifiers. The +initial rerun had timeouts and a cleanup failure; the final selected run passes +without relaxing timeouts. Fresh remote CI/review remains required. + +Plugin scheduling is published as #17518 at 8b4f551dc2. Its Actions checks passed, +but review found retained completion writes missing from shutdown and a stale +deadline update race. Repair those and preserve independent job progress when +one completion write fails before treating the slice as ready. + + +Approval queue follow-up: persistent projection failure reproduced 120 failed +transactions in two simulated minutes. Per-scope exponential backoff now caps +retries at one minute without discarding debt or letting repeated signals reset +the budget. A coordinator regression proves SQL silence during backoff and real +producer progress for another company. The candidate and seed writer share the +PR-product predicate. All 124 affected worker/queue/startup/retention/attention +cases pass, along with server types/build. One initial consumer command skipped +six queue tests because its DB variable was missing; the corrected run executes +and passes them. Fresh remote checks and review remain required after publication. + + +Slice 16 current parent: approval seeding integrates published join retention +`b41c51c70f`. Only the original 25 seeding paths differ from the tested parent; +all other files match it. The worker shares the first-linked-issue policy, and +generic writes preserve retention-plus-scoped-seeding signals while terminal +approval outcomes retain tool-review signals. All 522 cases in fifteen complete +suites, server types and production build pass at `4e725e2055`. An initial +integration typecheck caught incorrect gateway company references; those are +repaired with the existing session company. The final parent merge adds only +two documents. Fresh final-head CI/reviews and the parent's merge remain required. + + +Slice 16 now integrates landed join retention `8406047b4e` and targets master. +Its own diff is still the original 25 seeding paths; all database metadata matches +master. Continuing-arrival fairness, post-success backoff reset and outage backoff +through an empty rollover have real failed-then-passing regressions. External +seeding, queue and watchdog tests now migrate their own empty databases: all 55 +cases passed on three fresh fixtures, and the final added guard passes all 56 +cases on those migrated fixtures. Server types/build pass at `7948136ef8`. +The earlier cleanup-expectation failure and 522-case integration evidence retain +their own revisions. Fresh exact-head reviews and CI are the remaining merge gate. diff --git a/server/src/__tests__/approval-queue-seeding-work.test.ts b/server/src/__tests__/approval-queue-seeding-work.test.ts new file mode 100644 index 00000000000..9b9b7335d11 --- /dev/null +++ b/server/src/__tests__/approval-queue-seeding-work.test.ts @@ -0,0 +1,534 @@ +import { randomUUID } from "node:crypto"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; +import { activityLog, approvals, companies, createDb, applyPendingMigrations, decisionQueueItems, decisionQueues, decisionRetention, decisionTriageEvents, issueApprovals, issues, issueWorkProducts, subscribeDatabaseWork, type Db } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; +import { approvalRecords } from "../services/approvals.js"; +import { approvalRetentionService } from "../services/approval-retention.js"; +import { approvalQueueSeedingService, registerApprovalQueueSeedingWork } from "../services/approval-queue-seeding-work.js"; +import { attentionService } from "../services/attention.js"; +import { companyService } from "../services/companies.js"; +import { decisionQueueService } from "../services/decision-queues.js"; +import { issueApprovalService } from "../services/issue-approvals.js"; +import { issueService } from "../services/issues.js"; +import { workProductService } from "../services/work-products.js"; +import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js"; +import { idleWorkSnapshot } from "../services/task-admission.js"; + +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_APPROVAL_QUEUE_TEST_DATABASE_URL"); +const support = external ? { supported: true } : await getEmbeddedPostgresTestSupport(); +(support.supported ? describe : describe.skip)("approval queue seeding", () => { + let temporary: Awaited> | undefined; + let db: ReturnType, companyId: string; + let coordinator: ReturnType | undefined; + let draining = false; + const errors: unknown[] = []; + beforeAll(async () => { + if (external) await applyPendingMigrations(external); + else temporary = await startEmbeddedPostgresTestDatabase("approval-queue-"); + db = createDb(external ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 }); + }, 30_000); + beforeEach(async () => { + companyId = randomUUID(); draining = false; errors.length = 0; + await db.insert(companies).values({ id: companyId, name: "Approval queue", issuePrefix: `Q${companyId.slice(0, 6)}` }); + await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`))); + vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true }); + }); + afterEach(async () => { + await coordinator?.stop(); coordinator = undefined; + vi.useRealTimers(); vi.restoreAllMocks(); + await db.execute(sql`drop trigger if exists approval_queue_delete_gate on decision_queue_items`); + await db.execute(sql`drop function if exists approval_queue_delete_gate()`); + await db.execute(sql`drop trigger if exists approval_queue_fail on decision_queue_items`); + await db.execute(sql`drop function if exists approval_queue_fail()`); + await db.delete(decisionTriageEvents); await db.delete(decisionQueueItems); await db.delete(decisionQueues); + await db.delete(decisionRetention); await db.delete(activityLog); await db.delete(issueApprovals); + await db.delete(issueWorkProducts); await db.delete(approvals); await db.delete(issues); await db.delete(companies); + expect(errors).toEqual([]); + }); + afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); }); + const authActor = () => ({ type: "board" as const, source: "local_implicit" as const, userId: "board-user", companyIds: [companyId], isInstanceAdmin: true }); + const actor = { actorType: "user" as const, actorId: "board-user", agentId: null, userId: "board-user", runId: null, agentApiKeyId: null, responsibleUserId: "board-user" }; + const mutation = () => ({ companyId, key: "prs", authActor: authActor(), actor }); + const memberships = () => db.select().from(decisionQueueItems).where(eq(decisionQueueItems.companyId, companyId)); + const createApproval = (database: Db = db) => approvalRecords(database).create(companyId, { type: "request_board_approval", payload: {} }); + async function issue(hiddenAt: Date | null = null) { + const [row] = await db.insert(issues).values({ companyId, title: "Review PR", status: "todo", hiddenAt }).returning(); + return row; + } + const product = (issueId: string, type = "pull_request") => workProductService(db).createForIssue(issueId, companyId, { type, provider: "github", title: "PR", status: "open" }); + async function pending(target?: string) { + const targetId = target ?? (await issue()).id; + if (!target) await product(targetId); + const approval = await createApproval(); + await issueApprovalService(db).link(targetId, approval.id); + return approval; + } + async function start(service = approvalQueueSeedingService(db)) { + coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true, onError: error => errors.push(error) }); + await registerApprovalQueueSeedingWork(coordinator, service, () => !draining).ready; + return service; + } + async function settle() { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1_000); + expect(idleWorkSnapshot().active).toBe(0); + }, { timeout: 5000 }); + } + async function seeded(id: string) { + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1_000); + expect(await memberships()).toEqual(expect.arrayContaining([expect.objectContaining({ sourceKind: "approval", sourceId: id })])); + }, { timeout: 5000 }); + } + async function quiet() { + await settle(); expect(coordinator!.nextWakeAt()).toBeNull(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"), executes = vi.spyOn(db, "execute"); + await vi.advanceTimersByTimeAsync(300_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); executes.mockRestore(); + } + + it("retains a company commit arriving during its final empty read", async () => { + const service = approvalQueueSeedingService(db); + await service.sweep(); + expect(await service.next()).toBeNull(); + const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.approvalQueueSeeding, (event, hints) => { + if (event === "settled") service.request(hints); + }); + const select = db.select.bind(db); + let release!: () => void, observed!: () => void; + const observedEmpty = new Promise(resolve => { observed = resolve; }); + const barrier = new Promise(resolve => { release = resolve; }); + let intercepted = false; + const reads = vi.spyOn(db, "select").mockImplementation((...args: any[]) => { + const query = (select as any)(...args); + const from = query.from.bind(query); + query.from = (...tables: any[]) => { + const builder = from(...tables); + const limit = builder.limit.bind(builder); + builder.limit = (count: number) => { + const result = limit(count); + if (count === 1 && !intercepted) { + intercepted = true; + const then = result.then.bind(result); + result.then = (resolve: (rows: unknown[]) => unknown, reject: (err: unknown) => unknown) => + then(async (rows: unknown[]) => { observed(); await barrier; return resolve(rows); }, reject); + } + return result; + }; + return builder; + }; + return query; + }); + service.request(new Set([companyId])); + const sweep = service.sweep(); + try { + await observedEmpty; + const approval = await pending(); + release(); await sweep; reads.mockRestore(); + expect(await service.next()).not.toBeNull(); + await service.sweep(); + expect(await memberships()).toEqual([expect.objectContaining({ sourceId: approval.id })]); + expect(await service.next()).toBeNull(); + } finally { release(); await sweep; reads.mockRestore(); unsubscribe(); } + }); + + it("does not let a company write reconcile another company's approvals", async () => { + await start(); await quiet(); + const foreignId = randomUUID(); + await db.insert(companies).values({ id: foreignId, name: "Other tenant", issuePrefix: `F${foreignId.slice(0, 6)}` }); + const [foreignIssue] = await db.insert(issues).values({ companyId: foreignId, title: "Other PR", status: "todo" }).returning(); + const [foreignApproval] = await db.insert(approvals).values({ companyId: foreignId, type: "request_board_approval", payload: {} }).returning(); + await db.insert(issueApprovals).values({ companyId: foreignId, issueId: foreignIssue.id, approvalId: foreignApproval.id }); + await db.insert(issueWorkProducts).values({ companyId: foreignId, issueId: foreignIssue.id, type: "pull_request", provider: "github", title: "Other PR", status: "open" }); + const ownApproval = await pending(); await seeded(ownApproval.id); await quiet(); + expect(await db.select().from(decisionQueueItems).where(eq(decisionQueueItems.companyId, foreignId))).toEqual([]); + await companyService(db).update(foreignId, { status: "paused" }); await settle(); + await companyService(db).update(foreignId, { status: "active" }); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await db.select().from(decisionQueueItems).where(eq(decisionQueueItems.companyId, foreignId))) + .toEqual([expect.objectContaining({ sourceId: foreignApproval.id })]); + }); + await quiet(); + }); + + it("populates the PR queue without an attention reader after approvals leave retention maintenance", async () => { + await start(); await quiet(); + const approval = await pending(); + await approvalRetentionService(db).sweep(); + await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true, excludeSources: ["approval", "join_request"] }); + await seeded(approval.id); + expect((await decisionQueueService(db).list(companyId, authActor())).map(queue => queue.key)).toContain("prs"); + expect(await decisionQueueService(db).listItems(companyId, "prs", authActor())).toEqual(expect.arrayContaining([ + expect.objectContaining({ sourceKind: "approval", sourceId: approval.id }), + ])); + await quiet(); + }); + + it("recovers startup projection debt and stays quiet with already seeded and completed history", async () => { + const first = await pending(), completed = await pending(); + await approvalRecords(db).resolveApproval(completed.id, "approved", "board-user", null); + await start(); await seeded(first.id); await quiet(); + expect((await memberships()).map(row => row.sourceId)).toEqual([first.id]); + await coordinator!.stop(); coordinator = undefined; + await start(); await quiet(); + expect(await db.select().from(decisionTriageEvents).where(eq(decisionTriageEvents.action, "queue_item.seeded"))).toHaveLength(1); + }); + + it("wakes when a work product creates the first PR match", async () => { + const target = await issue(), approval = await pending(target.id); + await start(); await quiet(); expect(await memberships()).toEqual([]); + const row = await product(target.id, "artifact"); await quiet(); + await workProductService(db).update(row!.id, { type: "pull_request" }); + await seeded(approval.id); await quiet(); + }); + + it("wakes from batch-imported PR work products", async () => { + const target = await issue(), approval = await pending(target.id); + await start(); await quiet(); + await workProductService(db).createManyForImport([{ companyId, issueId: target.id, type: "pull_request", provider: "github", title: "Imported PR", status: "open", reviewState: "none", isPrimary: false, healthStatus: "unknown" }]); + await seeded(approval.id); await quiet(); + }); + + it("honors hidden targets and wakes when visibility changes", async () => { + const target = await issue(new Date()); await product(target.id); const approval = await pending(target.id); + await start(); await quiet(); expect(await memberships()).toEqual([]); + await issueService(db).update(target.id, { hiddenAt: null }); await seeded(approval.id); await quiet(); + }); + + it("uses the first linked issue and reevaluates after unlink", async () => { + const first = await issue(), second = await issue(); + const [withoutPr, withPr] = [first, second].sort((a, b) => a.id.localeCompare(b.id)); + await product(withPr.id); + const approval = await pending(withoutPr.id); await issueApprovalService(db).link(withPr.id, approval.id); + await start(); await quiet(); expect(await memberships()).toEqual([]); + await issueApprovalService(db).unlink(withoutPr.id, approval.id); await seeded(approval.id); await quiet(); + }); + + it("does not infer PRs from a later link or another company's work product", async () => { + const target = await issue(), approval = await pending(target.id); + const foreignId = randomUUID(); await db.insert(companies).values({ id: foreignId, name: "Other", issuePrefix: "OTHER" }); + await db.insert(issueWorkProducts).values({ companyId: foreignId, issueId: target.id, type: "pull_request", provider: "github", title: "Invalid foreign product", status: "open" }); + await start(); await quiet(); expect(await memberships()).toEqual([]); + await product(target.id); await seeded(approval.id); await quiet(); + }); + + it("keeps disabled seeds quiet and wakes when the queue rule is enabled", async () => { + await db.insert(decisionQueues).values({ companyId, key: "prs", title: "PRs", createdByType: "system", seedRulesEnabled: false }); + const approval = await pending(); await start(); await quiet(); expect(await memberships()).toEqual([]); + await decisionQueueService(db).update({ ...mutation(), patch: { seedRulesEnabled: true } }); + await seeded(approval.id); await quiet(); + }); + + it("waits for company activation", async () => { + const approval = await pending(); await companyService(db).update(companyId, { status: "paused" }); + await start(); await quiet(); expect(await memberships()).toEqual([]); + await companyService(db).update(companyId, { status: "active" }); await seeded(approval.id); await quiet(); + }); + + it("waits for resubmission and never removes retained membership after approval", async () => { + const approval = await pending(); await approvalRecords(db).requestRevision(approval.id, "board-user"); + await start(); await quiet(); expect(await memberships()).toEqual([]); + await approvalRecords(db).resubmit(approval.id); await seeded(approval.id); + await approvalRecords(db).resolveApproval(approval.id, "approved", "board-user", null); await quiet(); + expect(await memberships()).toHaveLength(1); + }); + + it("waits for outer commit and discards rolled-back producers", async () => { + const target = await issue(); await product(target.id); await start(); await quiet(); + await expect(db.transaction(async tx => { + const approval = await createApproval(tx as unknown as Db); + await issueApprovalService(tx as unknown as Db).link(target.id, approval.id); + await vi.advanceTimersByTimeAsync(5_000); expect(await memberships()).toEqual([]); + throw new Error("rollback"); + })).rejects.toThrow("rollback"); + await quiet(); + let id = ""; + await db.transaction(async tx => { + id = (await createApproval(tx as unknown as Db)).id; + await issueApprovalService(tx as unknown as Db).link(target.id, id); + await vi.advanceTimersByTimeAsync(5_000); expect(await memberships()).toEqual([]); + }); + await seeded(id); await quiet(); + }); + + it("retains the first write during drain without passive SQL", async () => { + await start(); await quiet(); draining = true; + const approval = await pending(); + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + await vi.advanceTimersByTimeAsync(60_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); + expect(await memberships()).toEqual([]); draining = false; + await seeded(approval.id); await quiet(); + }); + + it("recovers a failed projection after drain and stops retrying once it commits", async () => { + const service = approvalQueueSeedingService(db), sweep = service.sweep; + let running = false; + service.sweep = async signal => { running = true; try { await sweep(signal); } finally { running = false; } }; + await start(service); await quiet(); + await db.execute(sql`create function approval_queue_fail() returns trigger language plpgsql as $$ begin raise exception 'projection unavailable'; end $$`); + await db.execute(sql`create trigger approval_queue_fail before insert on decision_queue_items for each row execute function approval_queue_fail()`); + const approval = await pending(); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1_000); expect(errors.length).toBeGreaterThan(0); }, { timeout: 5000 }); + draining = true; + await vi.waitFor(() => expect(running).toBe(false), { timeout: 5000 }); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); errors.length = 0; + draining = false; await seeded(approval.id); await quiet(); + }); + + it("backs off persistent projection failures without forgetting the owed membership", async () => { + const service = approvalQueueSeedingService(db); + await service.sweep(); + const approval = await pending(); + service.request(new Set([companyId])); + await db.execute(sql`create function approval_queue_fail() returns trigger language plpgsql as $$ begin raise exception 'projection unavailable'; end $$`); + await db.execute(sql`create trigger approval_queue_fail before insert on decision_queue_items for each row execute function approval_queue_fail()`); + const transactions = vi.spyOn(db, "transaction"); + try { + for (let second = 0; second < 120; second++) { + service.request(new Set([companyId])); + await service.sweep().catch(() => undefined); + await vi.advanceTimersByTimeAsync(1000); + } + expect(transactions.mock.calls.length).toBeLessThanOrEqual(8); + expect(transactions.mock.calls.length).toBeGreaterThan(1); + expect(await memberships()).toEqual([]); + expect(await service.next()).not.toBeNull(); + } finally { + transactions.mockRestore(); + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); + } + await vi.advanceTimersByTimeAsync(60_000); + await service.sweep(); + expect(await memberships()).toEqual([expect.objectContaining({ sourceId: approval.id })]); + expect(await service.next()).toBeNull(); + }); + + it("resets consecutive failure backoff after a successful batch with remaining debt", async () => { + const target = await issue(); await product(target.id); + const rows = await db.insert(approvals).values(Array.from({ length: 101 }, () => ({ + companyId, type: "request_board_approval", payload: {}, + }))).returning(); + await db.insert(issueApprovals).values(rows.map(row => ({ companyId, issueId: target.id, approvalId: row.id }))); + const service = approvalQueueSeedingService(db); + await db.execute(sql`create function approval_queue_fail() returns trigger language plpgsql as $$ begin raise exception 'projection unavailable'; end $$`); + const fail = () => db.execute(sql`create trigger approval_queue_fail before insert on decision_queue_items for each row execute function approval_queue_fail()`); + await fail(); + for (let attempt = 0; attempt < 7; attempt++) { + await expect(service.sweep()).rejects.toThrow(); + await vi.advanceTimersByTimeAsync(60_000); + } + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); + await service.sweep(); + expect(await memberships()).toHaveLength(1); expect(service.next()).not.toBeNull(); + await fail(); + const at = Date.now(); const clock = vi.spyOn(Date, "now").mockReturnValue(at); + try { + await expect(service.sweep()).rejects.toThrow("Approval queue seeding failed"); + expect(service.next()).toBe(at + 1000); + } finally { clock.mockRestore(); } + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); + await vi.advanceTimersByTimeAsync(60_000); await service.sweep(); + expect(service.next()).not.toBeNull(); await service.sweep(); + expect(await memberships()).toHaveLength(101); expect(service.next()).toBeNull(); + }); + + it("preserves outage backoff across an empty cursor rollover", async () => { + const target = await issue(); await product(target.id); + const rows = await db.insert(approvals).values(Array.from({ length: 100 }, () => ({ + companyId, type: "request_board_approval", payload: {}, + }))).returning(); + await db.insert(issueApprovals).values(rows.map(row => ({ companyId, issueId: target.id, approvalId: row.id }))); + const service = approvalQueueSeedingService(db); + await db.execute(sql`create function approval_queue_fail() returns trigger language plpgsql as $$ begin raise exception 'projection unavailable'; end $$`); + await db.execute(sql`create trigger approval_queue_fail before insert on decision_queue_items for each row execute function approval_queue_fail()`); + await expect(service.sweep()).rejects.toThrow("Approval queue seeding failed"); + await vi.advanceTimersByTimeAsync(1000); await service.sweep(); + expect(await memberships()).toEqual([]); expect(service.next()).not.toBeNull(); + const at = Date.now(); const clock = vi.spyOn(Date, "now").mockReturnValue(at); + try { + await expect(service.sweep()).rejects.toThrow("Approval queue seeding failed"); + expect(service.next()).toBe(at + 2000); + } finally { clock.mockRestore(); } + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); + await vi.advanceTimersByTimeAsync(60_000); await service.sweep(); await service.sweep(); + expect(await memberships()).toHaveLength(100); expect(service.next()).toBeNull(); + }); + + it("keeps backoff SQL-quiet while another company's committed work proceeds", async () => { + const service = approvalQueueSeedingService(db), sweep = service.sweep; + let running = false; + service.sweep = async signal => { running = true; try { await sweep(signal); } finally { running = false; } }; + await start(service); await quiet(); + await db.execute(sql`create function approval_queue_fail() returns trigger language plpgsql as $$ begin + if new.company_id::text = TG_ARGV[0] then raise exception 'projection unavailable'; end if; return new; end $$`); + await db.execute(sql`create trigger approval_queue_fail before insert on decision_queue_items for each row execute function approval_queue_fail(${sql.raw("'" + companyId + "'")})`); + const approval = await pending(); + for (let count = 1; count <= 7; count++) { + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1000); expect(errors.length).toBeGreaterThanOrEqual(count); }, { timeout: 5000 }); + await vi.waitFor(() => expect(running).toBe(false)); + } + const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "transaction"); + await vi.advanceTimersByTimeAsync(30_000); + expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); + reads.mockRestore(); writes.mockRestore(); + const otherId = randomUUID(); + await db.insert(companies).values({ id: otherId, name: "Healthy tenant", issuePrefix: `H${otherId.slice(0, 6)}` }); + const [otherIssue] = await db.insert(issues).values({ companyId: otherId, title: "Other PR", status: "todo" }).returning(); + await workProductService(db).createForIssue(otherIssue.id, otherId, { type: "pull_request", provider: "github", title: "Other PR", status: "open" }); + const otherApproval = await approvalRecords(db).create(otherId, { type: "request_board_approval", payload: {} }); + await issueApprovalService(db).link(otherIssue.id, otherApproval.id); + await vi.waitFor(async () => { + await vi.advanceTimersByTimeAsync(1000); + expect(await db.select().from(decisionQueueItems).where(eq(decisionQueueItems.companyId, otherId))) + .toEqual([expect.objectContaining({ sourceId: otherApproval.id })]); + }, { timeout: 5000 }); + expect(await memberships()).toEqual([]); + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); errors.length = 0; + await vi.advanceTimersByTimeAsync(60_000); + await seeded(approval.id); await quiet(); + }); + + it("drains multiple pages without duplicating memberships or audit events", async () => { + const target = await issue(); await product(target.id); + const rows = await db.insert(approvals).values(Array.from({ length: 101 }, () => ({ companyId, type: "request_board_approval", payload: {} }))).returning(); + await db.insert(issueApprovals).values(rows.map(row => ({ companyId, issueId: target.id, approvalId: row.id }))); + await start(); + await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(1_000); expect(await memberships()).toHaveLength(101); }, { timeout: 10_000 }); + await quiet(); + await attentionService(db).list(companyId, { all: true, allowUnscopedAll: true }); await quiet(); + expect(await memberships()).toHaveLength(101); + expect(await db.select().from(decisionTriageEvents).where(eq(decisionTriageEvents.action, "queue_item.seeded"))).toHaveLength(101); + }); + + it("preserves the seed rule after manual removal of a still-matching item", async () => { + const approval = await pending(); await start(); await seeded(approval.id); await quiet(); + await decisionQueueService(db).removeItem({ ...mutation(), sourceKind: "approval", sourceId: approval.id }); + await seeded(approval.id); await quiet(); expect(await memberships()).toHaveLength(1); + }); + it("pages past failed projections and revisits them after recovery", async () => { + const target = await issue(); await product(target.id); + const rows = await db.insert(approvals).values(Array.from({ length: 101 }, () => ({ companyId, type: "request_board_approval", payload: {} }))).returning(); + await db.insert(issueApprovals).values(rows.map(row => ({ companyId, issueId: target.id, approvalId: row.id }))); + const last = rows.map(row => row.id).sort().at(-1)!; + await db.execute(sql`create function approval_queue_fail() returns trigger language plpgsql as $$ begin + if new.source_id <> TG_ARGV[0] then raise exception 'projection unavailable'; end if; return new; end $$`); + await db.execute(sql`create trigger approval_queue_fail before insert on decision_queue_items for each row execute function approval_queue_fail(${sql.raw("'" + last + "'")})`); + const service = approvalQueueSeedingService(db); + await expect(service.sweep()).rejects.toThrow("Approval queue seeding failed"); + await vi.advanceTimersByTimeAsync(1000); + await service.sweep(); expect((await memberships()).map(row => row.sourceId)).toEqual([last]); + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); + await service.sweep(); expect(await memberships()).toHaveLength(101); expect(await service.next()).toBeNull(); + }); + + it("retries a failed membership while new matching pages keep arriving", async () => { + const target = await issue(); await product(target.id); + const id = (n: number) => `00000000-0000-4000-8000-${String(n).padStart(12, "0")}`; + const append = async (start: number, count: number) => { + const rows = await db.insert(approvals).values(Array.from({ length: count }, (_, offset) => ({ + id: id(start + offset), companyId, type: "request_board_approval", payload: {}, + }))).returning(); + await db.insert(issueApprovals).values(rows.map(row => ({ companyId, issueId: target.id, approvalId: row.id }))); + }; + const service = approvalQueueSeedingService(db); await service.sweep(); + await append(1, 101); service.request(new Set([companyId])); + await db.execute(sql`create function approval_queue_fail() returns trigger language plpgsql as $$ begin + if new.source_id = TG_ARGV[0] then raise exception 'projection unavailable'; end if; return new; end $$`); + await db.execute(sql`create trigger approval_queue_fail before insert on decision_queue_items for each row execute function approval_queue_fail(${sql.raw("'" + id(1) + "'")})`); + await expect(service.sweep()).rejects.toThrow("Failed query"); + await append(102, 100); service.request(new Set([companyId])); + await vi.advanceTimersByTimeAsync(1000); await service.sweep(); + await append(202, 100); service.request(new Set([companyId])); + await db.execute(sql`drop trigger approval_queue_fail on decision_queue_items`); + await service.sweep(); + expect((await memberships()).some(row => row.sourceId === id(1))).toBe(true); + for (let pass = 0; pass < 4 && service.next() !== null; pass++) await service.sweep(); + expect(await memberships()).toHaveLength(301); expect(service.next()).toBeNull(); + }); + + it("rechecks the approval after a concurrent resolution commits", async () => { + const approval = await pending(); let release!: () => void, locked!: () => void; + const gate = new Promise(resolve => { release = resolve; }), admitted = new Promise(resolve => { locked = resolve; }); + const write = db.transaction(async tx => { + await tx.execute(sql`select id from approvals where id = ${approval.id} for update`); locked(); await gate; + await approvalRecords(tx as unknown as Db).resolveApproval(approval.id, "approved", "board-user", null); + }); + await admitted; const sweep = approvalQueueSeedingService(db).sweep(); + await vi.advanceTimersByTimeAsync(100); release(); await write; await sweep; + expect(await memberships()).toEqual([]); + }); + + it("preserves startup debt during drain and reconstructs unprocessed work after shutdown", async () => { + const approval = await pending(); draining = true; await start(); + const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(60_000); expect(reads).not.toHaveBeenCalled(); reads.mockRestore(); + await coordinator!.stop(); coordinator = undefined; + expect(await memberships()).toEqual([]); + draining = false; await start(); await seeded(approval.id); await quiet(); + }); + + it("preserves seeding for kept and archived approvals", async () => { + const approval = await pending(); + await db.insert(decisionRetention).values({ companyId, sourceKind: "approval", sourceId: approval.id, + sourceActivityAt: approval.updatedAt, keep: true, archivedAt: new Date() }); + await start(); await seeded(approval.id); await quiet(); + }); + + it("waits for an admitted transaction on shutdown and leaves the next row recoverable", async () => { + await pending(); await pending(); + let release!: () => void, admitted!: () => void; + const gate = new Promise(resolve => { release = resolve; }), entered = new Promise(resolve => { admitted = resolve; }); + const transaction = db.transaction.bind(db); + const intercepted = vi.spyOn(db, "transaction").mockImplementationOnce(async callback => { + admitted(); await gate; return transaction(callback); + }); + const started = start(); await entered; + let stopped = false; + const stopping = coordinator!.stop().then(() => { stopped = true; }); + await vi.advanceTimersByTimeAsync(100); expect(stopped).toBe(false); + release(); await started; await stopping; coordinator = undefined; intercepted.mockRestore(); + expect(await memberships()).toHaveLength(1); + await start(); await quiet(); expect(await memberships()).toHaveLength(2); + }); + + it("serializes manual removal and automatic seeding without a queue/item lock cycle", async () => { + const approval = await pending(); + const [link] = await db.select().from(issueApprovals).where(eq(issueApprovals.approvalId, approval.id)); + const queues = decisionQueueService(db); + const seed = () => queues.materializeSeedSources(companyId, [{ sourceKind: "approval", sourceId: approval.id, issueId: link.issueId }]); + await seed(); + await db.execute(sql`create function approval_queue_delete_gate() returns trigger language plpgsql as $$ begin + perform pg_advisory_xact_lock(hashtext(TG_ARGV[0])); return old; end $$`); + await db.execute(sql`create trigger approval_queue_delete_gate after delete on decision_queue_items for each row execute function approval_queue_delete_gate(${sql.raw("'" + companyId + "'")})`); + const blocker = await db.$client.reserve(); + await blocker`select pg_advisory_lock(hashtext(${companyId}))`; + let complete: Promise[]> | undefined; + try { + const removed = queues.removeItem({ ...mutation(), sourceKind: "approval", sourceId: approval.id }); + await vi.waitFor(async () => { + const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity + where datname = current_database() and state = 'active' and wait_event = 'advisory'`); + expect(row.count).toBeGreaterThan(0); + }, { timeout: 5000 }); + complete = Promise.allSettled([removed, seed()]); + await vi.waitFor(async () => { + const [row] = await db.execute<{ count: number }>(sql`select count(*)::int as count from pg_stat_activity + where datname = current_database() and state = 'active' and wait_event_type = 'Lock' and wait_event <> 'advisory'`); + expect(row.count).toBeGreaterThan(0); + }, { timeout: 5000 }); + } finally { + await blocker`select pg_advisory_unlock_all()`; blocker.release(); + } + const results = await complete!; + expect(results.map(result => result.status), results.filter(result => result.status === "rejected").map(result => String(result.reason?.cause?.message ?? result.reason)).join("; ")).toEqual(["fulfilled", "fulfilled"]); + expect(await memberships()).toHaveLength(1); + }); + +}); diff --git a/server/src/__tests__/decision-queues-routes.test.ts b/server/src/__tests__/decision-queues-routes.test.ts index b1897d06469..9546708b3d4 100644 --- a/server/src/__tests__/decision-queues-routes.test.ts +++ b/server/src/__tests__/decision-queues-routes.test.ts @@ -1,3 +1,4 @@ +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; import { randomUUID } from "node:crypto"; import express from "express"; import request from "supertest"; @@ -8,7 +9,7 @@ import { agents, approvals, companies, - createDb, + createDb, applyPendingMigrations, decisionQueueItems, decisionQueues, decisionTriage, @@ -26,7 +27,8 @@ import { errorHandler } from "../middleware/index.js"; import { decisionQueueRoutes } from "../routes/decision-queues.js"; import { decisionQueueService } from "../services/decision-queues.js"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +const external = resolveExternalTestDatabaseUrl("PAPERCLIP_DECISION_QUEUES_TEST_DATABASE_URL"); +const embeddedPostgresSupport = external ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -40,8 +42,9 @@ describeEmbeddedPostgres("decision queue routes", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-decision-queues-"); - db = createDb(tempDb.connectionString); + if (external) await applyPendingMigrations(external); + else tempDb = await startEmbeddedPostgresTestDatabase("paperclip-decision-queues-"); + db = createDb(external ?? tempDb!.connectionString); }, 30_000); afterEach(async () => { @@ -59,6 +62,7 @@ describeEmbeddedPostgres("decision queue routes", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); diff --git a/server/src/__tests__/environment-cleanup-work.test.ts b/server/src/__tests__/environment-cleanup-work.test.ts index 6a251dabdd0..5d71d2d4d42 100644 --- a/server/src/__tests__/environment-cleanup-work.test.ts +++ b/server/src/__tests__/environment-cleanup-work.test.ts @@ -228,7 +228,12 @@ describe.skipIf(!support.supported)("environment cleanup scheduling", () => { for (let i = 0; i < 33; i++) { const lease = await pending(); await environmentService(db).releaseLease(lease.id, "failed", { cleanupStatus: "success" }); } - await start(true); await settle(300_000); await settle(300_000); expect(inspections).toHaveBeenCalledTimes(33); await quiet(); + await start(true); + await vi.waitFor(async () => { + await settle(300_000); + expect(inspections).toHaveBeenCalledTimes(33); + }, { timeout: 5000 }); + await quiet(); }); it("keeps unknown historical resources scheduled until provider confirmation", async () => { const lease = await pending(); await environmentService(db).releaseLease(lease.id, "failed", { cleanupStatus: "success" }); diff --git a/server/src/__tests__/issue-comment-reopen-routes.test.ts b/server/src/__tests__/issue-comment-reopen-routes.test.ts index ba1df3c8b86..28c6900a190 100644 --- a/server/src/__tests__/issue-comment-reopen-routes.test.ts +++ b/server/src/__tests__/issue-comment-reopen-routes.test.ts @@ -1,8 +1,11 @@ import express from "express"; import request from "supertest"; import { getTableName } from "drizzle-orm"; -import { beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; -import { HttpError } from "../errors.js"; +import { beforeEach, describe, expect, it, vi } from "vitest"; +import { HttpError, unprocessable } from "../errors.js"; +import { normalizeIssueExecutionPolicy } from "../services/issue-execution-policy.js"; +import { issueRoutes } from "../routes/issues.js"; +import { errorHandler } from "../middleware/index.js"; vi.mock("../services/fast-responses.js", () => ({ enqueueFastResponse: vi.fn(async () => undefined), @@ -233,10 +236,6 @@ async function installActor( app: express.Express, actor?: Record, ) { - const [{ issueRoutes }, { errorHandler }] = await Promise.all([ - import("../routes/issues.js"), - import("../middleware/index.js"), - ]); app.use((req, _res, next) => { (req as any).actor = actor ?? { type: "board", @@ -261,8 +260,6 @@ async function normalizePolicy(input: { >; }>; }) { - const { normalizeIssueExecutionPolicy } = - await import("../services/issue-execution-policy.js"); return normalizeIssueExecutionPolicy(input); } @@ -376,15 +373,6 @@ function createScheduledRetry() { } describe("issue comment reopen routes", () => { - beforeAll(async () => { - // Load the route graph within the setup budget. A first-test import can - // time out and continue into the next test after its mocks are reset. - await Promise.all([ - import("../routes/issues.js"), - import("../middleware/index.js"), - ]); - }); - beforeEach(() => { vi.clearAllMocks(); mockIssueService.getById.mockReset(); @@ -3675,7 +3663,6 @@ describe("issue comment reopen routes", () => { authorAgentId: reviewerAgentId, authorUserId: null, }); - const { unprocessable } = await import("../errors.js"); mockIssueService.update.mockRejectedValue( unprocessable("Issue can only have one assignee"), ); diff --git a/server/src/__tests__/server-startup-feedback-export.test.ts b/server/src/__tests__/server-startup-feedback-export.test.ts index b08a2ed13e1..d0bb6ab1c33 100644 --- a/server/src/__tests__/server-startup-feedback-export.test.ts +++ b/server/src/__tests__/server-startup-feedback-export.test.ts @@ -1,3 +1,4 @@ +import { approvalQueueSeedingService } from "../services/approval-queue-seeding-work.js"; import { joinRequestRetentionService } from "../services/join-request-retention.js"; import { approvalRetentionService } from "../services/approval-retention.js"; import { createSecretProposalsService } from "../services/secret-proposals.js"; @@ -337,6 +338,11 @@ vi.mock("../services/join-request-retention.js", async importOriginal => ({ joinRequestRetentionService: vi.fn(() => ({ sweep: vi.fn(async () => ({ archived: 0 })), next: vi.fn(async () => null) })), })); +vi.mock("../services/approval-queue-seeding-work.js", async importOriginal => ({ + ...await importOriginal(), + approvalQueueSeedingService: vi.fn(() => ({ sweep: vi.fn(async () => undefined), next: vi.fn(() => null), request: vi.fn() })), +})); + vi.mock("../services/approval-retention.js", async importOriginal => ({ ...await importOriginal(), approvalRetentionService: vi.fn(() => ({ sweep: vi.fn(async () => ({ archived: 0 })), next: vi.fn(async () => null) })), @@ -542,6 +548,21 @@ describe("startServer feedback export wiring", () => { process.env.BETTER_AUTH_SECRET = "test-secret"; }); + it.each([false, true])("registers approval queue seeding only when maintenance is enabled (%s)", async enabled => { + loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); + await startServer(); await settleStartupHousekeeping(); + const registration = deliveryWorkRegisterMock.mock.calls.find(([queue]) => queue === DELIVERY_QUEUES.approvalQueueSeeding); + expect(Boolean(registration)).toBe(enabled); + if (enabled) { + const service = vi.mocked(approvalQueueSeedingService).mock.results.at(-1)!.value; + await registration![1].run(new AbortController().signal); + expect(service.sweep).toHaveBeenCalled(); + expect(registration![1].nextRunAt()).toBeNull(); + expect(await registration![1].hasPending()).toBe(false); + expect(service.next).toHaveBeenCalled(); + } + }); + it.each([false, true])("registers approval retention only when maintenance is enabled (%s)", async enabled => { loadConfigMock.mockReturnValue(buildTestConfig({ heartbeatSchedulerEnabled: enabled })); await startServer(); await settleStartupHousekeeping(); diff --git a/server/src/__tests__/task-watchdogs-scheduler.test.ts b/server/src/__tests__/task-watchdogs-scheduler.test.ts index e5fbf969ab4..e5bfe047adb 100644 --- a/server/src/__tests__/task-watchdogs-scheduler.test.ts +++ b/server/src/__tests__/task-watchdogs-scheduler.test.ts @@ -7,7 +7,7 @@ import { agents, approvals, companies, - createDb, + createDb, applyPendingMigrations, documents, heartbeatRuns, issueComments, @@ -28,7 +28,10 @@ import { } from "./helpers/embedded-postgres.js"; import { taskWatchdogService } from "../services/task-watchdogs.ts"; -const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); +import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js"; + +const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_WATCHDOG_TEST_DATABASE_URL"); +const embeddedPostgresSupport = externalUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport(); const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; if (!embeddedPostgresSupport.supported) { @@ -42,8 +45,9 @@ describeEmbeddedPostgres("task watchdog scheduler", () => { let tempDb: Awaited> | null = null; beforeAll(async () => { - tempDb = await startEmbeddedPostgresTestDatabase("paperclip-task-watchdogs-scheduler-"); - db = createDb(tempDb.connectionString); + if (externalUrl) await applyPendingMigrations(externalUrl); + else tempDb = await startEmbeddedPostgresTestDatabase("paperclip-task-watchdogs-scheduler-"); + db = createDb(externalUrl ?? tempDb!.connectionString); }, 20_000); afterEach(async () => { @@ -64,6 +68,7 @@ describeEmbeddedPostgres("task watchdog scheduler", () => { }); afterAll(async () => { + await db?.$client.end({ timeout: 1 }); await tempDb?.cleanup(); }); @@ -96,16 +101,18 @@ describeEmbeddedPostgres("task watchdog scheduler", () => { return id; } + let nextFixtureIssueNumber = 10_000; async function seedIssue(companyId: string, overrides: Partial = {}) { const id = overrides.id ?? randomUUID(); + const issueNumber = overrides.issueNumber ?? ++nextFixtureIssueNumber; await db.insert(issues).values({ id, companyId, title: overrides.title ?? "Watched issue", status: overrides.status ?? "done", priority: overrides.priority ?? "medium", - identifier: overrides.identifier ?? `WDOG-${Math.floor(Math.random() * 10_000)}`, - issueNumber: overrides.issueNumber ?? Math.floor(Math.random() * 10_000), + identifier: overrides.identifier ?? `WDOG-${issueNumber}`, + issueNumber, parentId: overrides.parentId, assigneeAgentId: overrides.assigneeAgentId, originKind: overrides.originKind, diff --git a/server/src/index.ts b/server/src/index.ts index a69a95b526d..17680e3d035 100644 --- a/server/src/index.ts +++ b/server/src/index.ts @@ -1,3 +1,4 @@ +import { approvalQueueSeedingService, registerApprovalQueueSeedingWork } from "./services/approval-queue-seeding-work.js"; import { joinRequestRetentionService, registerJoinRequestRetentionWork } from "./services/join-request-retention.js"; import { approvalRetentionService, registerApprovalRetentionWork } from "./services/approval-retention.js"; import { registerDecisionTargetWork } from "./services/decision-target-work.js"; @@ -1861,6 +1862,8 @@ async function startServerWithDatabaseTeardown( await step("environment_lease_cleanup", startEnvironmentCleanup); await step("archive_notification_worker", () => registerDecisionArchiveDeliveryWork(deliveryWork, retentionExecutor, () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); + await step("approval_queue_seeding", () => registerApprovalQueueSeedingWork(deliveryWork, approvalQueueSeedingService(db as any), + () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("approval_retention", () => registerApprovalRetentionWork(deliveryWork, approvalRetentionService(db as any), () => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready); await step("join_request_retention", () => registerJoinRequestRetentionWork(deliveryWork, joinRequestRetentionService(db as any), diff --git a/server/src/lib/approval-records.ts b/server/src/lib/approval-records.ts index 7bd14df166a..bfa7a38b3dc 100644 --- a/server/src/lib/approval-records.ts +++ b/server/src/lib/approval-records.ts @@ -1,5 +1,5 @@ import { notifyToolReviewChange } from "../services/tool-review-work-notifications.js"; -import { writeApprovalChange } from "../services/retention-work-notifications.js"; +import { writeApprovalChange } from "../services/approval-work-notifications.js"; import { and, eq, inArray } from "drizzle-orm"; import { approvals, type Db } from "@paperclipai/db"; import { notFound, unprocessable } from "../errors.js"; @@ -39,7 +39,7 @@ export function approvalResolutionRecords(db: Db) { } const now = new Date(); - const updated = await writeApprovalChange(database, async tx => { + const updated = await writeApprovalChange(database, existing.companyId, async tx => { await notifyToolReviewChange(tx, existing.companyId); return tx.update(approvals) .set({ diff --git a/server/src/services/approval-queue-seeding-work.ts b/server/src/services/approval-queue-seeding-work.ts new file mode 100644 index 00000000000..6a4b3472799 --- /dev/null +++ b/server/src/services/approval-queue-seeding-work.ts @@ -0,0 +1,106 @@ +import { and, asc, desc, eq, gt, lte, sql, type SQL } from "drizzle-orm"; +import { approvals, companies, issues, issueWorkProducts, type Db } from "@paperclipai/db"; +import { firstApprovalIssueId } from "./approval-visibility.js"; +import { executionIssueCondition } from "./issue-visibility.js"; +import { pullRequestSeedProductCondition } from "./decision-queue-seed-rules.js"; +import { decisionQueueService } from "./decision-queues.js"; +import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js"; +import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; + +const linkedIssue = firstApprovalIssueId(); + +export function approvalQueueSeedingService(db: Db) { + const pending = new Map([[undefined, { generation: 0, failures: 0, retryAt: 0 }]]); + function request(companyIds: ReadonlySet | undefined) { + for (const companyId of companyIds ?? []) { + const existing = pending.get(companyId); + if (existing) existing.generation++; + else pending.set(companyId, { generation: 0, failures: 0, retryAt: 0 }); + } + } + function candidates(database: Db, companyId: string | undefined, condition?: SQL, order = asc) { + return database.select({ id: approvals.id, companyId: approvals.companyId, issueId: linkedIssue }).from(approvals) + .innerJoin(companies, eq(companies.id, approvals.companyId)) + .where(and(eq(approvals.status, "pending"), eq(companies.status, "active"), companyId === undefined ? undefined : eq(approvals.companyId, companyId), condition, + sql`exists (select 1 from ${issues} where ${issues.id} = ${linkedIssue} + and ${issues.companyId} = ${approvals.companyId} and ${executionIssueCondition()} + and exists (select 1 from ${issueWorkProducts} where ${pullRequestSeedProductCondition(approvals.companyId)} + and ${issueWorkProducts.issueId} = ${issues.id}))`, + sql`not exists (select 1 from decision_queues queue where queue.company_id = ${approvals.companyId} and queue.key = 'prs' + and (not queue.seed_rules_enabled or exists (select 1 from decision_queue_items item where item.queue_id = queue.id + and item.company_id = ${approvals.companyId} and item.source_kind = 'approval' and item.source_id = ${approvals.id}::text)))`, + )).orderBy(order(approvals.id)); + } + async function sweep(signal?: AbortSignal) { + const scope = [...pending.entries()].find(([, state]) => state.retryAt <= Date.now()); + if (!scope) return; + const [companyId, state] = scope; + try { + const generation = state.generation; + state.end ??= (await candidates(db, companyId, undefined, desc).limit(1))[0]?.id; + const rows = state.end ? await candidates(db, companyId, and(lte(approvals.id, state.end), + state.cursor ? gt(approvals.id, state.cursor) : undefined)).limit(100) : []; + const errors: unknown[] = []; + for (const row of rows) { + if (signal?.aborted) break; + try { + await db.transaction(async tx => { + const database = tx as unknown as Db; + const [current] = await candidates(database, row.companyId, eq(approvals.id, row.id)).for("update", { of: approvals }); + if (!current) return; + await decisionQueueService(database).materializeSeedSources(current.companyId, [{ + sourceKind: "approval", sourceId: current.id, issueId: current.issueId, + }]); + }); + } catch (error) { errors.push(error); } + state.cursor = row.id; + } + if (rows.length < 100) { state.cursor = undefined; state.end = undefined; } + const remaining = await candidates(db, companyId).limit(1); + if (!signal?.aborted && generation === state.generation && remaining.length === 0) pending.delete(companyId); + if (errors.length === 1) throw errors[0]; + if (errors.length) throw new AggregateError(errors, "Approval queue seeding failed"); + if (rows.length > 0 || remaining.length === 0) state.failures = 0; + state.retryAt = 0; + } catch (error) { + state.failures = Math.min(state.failures + 1, 7); + state.retryAt = Date.now() + Math.min(60_000, 1_000 * 2 ** (state.failures - 1)); + throw error; + } finally { + if (pending.has(companyId)) { + pending.delete(companyId); + pending.set(companyId, state); + } + } + } + function next() { + let earliest: number | null = null; + for (const state of pending.values()) earliest = earliest === null ? state.retryAt : Math.min(earliest, state.retryAt); + return earliest; + } + return { sweep, next, request }; +} + +export function registerApprovalQueueSeedingWork( + coordinator: ReturnType, + service: ReturnType, + canRun: () => boolean, +) { + let earliest = 0; + return coordinator.register(DELIVERY_QUEUES.approvalQueueSeeding, { + retryMs: 1_000, + pausePassiveMaintenance: true, + onSettled: service.request, + run: async signal => { + if (!canRun()) return; + if (Date.now() < earliest) return; + earliest = Date.now() + 1_000; + await service.sweep(signal); + }, + hasPending: async () => (service.next() ?? Infinity) <= Date.now(), + nextRunAt: () => { + const next = service.next(); + return next === null ? null : Math.max(earliest, next); + }, + }); +} diff --git a/server/src/services/approval-work-notifications.ts b/server/src/services/approval-work-notifications.ts new file mode 100644 index 00000000000..5c914477d15 --- /dev/null +++ b/server/src/services/approval-work-notifications.ts @@ -0,0 +1,16 @@ +import type { Db } from "@paperclipai/db"; +import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; + +export const notifyApprovalQueueWork = (transaction: object, companyId: string) => notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalQueueSeeding, companyId); + +export async function notifyApprovalWork(transaction: object, companyId: string) { + await notifyDeliveryWork(transaction, DELIVERY_QUEUES.approvalRetention); + await notifyApprovalQueueWork(transaction, companyId); +} + +export function writeApprovalChange(db: Db, companyId: string, write: (transaction: Db) => Promise): Promise { + return db.transaction(async tx => { + await notifyApprovalWork(tx, companyId); + return write(tx as unknown as Db); + }); +} diff --git a/server/src/services/approvals.ts b/server/src/services/approvals.ts index 1095766c3ef..cfa7624eee8 100644 --- a/server/src/services/approvals.ts +++ b/server/src/services/approvals.ts @@ -1,5 +1,5 @@ import { notifyToolReviewChange } from "./tool-review-work-notifications.js"; -import { writeApprovalChange } from "./retention-work-notifications.js"; +import { writeApprovalChange } from "./approval-work-notifications.js"; import { approvalResolutionRecords, resolvableApprovalStatuses as resolvableStatuses } from "../lib/approval-records.js"; import { notifyHireApproved } from "./hire-hook.js"; @@ -55,7 +55,7 @@ export function approvalRecords(db: Db) { }, create: (companyId: string, data: Omit) => - writeApprovalChange(db, tx => tx + writeApprovalChange(db, companyId, tx => tx .insert(approvals) .values({ ...data, companyId }) .returning() @@ -68,7 +68,7 @@ export function approvalRecords(db: Db) { const now = new Date(); const [existing] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, id)); if (!existing) return null; - const updated = await writeApprovalChange(db, async tx => { + const updated = await writeApprovalChange(db, existing.companyId, async tx => { await notifyToolReviewChange(tx, existing.companyId); return tx.update(approvals) .set({ @@ -91,7 +91,7 @@ export function approvalRecords(db: Db) { } const now = new Date(); - return writeApprovalChange(db, tx => tx + return writeApprovalChange(db, existing.companyId, tx => tx .update(approvals) .set({ status: "revision_requested", @@ -112,7 +112,7 @@ export function approvalRecords(db: Db) { } const now = new Date(); - return writeApprovalChange(db, tx => tx + return writeApprovalChange(db, existing.companyId, tx => tx .update(approvals) .set({ status: "pending", diff --git a/server/src/services/budgets.ts b/server/src/services/budgets.ts index aba74d6e0c6..ccff957a9ed 100644 --- a/server/src/services/budgets.ts +++ b/server/src/services/budgets.ts @@ -1,4 +1,5 @@ -import { notifyCompanyRetentionWork, notifyRetentionWork } from "./retention-work-notifications.js"; +import { notifyApprovalWork } from "./approval-work-notifications.js"; +import { notifyCompanyRetentionWork } from "./retention-work-notifications.js"; import { addCents, compareCents, normalizeCents } from "@paperclipai/shared"; import { and, desc, eq, inArray, sql, or, isNull } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; @@ -259,7 +260,9 @@ async function markApprovalStatus( decidedByUserId: string, ) { if (!approvalId) return; - await notifyRetentionWork(db, "approval"); + const [approval] = await db.select({ companyId: approvals.companyId }).from(approvals).where(eq(approvals.id, approvalId)); + if (!approval) return; + await notifyApprovalWork(db, approval.companyId); await db .update(approvals) .set({ @@ -297,7 +300,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.paused) return; - await notifyCompanyRetentionWork(db); + await notifyCompanyRetentionWork(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync); await db @@ -354,7 +357,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica await notifyDeliveryWork(db, DELIVERY_QUEUES.skillSourceSync); const scope = await resolveScopeRecord(db, "company", policy.scopeId); if (scope.pauseReason !== "budget") return; - await notifyCompanyRetentionWork(db); + await notifyCompanyRetentionWork(db, policy.companyId); await notifyDeliveryWork(db, DELIVERY_QUEUES.agentLifecycle, policy.companyId); await db .update(companies) @@ -457,7 +460,7 @@ export function budgetServiceInTransaction(db: Db, publications: ActivityPublica windowEnd: end, }); - if (thresholdType === "hard") await notifyRetentionWork(db, "approval"); + if (thresholdType === "hard") await notifyApprovalWork(db, policy.companyId); const approval = thresholdType === "hard" ? await db .insert(approvals) diff --git a/server/src/services/companies.ts b/server/src/services/companies.ts index 4f17b854339..580df76b6e9 100644 --- a/server/src/services/companies.ts +++ b/server/src/services/companies.ts @@ -331,7 +331,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (companyPatch.status !== undefined && companyPatch.status !== existing.status) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); - await notifyCompanyRetentionWork(tx); + await notifyCompanyRetentionWork(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); } const updated = await tx @@ -483,7 +483,7 @@ export function companyService(db: Db, _budgetHooks: BudgetServiceHooks = {}) { if (!wasAlreadyArchived) { await notifyDeliveryWork(tx, DELIVERY_QUEUES.skillSourceSync); - await notifyCompanyRetentionWork(tx); + await notifyCompanyRetentionWork(tx, id); await notifyDeliveryWork(tx, DELIVERY_QUEUES.agentLifecycle, id); await tx .update(companies) diff --git a/server/src/services/company-deletion.ts b/server/src/services/company-deletion.ts index e6f716faca3..ba9d8f758ac 100644 --- a/server/src/services/company-deletion.ts +++ b/server/src/services/company-deletion.ts @@ -97,7 +97,7 @@ export async function deleteCompany(db: Db, id: string) { await tx.delete(agentRuntimeState).where(eq(agentRuntimeState.companyId, id)); await tx.delete(issueComments).where(eq(issueComments.companyId, id)); await tx.delete(approvalComments).where(eq(approvalComments.companyId, id)); - await notifyCompanyRetentionWork(tx); + await notifyCompanyRetentionWork(tx, id); await tx.delete(approvals).where(eq(approvals.companyId, id)); await tx.delete(companySecrets).where(eq(companySecrets.companyId, id)); await tx.delete(joinRequests).where(eq(joinRequests.companyId, id)); diff --git a/server/src/services/decision-queue-seed-rules.ts b/server/src/services/decision-queue-seed-rules.ts new file mode 100644 index 00000000000..c46b0cde464 --- /dev/null +++ b/server/src/services/decision-queue-seed-rules.ts @@ -0,0 +1,6 @@ +import { and, eq, type SQLWrapper } from "drizzle-orm"; +import { issueWorkProducts } from "@paperclipai/db"; + +export function pullRequestSeedProductCondition(companyId: string | SQLWrapper) { + return and(eq(issueWorkProducts.companyId, companyId), eq(issueWorkProducts.type, "pull_request")); +} diff --git a/server/src/services/decision-queues.ts b/server/src/services/decision-queues.ts index d529e1d0a8d..b0c871aa085 100644 --- a/server/src/services/decision-queues.ts +++ b/server/src/services/decision-queues.ts @@ -1,3 +1,5 @@ +import { pullRequestSeedProductCondition } from "./decision-queue-seed-rules.js"; +import { notifyApprovalQueueWork } from "./approval-work-notifications.js"; import { and, desc, eq, inArray, isNull, sql } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { @@ -43,6 +45,14 @@ export type DecisionMutationActor = { responsibleUserId: string | null; }; +export type DecisionQueueSeedSource = { + sourceKind: AttentionSourceKind; + sourceId: string; + issueId: string | null; + isPlanTarget?: boolean; + interactionKind?: string; +}; + type SeedDefinition = { key: string; title: string; @@ -367,6 +377,12 @@ export function decisionQueueService(db: Db) { .then((rows) => rows[0] ?? null); } + async function lockQueue(database: Db, companyId: string, id: string) { + const [queue] = await database.select({ id: decisionQueues.id }).from(decisionQueues) + .where(and(eq(decisionQueues.companyId, companyId), eq(decisionQueues.id, id))).for("update"); + if (!queue) throw notFound("Decision queue not found"); + } + async function visibleItems(companyId: string, queueId: string, authActor: AuthorizationActor) { const rows = await db.select().from(decisionQueueItems) .where(and(eq(decisionQueueItems.companyId, companyId), eq(decisionQueueItems.queueId, queueId))) @@ -380,6 +396,103 @@ export function decisionQueueService(db: Db) { return visible; } + async function materializeSeedSources(companyId: string, items: DecisionQueueSeedSource[]) { + if (items.length === 0) return; + const issueIds = [...new Set(items.map(item => item.issueId).filter((id): id is string => Boolean(id)))]; + const prIssueIds = new Set(issueIds.length === 0 ? [] : await db + .select({ issueId: issueWorkProducts.issueId }) + .from(issueWorkProducts) + .where(and( + pullRequestSeedProductCondition(companyId), + inArray(issueWorkProducts.issueId, issueIds), + )).then((rows) => rows.map((row) => row.issueId))); + + const matches = new Map(); + for (const item of items) { + if (prIssueIds.has(item.issueId ?? "")) { + matches.set("prs", [...(matches.get("prs") ?? []), item]); + } + if (item.sourceKind === "issue_thread_interaction" && item.isPlanTarget === true) { + matches.set("plans", [...(matches.get("plans") ?? []), item]); + } + if (item.sourceKind === "issue_thread_interaction" && item.interactionKind === "ask_user_questions") { + matches.set("questions", [...(matches.get("questions") ?? []), item]); + } + } + + for (const seed of DECISION_QUEUE_SEEDS) { + const matchingItems = matches.get(seed.key) ?? []; + if (matchingItems.length === 0) continue; + await db.transaction(async (tx) => { + const txDb = tx as unknown as Db; + const insertedQueue = await txDb.insert(decisionQueues).values({ + companyId, + key: seed.key, + title: seed.title, + description: seed.description, + createdByType: "system", + seedRules: seed.rules, + seedRulesEnabled: true, + }).onConflictDoNothing({ target: [decisionQueues.companyId, decisionQueues.key] }).returning(); + const queue = insertedQueue[0] ?? await txDb.select().from(decisionQueues) + .where(and(eq(decisionQueues.companyId, companyId), eq(decisionQueues.key, seed.key))) + .for("update") + .then((rows) => rows[0] ?? null); + if (!queue || !queue.seedRulesEnabled) return; + if (insertedQueue[0]) { + await txDb.insert(decisionTriageEvents).values({ + companyId, + queueId: queue.id, + action: "queue.seeded", + ...eventActorColumns(SYSTEM_ACTOR), + details: { key: seed.key, rules: seed.rules.map((rule) => rule.key) }, + }); + await recordActivity(txDb, SYSTEM_ACTOR, { + companyId, + action: "decision_queue.seeded", + entityType: "decision_queue", + entityId: queue.id, + details: { key: seed.key }, + }); + } + let insertedAnyItem = false; + for (const item of matchingItems) { + const inserted = await txDb.insert(decisionQueueItems).values({ + companyId, + queueId: queue.id, + sourceKind: item.sourceKind, + sourceId: item.sourceId, + addedByType: "system", + }).onConflictDoNothing({ + target: [decisionQueueItems.queueId, decisionQueueItems.sourceKind, decisionQueueItems.sourceId], + }).returning(); + if (!inserted[0]) continue; + insertedAnyItem = true; + await txDb.insert(decisionTriageEvents).values({ + companyId, + queueId: queue.id, + sourceKind: item.sourceKind, + sourceId: item.sourceId, + action: "queue_item.seeded", + ...eventActorColumns(SYSTEM_ACTOR), + details: { seedKey: seed.key }, + }); + await recordActivity(txDb, SYSTEM_ACTOR, { + companyId, + action: "decision_queue_item.seeded", + entityType: "decision_queue", + entityId: queue.id, + details: { sourceKind: item.sourceKind, sourceId: item.sourceId, seedKey: seed.key }, + }); + } + if (insertedAnyItem) { + await txDb.update(decisionQueues).set({ updatedAt: new Date() }) + .where(and(eq(decisionQueues.companyId, companyId), eq(decisionQueues.id, queue.id))); + } + }); + } + } + return { create: async (input: { companyId: string; @@ -392,6 +505,7 @@ export function decisionQueueService(db: Db) { }) => { const result = await db.transaction(async (tx) => { const txDb = tx as unknown as Db; + if (input.key === "prs") await notifyApprovalQueueWork(txDb, input.companyId); const inserted = await txDb.insert(decisionQueues).values({ companyId: input.companyId, key: input.key, @@ -449,6 +563,7 @@ export function decisionQueueService(db: Db) { }) => { const row = await db.transaction(async (tx) => { const txDb = tx as unknown as Db; + if (input.key === "prs") await notifyApprovalQueueWork(txDb, input.companyId); const updated = await txDb.update(decisionQueues).set({ ...input.patch, updatedAt: new Date() }) .where(and(eq(decisionQueues.companyId, input.companyId), eq(decisionQueues.key, input.key))) .returning().then((rows) => rows[0] ?? null); @@ -491,6 +606,7 @@ export function decisionQueueService(db: Db) { await requireSourceRead(db, input.authActor, input.companyId, input.sourceKind, input.sourceId); return db.transaction(async (tx) => { const txDb = tx as unknown as Db; + await lockQueue(txDb, input.companyId, queue.id); const inserted = await txDb.insert(decisionQueueItems).values({ companyId: input.companyId, queueId: queue.id, @@ -556,6 +672,8 @@ export function decisionQueueService(db: Db) { } return db.transaction(async (tx) => { const txDb = tx as unknown as Db; + await lockQueue(txDb, input.companyId, queue.id); + if (input.key === "prs" && input.sourceKind === "approval") await notifyApprovalQueueWork(txDb, input.companyId); const removed = await txDb.delete(decisionQueueItems).where(and( eq(decisionQueueItems.companyId, input.companyId), eq(decisionQueueItems.queueId, queue.id), @@ -682,101 +800,13 @@ export function decisionQueueService(db: Db) { }); }, - materializeSeededQueues: async (companyId: string, items: AttentionItem[]) => { - if (items.length === 0) return; - const issueIds = [...new Set(items.map(itemIssueId).filter((id): id is string => Boolean(id)))]; - const prIssueIds = new Set(issueIds.length === 0 ? [] : await db - .select({ issueId: issueWorkProducts.issueId }) - .from(issueWorkProducts) - .where(and( - eq(issueWorkProducts.companyId, companyId), - eq(issueWorkProducts.type, "pull_request"), - inArray(issueWorkProducts.issueId, issueIds), - )).then((rows) => rows.map((row) => row.issueId))); - - const matches = new Map(); - for (const item of items) { - if (prIssueIds.has(itemIssueId(item) ?? "")) { - matches.set("prs", [...(matches.get("prs") ?? []), item]); - } - if (item.sourceKind === "issue_thread_interaction" && item.subject.metadata?.isPlanTarget === true) { - matches.set("plans", [...(matches.get("plans") ?? []), item]); - } - if (item.sourceKind === "issue_thread_interaction" && item.subject.metadata?.kind === "ask_user_questions") { - matches.set("questions", [...(matches.get("questions") ?? []), item]); - } - } - - for (const seed of DECISION_QUEUE_SEEDS) { - const matchingItems = matches.get(seed.key) ?? []; - if (matchingItems.length === 0) continue; - await db.transaction(async (tx) => { - const txDb = tx as unknown as Db; - const insertedQueue = await txDb.insert(decisionQueues).values({ - companyId, - key: seed.key, - title: seed.title, - description: seed.description, - createdByType: "system", - seedRules: seed.rules, - seedRulesEnabled: true, - }).onConflictDoNothing({ target: [decisionQueues.companyId, decisionQueues.key] }).returning(); - const queue = insertedQueue[0] ?? await txDb.select().from(decisionQueues) - .where(and(eq(decisionQueues.companyId, companyId), eq(decisionQueues.key, seed.key))) - .then((rows) => rows[0] ?? null); - if (!queue || !queue.seedRulesEnabled) return; - if (insertedQueue[0]) { - await txDb.insert(decisionTriageEvents).values({ - companyId, - queueId: queue.id, - action: "queue.seeded", - ...eventActorColumns(SYSTEM_ACTOR), - details: { key: seed.key, rules: seed.rules.map((rule) => rule.key) }, - }); - await recordActivity(txDb, SYSTEM_ACTOR, { - companyId, - action: "decision_queue.seeded", - entityType: "decision_queue", - entityId: queue.id, - details: { key: seed.key }, - }); - } - let insertedAnyItem = false; - for (const item of matchingItems) { - const inserted = await txDb.insert(decisionQueueItems).values({ - companyId, - queueId: queue.id, - sourceKind: item.sourceKind, - sourceId: item.subject.id, - addedByType: "system", - }).onConflictDoNothing({ - target: [decisionQueueItems.queueId, decisionQueueItems.sourceKind, decisionQueueItems.sourceId], - }).returning(); - if (!inserted[0]) continue; - insertedAnyItem = true; - await txDb.insert(decisionTriageEvents).values({ - companyId, - queueId: queue.id, - sourceKind: item.sourceKind, - sourceId: item.subject.id, - action: "queue_item.seeded", - ...eventActorColumns(SYSTEM_ACTOR), - details: { seedKey: seed.key }, - }); - await recordActivity(txDb, SYSTEM_ACTOR, { - companyId, - action: "decision_queue_item.seeded", - entityType: "decision_queue", - entityId: queue.id, - details: { sourceKind: item.sourceKind, sourceId: item.subject.id, seedKey: seed.key }, - }); - } - if (insertedAnyItem) { - await txDb.update(decisionQueues).set({ updatedAt: new Date() }) - .where(and(eq(decisionQueues.companyId, companyId), eq(decisionQueues.id, queue.id))); - } - }); - } - }, + materializeSeedSources, + materializeSeededQueues: (companyId: string, items: AttentionItem[]) => materializeSeedSources(companyId, items.map(item => ({ + sourceKind: item.sourceKind, + sourceId: item.subject.id, + issueId: itemIssueId(item), + isPlanTarget: item.subject.metadata?.isPlanTarget === true, + interactionKind: typeof item.subject.metadata?.kind === "string" ? item.subject.metadata.kind : undefined, + }))), }; } diff --git a/server/src/services/delivery-work-notifications.ts b/server/src/services/delivery-work-notifications.ts index f6d1d6bb5f7..9a42db08f16 100644 --- a/server/src/services/delivery-work-notifications.ts +++ b/server/src/services/delivery-work-notifications.ts @@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = { executionStatus: "execution-status", executionDisposition: "execution-disposition", environmentCleanup: "environment-cleanup", + approvalQueueSeeding: "approval-queue-seeding", approvalRetention: "approval-retention", joinRequestRetention: "join-request-retention", decisionTargets: "decision-targets", diff --git a/server/src/services/issue-approvals.ts b/server/src/services/issue-approvals.ts index b00133898b0..682262ca899 100644 --- a/server/src/services/issue-approvals.ts +++ b/server/src/services/issue-approvals.ts @@ -1,4 +1,4 @@ -import { writeApprovalChange } from "./retention-work-notifications.js"; +import { writeApprovalChange } from "./approval-work-notifications.js"; import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { approvals, issueApprovals, issues } from "@paperclipai/db"; @@ -108,7 +108,7 @@ export function issueApprovalService(db: Db) { link: async (issueId: string, approvalId: string, actor?: LinkActor) => { const { issue } = await assertIssueAndApprovalSameCompany(issueId, approvalId); - await writeApprovalChange(db, tx => tx + await writeApprovalChange(db, issue.companyId, tx => tx .insert(issueApprovals) .values({ companyId: issue.companyId, @@ -128,7 +128,7 @@ export function issueApprovalService(db: Db) { unlink: async (issueId: string, approvalId: string) => { const { issue } = await assertIssueAndApprovalSameCompany(issueId, approvalId); - await writeApprovalChange(db, tx => tx + await writeApprovalChange(db, issue.companyId, tx => tx .delete(issueApprovals) .where(and(eq(issueApprovals.issueId, issueId), eq(issueApprovals.approvalId, approvalId)))); }, @@ -158,7 +158,7 @@ export function issueApprovalService(db: Db) { } } - await writeApprovalChange(db, tx => tx + await writeApprovalChange(db, approval.companyId, tx => tx .insert(issueApprovals) .values( uniqueIssueIds.map((issueId) => ({ diff --git a/server/src/services/issues.ts b/server/src/services/issues.ts index 53f3cff74a6..5c559559a70 100644 --- a/server/src/services/issues.ts +++ b/server/src/services/issues.ts @@ -1,4 +1,4 @@ -import { notifyRetentionWork } from "./retention-work-notifications.js"; +import { notifyApprovalWork } from "./approval-work-notifications.js"; import { notifyDecisionTargetWork, notifyDecisionTargetIssueRemoval } from "./decision-targets.js"; import { assertOnboardingFirstTaskCreationAllowed } from "./onboarding-first-task-admission.js"; @@ -11391,7 +11391,7 @@ export function issueService(db: Db) { await notifyDecisionTargetWork(tx, `issue:${id}`); } if (patch.hiddenAt !== undefined || patch.harnessKind !== undefined || patch.conversationAgentId !== undefined) { - await notifyRetentionWork(tx, "approval"); + await notifyApprovalWork(tx, receiptExisting.companyId); } const updated = await tx .update(issues) @@ -11800,7 +11800,8 @@ export function issueService(db: Db) { try { await notifyTerminalWorkspaceWork(tx, `issue:${id}`); await requestRestorePointDeletion(tx, { sourceIssueId: id }); - await notifyRetentionWork(tx, "approval"); + const [owner] = await tx.select({ companyId: issues.companyId }).from(issues).where(eq(issues.id, id)); + if (owner) await notifyApprovalWork(tx, owner.companyId); await notifyDecisionTargetIssueRemoval(tx, id); removedIssue = await tx .delete(issues) diff --git a/server/src/services/retention-work-notifications.ts b/server/src/services/retention-work-notifications.ts index a52ce2de883..ffedb0b1a4d 100644 --- a/server/src/services/retention-work-notifications.ts +++ b/server/src/services/retention-work-notifications.ts @@ -1,3 +1,4 @@ +import { notifyApprovalWork } from "./approval-work-notifications.js"; import type { Db } from "@paperclipai/db"; import type { AttentionSourceKind } from "@paperclipai/shared"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; @@ -7,15 +8,8 @@ export async function notifyRetentionWork(transaction: object, kind: AttentionSo if (kind === "join_request") await notifyDeliveryWork(transaction, DELIVERY_QUEUES.joinRequestRetention); } -export function writeApprovalChange(db: Db, write: (transaction: Db) => Promise): Promise { - return db.transaction(async tx => { - await notifyRetentionWork(tx, "approval"); - return write(tx as unknown as Db); - }); -} - -export async function notifyCompanyRetentionWork(transaction: object) { - await notifyRetentionWork(transaction, "approval"); +export async function notifyCompanyRetentionWork(transaction: object, companyId: string) { + await notifyApprovalWork(transaction, companyId); await notifyRetentionWork(transaction, "join_request"); } diff --git a/server/src/services/routines.ts b/server/src/services/routines.ts index 372c8d4f45d..9ea1884369c 100644 --- a/server/src/services/routines.ts +++ b/server/src/services/routines.ts @@ -1,4 +1,4 @@ -import { notifyRetentionWork } from "./retention-work-notifications.js"; +import { notifyApprovalWork } from "./approval-work-notifications.js"; import { notifyDecisionTargetIssueRemoval } from "./decision-targets.js"; import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js"; import { verifyAppWebhook } from "./app-webhook.js"; @@ -1966,7 +1966,7 @@ export function routineService( return updated ?? createdRun; } catch (error) { if (createdIssue) { - await notifyRetentionWork(txDb, "approval"); + await notifyApprovalWork(txDb, createdIssue.companyId); await notifyDecisionTargetIssueRemoval(txDb, createdIssue.id); await txDb.delete(issues).where(eq(issues.id, createdIssue.id)); } diff --git a/server/src/services/tool-gateway.ts b/server/src/services/tool-gateway.ts index ae81998e947..c5c9cdb07b6 100644 --- a/server/src/services/tool-gateway.ts +++ b/server/src/services/tool-gateway.ts @@ -1,4 +1,4 @@ -import { writeApprovalChange } from "./retention-work-notifications.js"; +import { writeApprovalChange } from "./approval-work-notifications.js"; import { toolReviewProjectionService } from "./tool-review-projection.js"; import { toolReviewInterruptionService } from "./tool-review-interruption.js"; import { bindVerifiedToolApprovalArguments, claimToolTestApproval, releaseToolApproval, persistToolTestOutcome, startToolInvocation, type ToolApprovalClaim } from "./tool-approval-execution.js"; @@ -2650,7 +2650,7 @@ export function createToolGatewayService( let formalApprovalId: string | null = null; if (toolRequiresFormalApproval(input.tool)) { - const [approval] = await writeApprovalChange(db, tx => tx + const [approval] = await writeApprovalChange(db, input.session.companyId, tx => tx .insert(approvals) .values({ companyId: input.session.companyId, @@ -2676,7 +2676,7 @@ export function createToolGatewayService( .returning()); formalApprovalId = approval.id; const approvalIssueId = input.session.issueId; - await writeApprovalChange(db, tx => tx + await writeApprovalChange(db, input.session.companyId, tx => tx .insert(issueApprovals) .values({ companyId: input.session.companyId, diff --git a/server/src/services/work-products.ts b/server/src/services/work-products.ts index e4a8cbb5a10..4aa44ef7f49 100644 --- a/server/src/services/work-products.ts +++ b/server/src/services/work-products.ts @@ -1,3 +1,4 @@ +import { notifyApprovalQueueWork } from "./approval-work-notifications.js"; import { and, desc, eq, inArray } from "drizzle-orm"; import type { Db } from "@paperclipai/db"; import { executionWorkspaces, heartbeatRunEvents, issueWorkProducts, workspaceRuntimeServices } from "@paperclipai/db"; @@ -273,7 +274,10 @@ export function workProductService( createForIssue: async (issueId: string, companyId: string, data: Omit) => { const row = await db.transaction(async (tx) => { - if (data.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`); + if (data.type === "pull_request") { + await notifyTerminalWorkspaceWork(tx, `issue:${issueId}`); + await notifyApprovalQueueWork(tx, companyId); + } await validateExecutionWorkspace(tx, companyId, data.executionWorkspaceId); if (data.isPrimary) { await tx @@ -309,7 +313,10 @@ export function workProductService( .then((rows) => rows[0] ?? null); if (!existing) return null; - if (existing.type === "pull_request" || patch.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); + if (existing.type === "pull_request" || patch.type === "pull_request") { + await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); + await notifyApprovalQueueWork(tx, existing.companyId); + } await validateExecutionWorkspace(tx, existing.companyId, patch.executionWorkspaceId); @@ -378,6 +385,9 @@ export function workProductService( // prefix behind (which a retry would then duplicate). Mirrors the // per-writer transaction the batched issue/document writers use. await db.transaction(async (tx) => { + for (const companyId of new Set(values.filter(value => value.type === "pull_request").map(value => value.companyId))) { + await notifyApprovalQueueWork(tx, companyId); + } for (const value of values) { if (value.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${value.issueId}`); } @@ -387,9 +397,12 @@ export function workProductService( remove: async (id: string) => { const row = await db.transaction(async tx => { - const [existing] = await tx.select({ type: issueWorkProducts.type, issueId: issueWorkProducts.issueId }) + const [existing] = await tx.select({ type: issueWorkProducts.type, issueId: issueWorkProducts.issueId, companyId: issueWorkProducts.companyId }) .from(issueWorkProducts).where(eq(issueWorkProducts.id, id)); - if (existing?.type === "pull_request") await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); + if (existing?.type === "pull_request") { + await notifyTerminalWorkspaceWork(tx, `issue:${existing.issueId}`); + await notifyApprovalQueueWork(tx, existing.companyId); + } return tx.delete(issueWorkProducts).where(eq(issueWorkProducts.id, id)) .returning().then(rows => rows[0] ?? null); }); From 0d22e27dc4a2e6e1fb75a6d8902e895b3c773284 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:53:02 -0500 Subject: [PATCH 36/41] fix(runner): preserve closed native startup diagnostics (#17962) ## Thinking Path > - Paperclip runs agents through a controlled execution boundary. > - Native Runner starts ACPX providers through a TypeScript sidecar and a Rust transport. > - That transport hides arbitrary provider messages because they can contain credentials. > - Grok admission failures currently have no stable code. Other startup failures can also lose their owning stage. > - A real native startup failure therefore reports only `classification=unclassified`, even when the ordinary subscription hello test succeeds. > - This pull request preserves fixed failure categories without forwarding provider text or changing admission policy. > - The benefit is a useful next diagnostic result before another functional repair or retry. ## Linked Issues or Issue Description Refs #17922 and #17942. **What happened?** A native Grok session failed before its first tool. Runner assets passed verification. The Rust rejection discarded the cause because only closed error codes may cross that boundary. The existing admission guard used plain errors. Startup errors outside that guard could also lose their stage. **Expected behavior** Report a fixed Grok admission reason, or a fixed startup stage when no known error code exists. Keep credentials, provider messages, session IDs, and paths out of the classification. Keep the failure and cleanup effective. **Steps to reproduce** 1. Have a Grok ACP fixture return a session without its native effort acknowledgement. 2. Start it through the real ACPX client and pass the failure through the sidecar classifier. 3. Before this change, the result is `acpx_sidecar_command_failed`. Rust reduces that to `unclassified`. 4. The same loss occurs for injected sandbox, credential, and runtime admission failures. ## What Changed - Give Grok effort admission errors fixed codes for missing acknowledgement, model or effort mismatch, session mismatch, conflicting configuration, drift, and invalid protocol use. - Record closed host startup stages in a private weak map. Preserve the original error object, cancellation, and resource cleanup. - Prefer existing known error codes over a fallback stage. Ignore arbitrary provider-supplied stage properties and messages. - Map only exact new codes to Rust diagnostic categories. Add real ACPX-client and sidecar-pipe regressions with secret canaries and unknown-code controls. - Bind the changed shared source and new diagnostic modules in Copilot profile 70. Preserve profile 69 and earlier history. - Document the diagnostic contract. There are no dependency, lockfile, workflow, auth, model, permission, or retry-policy changes. ## Verification - Red before the production change: four TypeScript cases lost the native reason or host stage. A Rust case returned `unclassified` for the new closed categories. - Final focused checks: 320 tests passed: 241 across sidecar classification, Grok admission, the real ACPX client, recovery, runtime adapter, and Copilot installation; all 79 host integration tests also passed in isolation. These include nine startup-stage cases, translated errors, actual missing Pi skill materialization, missing Copilot/Cursor credentials, and a missing workspace binding. - Review follow-up: a translated Copilot rejection lost its stage, and a real Pi skill-loading failure reported configuration. Both regressions failed before the two-line repair and pass afterward. Stage recording now follows provider translation and starts before Pi skill acquisition. The current unpublished Copilot 70 identity was regenerated. - Formal review follow-up: missing Copilot and Cursor credentials reported configuration. Both cases failed before the two-line stage repair and pass afterward. A missing-workspace case also verifies the separate binding catch. - Seven Rust unit checks and 17 sidecar transport integration checks passed. The pipe checks preserve known categories and reject unknown suffixes without disclosing the provider message. - `pnpm -r typecheck` and `pnpm build` passed. Rust formatting and the generated profile/source identity checks passed. - Host integration history is retained: one complete 73-case host run passed. Other batches hit credential-port ownership errors in different existing cases; the four new stage regressions passed. The final 79-case host run passed after the last review fix. The prior intermittent lease failures remain recorded and are not claimed fixed here. - `pnpm test:run` was attempted and stopped after repeated embedded Postgres bootstrap failures. The same local database failure predates this change. Full CI must pass before merge. - A metadata-only probe with pinned Grok 1.0.13, private HOME, synthetic credentials, a local catalog, and no inference passed fresh Grok 4.7/High admission and cold restore. This is supporting ABI evidence. It does not prove live subscription execution. - Initial CI history is retained: server shard 1 failed its unchanged final rate-limit slot test (two requests allowed instead of one). No server policy code changed here. The review follow-up requires fresh full CI; this is not claimed fixed. - Second CI history is retained: chat shard 2 failed the unchanged Slack session-sync quiet-queue test (three selects during the quiet window). The first rate-limit failure did not recur there. No chat or server policy code changed in this PR. Final head 654bc5aa21f1554d30cd8e53782b04ced8a16652 passed all 61 contexts (56 success, four intentional skips, one informational neutral review). Both final-head reviews are 5/5 with no new findings; the Paperclip app publishes its completed review as an informational neutral check. - Live native acceptance is still pending. After deployment, retry the original bounded task and inspect its actual shell output, exit status, run result, and saved task state. This PR fixes the evidence gap; it does not claim the startup incident is resolved. ## Risks - Shared startup failures can now receive a stage category. The private weak map cannot import provider-defined fields. Known error codes keep priority. - A category could point at the wrong stage. Tests inject failures at the installation, sandbox, credential, and runtime boundaries. Existing cancellation and cleanup tests remain in place. - The Copilot profile identity changes because shared source changes. Historical identities remain decodable and cannot authorize a current launch. - The next live failure may require another functional repair. Admission remains closed until the requested configuration is verified. ## Model Used - OpenAI Codex, based on GPT-6, with reasoning, tool use, and code execution. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks; full-suite environment failures are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/grok-native-runner.md | 5 ++ packages/paperclip-runner/acpx-profiles.json | 4 +- .../runner-core/src/acpx_sidecar_transport.rs | 31 +++++++ .../runner-core/src/bin/fake-acpx-sidecar.rs | 2 +- .../src/generated_acpx_profiles.rs | 2 +- .../tests/acpx_sidecar_transport.rs | 28 ++++++ .../scripts/copilot-profile-identity.mjs | 6 +- .../src/cli/acpx-sidecar-input.test.ts | 31 +++++++ .../src/cli/acpx-sidecar-input.ts | 7 +- .../src/drivers/acpx/admission-diagnostics.ts | 26 ++++++ .../drivers/acpx/copilot-installation.test.ts | 21 ++++- .../src/drivers/acpx/generated-profiles.ts | 4 +- .../src/drivers/acpx/grok-effort.ts | 39 ++++++--- .../acpx/grok-native-session-options.test.ts | 32 +++++++ .../src/drivers/acpx/profile-compatibility.ts | 4 +- .../src/drivers/acpx/runtime-host.test.ts | 67 ++++++++++++++ .../src/drivers/acpx/runtime-host.ts | 16 +++- .../copilot-profile-v70-identity.json | 87 +++++++++++++++++++ 18 files changed, 383 insertions(+), 29 deletions(-) create mode 100644 packages/paperclip-runner/src/drivers/acpx/admission-diagnostics.ts create mode 100644 packages/paperclip-runner/test/fixtures/copilot-profile-v70-identity.json diff --git a/doc/grok-native-runner.md b/doc/grok-native-runner.md index 851c60d23c4..7a8ba74fa65 100644 --- a/doc/grok-native-runner.md +++ b/doc/grok-native-runner.md @@ -26,6 +26,11 @@ acknowledgement and the effective model before a prompt. A lazy restored handle must connect through read-only native session inspection first. Reapplying the bound metadata on restore prevents Grok from resetting effort to its default. Missing configuration, a rejected value, or effort drift stops admission. +Admission failures keep a closed Grok reason code through the sidecar boundary. +Other startup failures retain their closed admission stage when no more specific +known error code is available. These diagnostics contain no native payload, +credential, arbitrary provider message, or workspace path. They do not change +admission or retry policy. Omitting effort retains Grok's provider default. This setting remains outside the model-visible task prompt. Historical execution inputs retain their schemas. diff --git a/packages/paperclip-runner/acpx-profiles.json b/packages/paperclip-runner/acpx-profiles.json index c96818e7445..1df6ae1c2de 100644 --- a/packages/paperclip-runner/acpx-profiles.json +++ b/packages/paperclip-runner/acpx-profiles.json @@ -32,12 +32,12 @@ "requiresProviderPolicy": true }, "copilot": { - "agentProfileVersion": 69, + "agentProfileVersion": 70, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719", + "commandDigest": "sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc", "requiresProviderPolicy": true }, "claude": { diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs index 43edd3e0196..a0b963ad2b6 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_sidecar_transport.rs @@ -893,6 +893,23 @@ fn stderr_diagnostic_categories(value: &str) -> BTreeSet<&'static str> { fn response_error_classification(error: &ResponseError) -> &'static str { match error.code.as_str() { + "GROK_EFFORT_ACKNOWLEDGEMENT_MISSING" => return "grok_effort_acknowledgement_missing", + "GROK_EFFORT_MISMATCH" => return "grok_effort_mismatch", + "GROK_MODEL_MISMATCH" => return "grok_model_mismatch", + "GROK_SESSION_MISMATCH" => return "grok_session_mismatch", + "GROK_CONFIGURATION_CONFLICT" => return "grok_configuration_conflict", + "GROK_CONFIGURATION_DRIFT" => return "grok_configuration_drift", + "GROK_ADMISSION_PROTOCOL_VIOLATION" => return "grok_admission_protocol_violation", + "ACPX_ADMISSION_BINDING" => return "admission_binding", + "ACPX_ADMISSION_INSTALLATION" => return "admission_installation", + "ACPX_ADMISSION_SANDBOX" => return "admission_sandbox", + "ACPX_ADMISSION_CREDENTIAL" => return "admission_credential", + "ACPX_ADMISSION_CONFIGURATION" => return "admission_configuration", + "ACPX_ADMISSION_SKILLS" => return "admission_skills", + "ACPX_ADMISSION_COMMAND" => return "admission_command", + "ACPX_ADMISSION_TOOL_BRIDGE" => return "admission_tool_bridge", + "ACPX_ADMISSION_RUNTIME" => return "admission_runtime", + "ACPX_ADMISSION_IDENTITY" => return "admission_identity", "ACPX_RUNTIME_PACKAGE_MISSING" => return "runtime_package_missing", "ACP_MODEL_UNSUPPORTED" => return "requested_model_unsupported", "AGENT_STARTUP_FAILED" => return "agent_startup_failed", @@ -1120,6 +1137,20 @@ mod tests { message: message.to_owned(), retryable: false, }; + for (code, expected) in [ + ("ACPX_ADMISSION_SANDBOX", "admission_sandbox"), + ("ACPX_ADMISSION_CREDENTIAL", "admission_credential"), + ("ACPX_ADMISSION_RUNTIME", "admission_runtime"), + ( + "GROK_EFFORT_ACKNOWLEDGEMENT_MISSING", + "grok_effort_acknowledgement_missing", + ), + ] { + assert_eq!( + response_error_classification(&error(code, "private-token-canary")), + expected + ); + } assert_eq!( response_error_classification(&error( "acpx_sidecar_command_failed", diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs index 1e9eadd2971..563855b6e9c 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/bin/fake-acpx-sidecar.rs @@ -262,7 +262,7 @@ fn run() -> Result<(), Box> { "id": id, "ok": false, "error": { - "code": "Q7Z9", + "code": request["params"]["errorCode"].as_str().unwrap_or("Q7Z9"), "message": "violet-circuit-4821", "retryable": false, }, diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs index f99ff39b7a0..d5ed6013196 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/generated_acpx_profiles.rs @@ -47,7 +47,7 @@ pub(crate) fn acpx_release_profile(agent: &str) -> Option { agent_runtime_package: None, agent_runtime_version: None, command_digest: - "sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719", + "sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc", requires_provider_policy: true, }, "claude" => AcpxReleaseProfile { diff --git a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs index 08ed56fdba0..62cda02e06e 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/tests/acpx_sidecar_transport.rs @@ -219,6 +219,34 @@ fn keeps_valid_command_rejections_separate_from_protocol_failures() { transport.shutdown().expect("fake sidecar should stop"); } +#[test] +fn preserves_closed_admission_categories_over_the_sidecar_pipe() { + let mut transport = transport("remote-error", Duration::from_secs(1)); + for (code, category) in [ + ("ACPX_ADMISSION_INSTALLATION", "admission_installation"), + ("ACPX_ADMISSION_CREDENTIAL", "admission_credential"), + ( + "GROK_EFFORT_ACKNOWLEDGEMENT_MISSING", + "grok_effort_acknowledgement_missing", + ), + ("GROK_MODEL_MISMATCH", "grok_model_mismatch"), + ("ACPX_ADMISSION_CREDENTIAL_SECRET", "unclassified"), + ("GROK_MODEL_MISMATCH_SECRET", "unclassified"), + ] { + let error = transport + .request( + GeneratedAcpxSidecarCommand::SessionOpen, + json!({"errorCode":code}), + ) + .unwrap_err(); + let message = error.to_string(); + assert!(message.contains(&format!("classification={category}"))); + assert!(!message.contains("violet-circuit-4821")); + assert!(!message.contains("SECRET")); + } + transport.shutdown().unwrap(); +} + #[test] fn redacts_sidecar_stderr_when_the_process_exits() { let mut transport = transport("exit-secret", Duration::from_secs(1)); diff --git a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs index 21737c1085a..400c9615d48 100644 --- a/packages/paperclip-runner/scripts/copilot-profile-identity.mjs +++ b/packages/paperclip-runner/scripts/copilot-profile-identity.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; import { resolve, dirname } from "node:path"; const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); -const fixture = resolve(root, "test/fixtures/copilot-profile-v69-identity.json"); +const fixture = resolve(root, "test/fixtures/copilot-profile-v70-identity.json"); const historical = JSON.parse(readFileSync(resolve(root, "test/fixtures/copilot-profile-v12-identity.json"), "utf8")); -const declaration = { ...historical.declaration, agentProfileVersion: 69 }; +const declaration = { ...historical.declaration, agentProfileVersion: 70 }; delete declaration.permissionIdentitySourceSha256; const sources = { displayEventSourceSha256: "src/drivers/acpx/copilot-events.ts", @@ -44,6 +44,8 @@ const sources = { assetOwnershipSourceSha256: "src/drivers/acpx/provider-assets-root.ts", assetInstallerSourceSha256: "scripts/install-copilot-assets.mjs", credentialBindingSourceSha256: "src/drivers/acpx/environment.ts", + admissionDiagnosticsSourceSha256: "src/drivers/acpx/admission-diagnostics.ts", + grokAdmissionSourceSha256: "src/drivers/acpx/grok-effort.ts", sidecarErrorSourceSha256: "src/cli/acpx-sidecar-input.ts", profileSourceSha256: "src/drivers/acpx/copilot-profile.ts", metadataProbeSourceSha256: "src/drivers/copilot-metadata-probe.ts", diff --git a/packages/paperclip-runner/src/cli/acpx-sidecar-input.test.ts b/packages/paperclip-runner/src/cli/acpx-sidecar-input.test.ts index 0b6a280bdb3..6ebd3259074 100644 --- a/packages/paperclip-runner/src/cli/acpx-sidecar-input.test.ts +++ b/packages/paperclip-runner/src/cli/acpx-sidecar-input.test.ts @@ -1,4 +1,6 @@ import { describe, expect, it } from "vitest"; +import { createGrokEffortAdmission } from "../drivers/acpx/grok-effort.js"; +import { recordAcpxAdmissionFailure } from "../drivers/acpx/admission-diagnostics.js"; import { acpxBootstrapBlockedError, @@ -8,6 +10,35 @@ import { } from "./acpx-sidecar-input.js"; describe("ACPX sidecar input sequencing", () => { + it("prefers a known cause to a stage and ignores provider-supplied stage fields", () => { + const cause = Object.assign(new Error("private-token-canary"), { code: "AUTH_REQUIRED" }); + const error = new AggregateError([cause], "private-wrapper"); + recordAcpxAdmissionFailure(error, "runtime"); + expect(acpxSidecarErrorCode(error)).toBe("AUTH_REQUIRED"); + expect(acpxSidecarErrorCode(Object.assign(new Error("private-token-canary"), { + acpxAdmissionStage: "credential", code: "ACPX_ADMISSION_CREDENTIAL", + }))).toBe("acpx_sidecar_command_failed"); + }); + it("retains a closed fallback stage through aggregate cleanup without changing the original error", () => { + const error = Object.freeze(new Error("private-token-canary")); + expect(recordAcpxAdmissionFailure(error, "credential")).toBe(error); + recordAcpxAdmissionFailure(error, "runtime"); + expect(acpxSidecarErrorCode(new AggregateError([error], "private-wrapper"))).toBe("ACPX_ADMISSION_CREDENTIAL"); + expect(acpxSidecarErrorCode(new Error("ACPX_ADMISSION_CREDENTIAL private-token-canary"))).toBe("acpx_sidecar_command_failed"); + }); + it("preserves a Grok admission rejection without exporting the native payload", () => { + const admission = createGrokEffortAdmission("high", "grok-4.7"); + const guard = admission.createGuard(); + guard("outbound", { id: 1, method: "session/new", params: {} }); + let failure: Error | undefined; + try { + guard("inbound", { id: 1, result: { sessionId: "private-session", _meta: { privateToken: "private-token-canary" } } }); + } catch (error) { failure = error as Error; } + expect(failure).toBeInstanceOf(Error); + expect(acpxSidecarErrorCode(failure!)).toBe("GROK_EFFORT_ACKNOWLEDGEMENT_MISSING"); + expect(failure!.message).not.toContain("private-token-canary"); + expect(() => admission.assertReady()).toThrow(failure); + }); it("classifies a retired tool callback without copying provider text into its identity", () => { const error = Object.assign(new Error("private-token-canary"), { code: "ACPX_TOOL_CALL_STALE" }); expect(acpxSidecarErrorCode(error)).toBe("ACPX_TOOL_CALL_STALE"); diff --git a/packages/paperclip-runner/src/cli/acpx-sidecar-input.ts b/packages/paperclip-runner/src/cli/acpx-sidecar-input.ts index e9f4c645028..5d544314fdd 100644 --- a/packages/paperclip-runner/src/cli/acpx-sidecar-input.ts +++ b/packages/paperclip-runner/src/cli/acpx-sidecar-input.ts @@ -1,4 +1,6 @@ +import { GROK_ADMISSION_ERROR_CODES } from "../drivers/acpx/grok-effort.js"; import { enqueueSerialInput } from "./serial-input-queue.js"; +import { acpxAdmissionFailureCode } from "../drivers/acpx/admission-diagnostics.js"; const ACPX_BOOTSTRAP_COMMANDS = new Set(["initialize", "session.open"]); @@ -38,10 +40,12 @@ export function acpxBootstrapBlockedError( export function acpxSidecarErrorCode(error: Error): string { const pending: Error[] = [error]; const observed = new Set(); + let admissionCode: string | undefined; while (pending.length > 0 && observed.size < 16) { const current = pending.shift()!; if (observed.has(current)) continue; observed.add(current); + admissionCode ??= acpxAdmissionFailureCode(current); const code = directAcpxSidecarErrorCode(current); if (code !== null) return code; @@ -53,7 +57,7 @@ export function acpxSidecarErrorCode(error: Error): string { } if (details.cause instanceof Error) pending.push(details.cause); } - return "acpx_sidecar_command_failed"; + return admissionCode ?? "acpx_sidecar_command_failed"; } function directAcpxSidecarErrorCode(error: Error): string | null { @@ -131,6 +135,7 @@ const GENERIC_ACPX_OUTPUT_CODES = new Set([ ]); const STABLE_ACPX_SIDECAR_CODES = new Set([ + ...GROK_ADMISSION_ERROR_CODES, "ACPX_RUNTIME_PACKAGE_MISSING", "COPILOT_POLICY_VIOLATION", "COPILOT_DETACHED_WORK_UNSUPPORTED", diff --git a/packages/paperclip-runner/src/drivers/acpx/admission-diagnostics.ts b/packages/paperclip-runner/src/drivers/acpx/admission-diagnostics.ts new file mode 100644 index 00000000000..c350ccbbaf4 --- /dev/null +++ b/packages/paperclip-runner/src/drivers/acpx/admission-diagnostics.ts @@ -0,0 +1,26 @@ +/** Closed startup stages, independent of provider output and credentials. */ +export const ACPX_ADMISSION_CODES = { + binding: "ACPX_ADMISSION_BINDING", + installation: "ACPX_ADMISSION_INSTALLATION", + sandbox: "ACPX_ADMISSION_SANDBOX", + credential: "ACPX_ADMISSION_CREDENTIAL", + configuration: "ACPX_ADMISSION_CONFIGURATION", + skills: "ACPX_ADMISSION_SKILLS", + command: "ACPX_ADMISSION_COMMAND", + tool_bridge: "ACPX_ADMISSION_TOOL_BRIDGE", + runtime: "ACPX_ADMISSION_RUNTIME", + identity: "ACPX_ADMISSION_IDENTITY", +} as const; + +const failures = new WeakMap(); + +/** Preserve the original error and cleanup semantics, including cancellation. */ +export function recordAcpxAdmissionFailure(error: unknown, stage: keyof typeof ACPX_ADMISSION_CODES): unknown { + if (error instanceof Error && !failures.has(error)) failures.set(error, stage); + return error; +} + +export function acpxAdmissionFailureCode(error: Error): string | undefined { + const stage = failures.get(error); + return stage === undefined ? undefined : ACPX_ADMISSION_CODES[stage]; +} diff --git a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts index 4436f17831c..0a9c647d407 100644 --- a/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/copilot-installation.test.ts @@ -87,9 +87,26 @@ vi.mock("./installation-integrity.js", () => ({ verifyNativeAcpxInstallation: vi describe("Copilot build-owned installation", () => { it("admits the current profile with environment launch preparation", () => { - expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(69); + expect(QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion).toBe(70); expect(QUALIFIED_ACPX_PROFILES.copilot.qualificationStatus).toBeUndefined(); - expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719"); + expect(QUALIFIED_ACPX_PROFILES.copilot.commandDigest).toBe("sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc"); + }); + + it("preserves v69 while binding closed startup diagnostics in v70", async () => { + const bytes = readFileSync(new URL("../../../test/fixtures/copilot-profile-v69-identity.json", import.meta.url)); + expect(createHash("sha256").update(bytes).digest("hex")).toBe("a878caa5f40cfad517e852384507b50c1098e219f1dd6822335b1119b03eaea5"); + const prior = JSON.parse(bytes.toString("utf8")); + const current = JSON.parse(readFileSync(new URL("../../../test/fixtures/copilot-profile-v70-identity.json", import.meta.url), "utf8")); + expect(Object.keys(current.declaration).filter(key => current.declaration[key] !== prior.declaration[key]).sort()).toEqual([ + "admissionDiagnosticsSourceSha256", "agentProfileVersion", "grokAdmissionSourceSha256", + "profileCompatibilitySourceSha256", "runtimeHostSourceSha256", "sidecarErrorSourceSha256", + ]); + for (const [name, field] of [["admission-diagnostics.ts", "admissionDiagnosticsSourceSha256"], ["grok-effort.ts", "grokAdmissionSourceSha256"]]) { + expect(current.declaration[field]).toBe(createHash("sha256").update(readFileSync(new URL(name!, import.meta.url))).digest("hex")); + } + expect(isSupportedAcpxProfileVersion("copilot", 69)).toBe(true); + await expect(verifyCopilotInstallation({ ...QUALIFIED_ACPX_PROFILES.copilot, + agentProfileVersion: 69, commandDigest: prior.commandDigest })).rejects.toThrow("exact pinned"); }); it("preserves published and candidate history while binding native Grok admission in v69", async () => { diff --git a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts index 9fcf232a819..b42655d2f28 100644 --- a/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts +++ b/packages/paperclip-runner/src/drivers/acpx/generated-profiles.ts @@ -48,12 +48,12 @@ export const QUALIFIED_ACPX_PROFILE_DATA = { "protocolVersion": 1, "acpxVersion": "0.13.1", "agent": "copilot", - "agentProfileVersion": 69, + "agentProfileVersion": 70, "agentServerPackage": "@github/copilot", "agentServerVersion": "1.0.88", "agentRuntimePackage": null, "agentRuntimeVersion": null, - "commandDigest": "sha256:5444547ca83543bc899ae5a8a2e895c19a86559339e60a1970786ce900742719", + "commandDigest": "sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc", "permissionPolicy": "interactive" }, "claude": { diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts b/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts index 8dd25d3f123..f8f1bea2c65 100644 --- a/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts +++ b/packages/paperclip-runner/src/drivers/acpx/grok-effort.ts @@ -1,3 +1,13 @@ +export const GROK_ADMISSION_ERROR_CODES = [ + "GROK_EFFORT_ACKNOWLEDGEMENT_MISSING", + "GROK_EFFORT_MISMATCH", + "GROK_MODEL_MISMATCH", + "GROK_SESSION_MISMATCH", + "GROK_CONFIGURATION_CONFLICT", + "GROK_CONFIGURATION_DRIFT", + "GROK_ADMISSION_PROTOCOL_VIOLATION", +] as const; + export type GrokEffort = "minimal" | "low" | "medium" | "high" | "xhigh"; export function resolveGrokEffort(agent: string, value: unknown): GrokEffort | undefined { if (value === undefined || value === "") return undefined; @@ -8,22 +18,23 @@ export function resolveGrokEffort(agent: string, value: unknown): GrokEffort | u export function createGrokEffortAdmission(expected: GrokEffort, expectedModel: string) { type State = { sessionId: string | null; ready: boolean; error: Error | null }; let current: State | null = null; - const failure = (detail: string) => new Error(`Grok effort admission failed: ${detail}`); + const failure = (code: typeof GROK_ADMISSION_ERROR_CODES[number], detail: string) => + Object.assign(new Error(`Grok effort admission failed: ${detail}`), { code }); const assertReady = () => { if (current?.error) throw current.error; - if (!current?.sessionId || !current.ready) throw failure("native session does not acknowledge the selected effort"); + if (!current?.sessionId || !current.ready) throw failure("GROK_EFFORT_ACKNOWLEDGEMENT_MISSING", "native session does not acknowledge the selected effort"); }; const readAcknowledgement = (result: Record) => { const config = object(object(result._meta)["x.ai/sessionConfig"]); const options = Array.isArray(config.options) ? config.options.map(object) : []; const selected = options.filter(option => option.category === "mode" && option.selected === true); - if (selected.length !== 1) throw failure("native effort selection is missing or ambiguous"); - if (selected[0]!.id !== expected) throw failure("native effort differs from the selected effort"); + if (selected.length !== 1) throw failure("GROK_EFFORT_ACKNOWLEDGEMENT_MISSING", "native effort selection is missing or ambiguous"); + if (selected[0]!.id !== expected) throw failure("GROK_EFFORT_MISMATCH", "native effort differs from the selected effort"); const model = object(result.models).currentModelId; const selectedModels = options.filter(option => option.category === "model" && option.selected === true); if (typeof model !== "string" || !model.trim() || model !== expectedModel || selectedModels.length > 1 || selectedModels.length === 1 && selectedModels[0]!.id !== model) { - throw failure("native model acknowledgement conflicts with the selected model"); + throw failure("GROK_MODEL_MISMATCH", "native model acknowledgement conflicts with the selected model"); } }; return { @@ -34,26 +45,26 @@ export function createGrokEffortAdmission(expected: GrokEffort, expectedModel: s current = state; let pending: { id: string | number; method: string; sessionId: unknown } | null = null; return (direction: "inbound" | "outbound", value: unknown): void => { - if (current !== state) throw failure("connection authority was replaced"); + if (current !== state) throw failure("GROK_ADMISSION_PROTOCOL_VIOLATION", "connection authority was replaced"); if (state.error) throw state.error; const message = object(value), params = object(message.params); try { if (direction === "outbound") { if (message.method === "session/prompt") { assertReady(); - if (params.sessionId !== state.sessionId) throw failure("prompt belongs to a different session"); + if (params.sessionId !== state.sessionId) throw failure("GROK_SESSION_MISMATCH", "prompt belongs to a different session"); } else if (message.method === "session/new" || message.method === "session/load" || message.method === "session/resume") { - if (typeof message.id !== "string" && typeof message.id !== "number" || pending) throw failure("uncorrelated or overlapping session admission"); + if (typeof message.id !== "string" && typeof message.id !== "number" || pending) throw failure("GROK_ADMISSION_PROTOCOL_VIOLATION", "uncorrelated or overlapping session admission"); const meta = object(params._meta); if (meta.reasoningEffort !== undefined && meta.reasoningEffort !== expected || meta.modelId !== undefined && meta.modelId !== expectedModel) { - throw failure("native session request conflicts with the bound configuration"); + throw failure("GROK_CONFIGURATION_CONFLICT", "native session request conflicts with the bound configuration"); } params._meta = { ...meta, modelId: expectedModel, reasoningEffort: expected }; state.ready = false; state.sessionId = null; pending = { id: message.id, method: message.method, sessionId: params.sessionId }; } else if (message.method === "session/set_config_option" || message.method === "session/set_model") { - throw failure("model and effort must be selected when the native session opens"); + throw failure("GROK_ADMISSION_PROTOCOL_VIOLATION", "model and effort must be selected when the native session opens"); } return; } @@ -61,7 +72,7 @@ export function createGrokEffortAdmission(expected: GrokEffort, expectedModel: s if (!state.ready) return; const update = object(params.update); if (params.sessionId !== state.sessionId || update.reasoning_effort !== expected - || update.model_id !== expectedModel) throw failure("native model or effort drifted from the selected configuration"); + || update.model_id !== expectedModel) throw failure("GROK_CONFIGURATION_DRIFT", "native model or effort drifted from the selected configuration"); return; } if (message.method !== undefined || !pending || message.id !== pending.id) return; @@ -69,14 +80,14 @@ export function createGrokEffortAdmission(expected: GrokEffort, expectedModel: s if (message.error !== undefined) return; const result = object(message.result); const sessionId = request.method === "session/new" ? result.sessionId : request.sessionId; - if (typeof sessionId !== "string" || !sessionId.trim()) throw failure("native acknowledgement has no session identity"); + if (typeof sessionId !== "string" || !sessionId.trim()) throw failure("GROK_SESSION_MISMATCH", "native acknowledgement has no session identity"); const detail = object(object(result._meta)["x.ai/sessionDetail"]); if (result.sessionId !== undefined && result.sessionId !== sessionId - || detail.sessionId !== undefined && detail.sessionId !== sessionId) throw failure("native acknowledgement belongs to a different session"); + || detail.sessionId !== undefined && detail.sessionId !== sessionId) throw failure("GROK_SESSION_MISMATCH", "native acknowledgement belongs to a different session"); readAcknowledgement(result); state.sessionId = sessionId; state.ready = true; } catch (error) { - state.error = error instanceof Error ? error : failure("invalid native acknowledgement"); + state.error = error instanceof Error ? error : failure("GROK_ADMISSION_PROTOCOL_VIOLATION", "invalid native acknowledgement"); throw state.error; } }; diff --git a/packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts b/packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts index 6770890cf53..2b2e8798eec 100644 --- a/packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/grok-native-session-options.test.ts @@ -5,6 +5,38 @@ import { join } from "node:path"; import { createAcpRuntime, createAgentRegistry, createRuntimeStore, type AcpRuntimeHandle } from "acpx/runtime"; import { expect, it } from "vitest"; import { createGrokEffortAdmission } from "./grok-effort.js"; +import { acpxSidecarErrorCode } from "../../cli/acpx-sidecar-input.js"; + +it("retains a native Grok acknowledgement failure through the real ACPX connection", async () => { + const root = await mkdtemp(join(tmpdir(), "grok-rejected-admission-")); + const fixture = join(root, "grok.cjs"); + await writeFile(fixture, String.raw` + const readline = require("node:readline"); + readline.createInterface({input:process.stdin}).on("line", line => { + const request = JSON.parse(line); + const result = request.method === "initialize" + ? {protocolVersion:1,agentCapabilities:{},authMethods:[]} + : {sessionId:"private-session",models:{currentModelId:"grok-4.7",availableModels:[{modelId:"grok-4.7",name:"grok-4.7"}]},_meta:{privateToken:"private-token-canary"}}; + process.stdout.write(JSON.stringify({jsonrpc:"2.0",id:request.id,result})+"\n"); + }); + `); + const admission = createGrokEffortAdmission("high", "grok-4.7"); + const children: ReturnType[] = []; + const runtime = createAcpRuntime({ cwd: root, sessionStore: createRuntimeStore({stateDir:join(root,"state")}), + agentRegistry: createAgentRegistry({overrides:{grok:[fixture,"agent","stdio"]}}), permissionMode:"approve-reads", + protocolGuardFactory: () => admission.createGuard(), + spawnAgent: () => { const child = spawn(process.execPath,[fixture],{cwd:root,env:{},stdio:"pipe"}); children.push(child); return child; }, + }); + try { + const failure = await runtime.ensureSession({sessionKey:"native-rejection",agent:"grok",mode:"persistent",sessionOptions:{model:"grok-4.7"}}).catch(error => error); + expect(acpxSidecarErrorCode(failure)).toBe("GROK_EFFORT_ACKNOWLEDGEMENT_MISSING"); + expect(failure.message).not.toContain("private-token-canary"); + expect(() => admission.assertReady()).toThrow("missing or ambiguous"); + } finally { + for (const child of children) if (child.exitCode === null && child.signalCode === null) child.kill("SIGKILL"); + await rm(root,{recursive:true,force:true}); + } +}); it.each(["low", "high"] as const)("transports and restores native Grok %s effort through the real ACPX client", async effort => { const root = await mkdtemp(join(tmpdir(), "grok-native-session-options-")); diff --git a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts index 6f9990c550f..a11d0c684ff 100644 --- a/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts +++ b/packages/paperclip-runner/src/drivers/acpx/profile-compatibility.ts @@ -2,13 +2,13 @@ import { QUALIFIED_ACPX_PROFILE_DATA } from "./generated-profiles.js"; /** Decodable historical profile revisions, not permission to launch them. * Exact current profile/package/digest admission is checked separately. */ -type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68; +type HistoricalAcpxProfileVersion = 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 | 65 | 66 | 67 | 68 | 69; type CurrentAcpxProfileVersion = typeof QUALIFIED_ACPX_PROFILE_DATA[keyof typeof QUALIFIED_ACPX_PROFILE_DATA]["agentProfileVersion"]; export type AcpxProfileVersion = HistoricalAcpxProfileVersion | CurrentAcpxProfileVersion; const historicalVersions: Readonly> = { pi: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29], claude: [1, 2, 3, 4, 5], codex: [1, 2, 3, 4, 5], - grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68], + grok: [1, 2, 3, 4, 5], copilot: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69], cursor: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], }; diff --git a/packages/paperclip-runner/src/drivers/acpx/runtime-host.test.ts b/packages/paperclip-runner/src/drivers/acpx/runtime-host.test.ts index f97924870e9..8d14bb2f9da 100644 --- a/packages/paperclip-runner/src/drivers/acpx/runtime-host.test.ts +++ b/packages/paperclip-runner/src/drivers/acpx/runtime-host.test.ts @@ -21,6 +21,7 @@ import { } from "../../contracts/runtime-context.js"; import { releaseMaterializedNativeRuntimeSkills } from "../runtime-context-materializer.js"; import { openCodexAcpxRuntime } from "./codex-runtime-adapter.js"; +import { acpxSidecarErrorCode } from "../../cli/acpx-sidecar-input.js"; import { stageManagedCodexCredential } from "./codex-credentials.js"; import type { VerifiedAcpxCommandLease, @@ -202,6 +203,72 @@ afterEach(async () => { }); describe("ACPX runtime host", () => { + it.each(["copilot", "cursor"] as const)("reports missing %s credentials as credential admission", async agent => { + const fixture = await hostFixture(); + const openRuntime = vi.fn(async () => runtimePort()); + const failure = await AcpxRuntimeHost.open({ ...fixture.options, agent, model:"gpt-5.6-sol", + permissionMode:"approve-all",environment:{} }, fixture.dependencies({openRuntime})).catch(error => error); + expect(failure).toMatchObject({code:agent === "copilot" ? "COPILOT_AUTH_REQUIRED" : "CURSOR_CREDENTIALS_MISSING"}); + expect(acpxSidecarErrorCode(failure)).toBe("ACPX_ADMISSION_CREDENTIAL"); + expect(openRuntime).not.toHaveBeenCalled(); + expect(fixture.commandClose).not.toHaveBeenCalled(); + }); + it("retains binding admission when the selected workspace is missing", async () => { + const fixture = await hostFixture(); + await rm(fixture.options.workingDirectory,{recursive:true}); + const openRuntime = vi.fn(async () => runtimePort()); + const dependencies = fixture.dependencies({openRuntime}); + const verifyInstallation = vi.fn(dependencies.verifyInstallation!); + const failure = await AcpxRuntimeHost.open({ ...fixture.options,agent:"codex",model:"gpt-5.6-sol", + permissionMode:"approve-all" }, {...dependencies,verifyInstallation}).catch(error => error); + expect(acpxSidecarErrorCode(failure)).toBe("ACPX_ADMISSION_BINDING"); + expect(verifyInstallation).not.toHaveBeenCalled(); + expect(openRuntime).not.toHaveBeenCalled(); + }); + it("retains the admission stage when provider classification replaces the error", async () => { + const fixture = await hostFixture(); + const openRuntime = vi.fn(async () => { throw new Error("Unauthorized private-token-canary"); }); + const failure = await AcpxRuntimeHost.open({ ...fixture.options, agent: "copilot", model: "gpt-5.6-sol", + permissionMode: "approve-all", environment: {COPILOT_GITHUB_TOKEN:"synthetic-test-token"} }, fixture.dependencies({openRuntime})).catch(error => error); + expect(failure).toMatchObject({code:"COPILOT_AUTH_REQUIRED"}); + expect(acpxSidecarErrorCode(failure)).toBe("ACPX_ADMISSION_RUNTIME"); + expect(failure.message).not.toContain("private-token-canary"); + expect(openRuntime).toHaveBeenCalledOnce(); + expect(fixture.commandClose).toHaveBeenCalledOnce(); + }); + it("reports failed Pi skill materialization as skills admission before launch", async () => { + const fixture = await hostFixture(); + const digest = "0".repeat(64); + const bundle = {schema:NATIVE_RUNTIME_ASSET_SCHEMA,digest,manifestDigest:digest, + rootPath:join(fixture.root,"missing-assigned-skill"),fileCount:1,totalBytes:1}; + const snapshot = {prompt:{revision:PAPERCLIP_EXECUTION_PROMPT_REVISION,text:PAPERCLIP_EXECUTION_PROMPT,digest:nativeRuntimePromptDigest()}, + instructions:{entryPath:"AGENTS.md",bundle},skills:[{key:"assigned",runtimeName:"assigned",versionId:"v1",bundle}], + mcp:{assignmentSetId:"none",digest,bindingId:null}}; + const context = {...snapshot,aggregateDigest:canonicalNativeRuntimeContextDigest(snapshot)}; + const openRuntime = vi.fn(async () => runtimePort()); + const failure = await AcpxRuntimeHost.open({ ...fixture.options, agent:"pi",model:"anthropic/claude-sonnet-5", + piThinkingLevel:"low",permissionMode:"approve-all",providerPolicy:{readOnly:false},runtimeContext:context },fixture.dependencies({openRuntime})).catch(error => error); + expect(failure).toBeInstanceOf(Error); + expect(failure.message).toContain("missing-assigned-skill"); + expect(acpxSidecarErrorCode(failure)).toBe("ACPX_ADMISSION_SKILLS"); + expect(openRuntime).not.toHaveBeenCalled(); + expect(fixture.commandClose).not.toHaveBeenCalled(); + }); + it.each(["installation", "sandbox", "credential", "runtime"] as const)("retains the failed %s admission stage through the sidecar boundary", async stage => { + const fixture = await hostFixture(); + const failure = new Error("private-token-canary"); + const openRuntime = vi.fn(async () => { throw failure; }); + const dependencies = fixture.dependencies({ openRuntime }); + if (stage === "installation") dependencies.verifyInstallation = async () => { throw failure; }; + if (stage === "sandbox") dependencies.prepareSandbox = async () => { throw failure; }; + if (stage === "credential") dependencies.stageCredential = async () => { throw failure; }; + const rejected = await AcpxRuntimeHost.open({ ...fixture.options, agent: "codex", model: "gpt-5.6-sol", + permissionMode: "approve-all", environment: { OPENAI_API_KEY: "synthetic-test-key" } }, dependencies).catch(error => error); + expect(rejected).toBe(failure); + expect(acpxSidecarErrorCode(rejected)).toBe(`ACPX_ADMISSION_${stage.toUpperCase()}`); + if (stage !== "runtime") expect(openRuntime).not.toHaveBeenCalled(); + if (stage === "runtime") expect(fixture.commandClose).toHaveBeenCalledOnce(); + }); it("rejects preconfigured Codex before admission resources or runtime launch", async () => { const fixture = await hostFixture(); const openRuntime = vi.fn(), prepareSandbox = vi.fn(); diff --git a/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts b/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts index ddba9de9944..38b39f80593 100644 --- a/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts +++ b/packages/paperclip-runner/src/drivers/acpx/runtime-host.ts @@ -1,4 +1,5 @@ import { resolveGrokDisableWebSearch } from "./grok-web-search.js"; +import { recordAcpxAdmissionFailure, type ACPX_ADMISSION_CODES } from "./admission-diagnostics.js"; import { composeCopilotSystemInstructions } from "./copilot-profile.js"; import { resolveClaudeEffort } from "./claude-effort.js"; import { resolveGrokEffort } from "./grok-effort.js"; @@ -384,7 +385,7 @@ export class AcpxRuntimeHost { ? { providerPolicy: options.providerPolicy } : {}), }), dependencies.retainAdmissionCleanup, - ); + ).catch(error => { throw recordAcpxAdmissionFailure(error, "binding"); }); if (options.expectedIdentity) { verifyExpectedAcpxIdentity(options.expectedIdentity, binding, null); } @@ -405,7 +406,7 @@ export class AcpxRuntimeHost { profile, ), dependencies.retainAdmissionCleanup, - ); + ).catch(error => { throw recordAcpxAdmissionFailure(error, "installation"); }); if (installation.commandDigest !== profile.commandDigest) { throw new Error("Verified ACPX installation does not match its profile"); } @@ -451,6 +452,7 @@ export class AcpxRuntimeHost { retainedRejectedRuntimeAdmissions.add(retained); retainRuntimeHostCleanup(ownedCleanup); }; + let admissionStage: keyof typeof ACPX_ADMISSION_CODES = "sandbox"; try { const sandbox = await runAbortableAdmissionStage( options.signal, @@ -473,6 +475,7 @@ export class AcpxRuntimeHost { }), dependencies.retainAdmissionCleanup, ); + admissionStage = "credential"; if (options.agent === "codex" && !preconfigured) { credential = await acquireAbortableAdmissionResource({ signal: options.signal, @@ -508,6 +511,7 @@ export class AcpxRuntimeHost { }); } options.signal?.throwIfAborted(); + admissionStage = "configuration"; const configurationLifetime = credential; await runAbortableAdmissionStage( options.signal, @@ -522,7 +526,9 @@ export class AcpxRuntimeHost { }, ); options.signal?.throwIfAborted(); + admissionStage = "credential"; assertAcpxProfileEnvironment(options.agent, sandbox.launchEnvironment); + admissionStage = "configuration"; let launchEnvironment = sandbox.launchEnvironment; // This copy is registered by the authenticated controller for its company, // agent and run. Environment/config values cannot widen the grant. Each @@ -548,6 +554,7 @@ export class AcpxRuntimeHost { ); options.signal?.throwIfAborted(); } + admissionStage = "skills"; if (options.agent === "pi") { const skills = await acquireAbortableAdmissionResource({ signal: options.signal, @@ -582,6 +589,7 @@ export class AcpxRuntimeHost { ); options.signal?.throwIfAborted(); } + admissionStage = "command"; command = await acquireAbortableAdmissionResource({ signal: options.signal, acquire: () => installation.openCommand({ signal: options.signal }), @@ -595,6 +603,7 @@ export class AcpxRuntimeHost { signal => installation.openCommand({ signal }), ); command = commandOwner.command; + admissionStage = "tool_bridge"; toolBridge = options.semanticTools ? await acquireAbortableAdmissionResource({ signal: options.signal, @@ -613,6 +622,7 @@ export class AcpxRuntimeHost { if (admittedLifetime === null) { throw new Error("ACPX provider lifetime lease is unavailable"); } + admissionStage = "runtime"; runtime = await acquireAbortableAdmissionResource({ signal: options.signal, acquire: async () => { @@ -683,6 +693,7 @@ export class AcpxRuntimeHost { }); }, }); + admissionStage = "identity"; const runtimeIdentity = await runAbortableAdmissionStage( options.signal, () => @@ -734,6 +745,7 @@ export class AcpxRuntimeHost { }); } catch (error) { error = classifyAcpxProfileError(options.agent, error) ?? error; + recordAcpxAdmissionFailure(error, admissionStage); const cleanup = cleanupRuntimeResources( runtime, toolBridge, diff --git a/packages/paperclip-runner/test/fixtures/copilot-profile-v70-identity.json b/packages/paperclip-runner/test/fixtures/copilot-profile-v70-identity.json new file mode 100644 index 00000000000..8908e2f0fb1 --- /dev/null +++ b/packages/paperclip-runner/test/fixtures/copilot-profile-v70-identity.json @@ -0,0 +1,87 @@ +{ + "commandDigest": "sha256:618c3b69bc98b9a7e3c1b3411700eab8b90fd9245ba35d0bc85484319247fdbc", + "declaration": { + "schema": "paperclip.acpx_profile_declaration.v1", + "agent": "copilot", + "agentProfileVersion": 70, + "acpxVersion": "0.13.1", + "agentServerVersion": "1.0.88", + "protocolPolicyRevision": "copilot-agent-manual-v1", + "acpxPatchSha256": "00d6af17216cb0cad902b48633eb19bf0ec6b05837a984cb24661fca593f09c1", + "policySha256": "a4be69be43db84052560420a3ca546a665c9165573c24d543eb33d144ba138a3", + "distributionSourceSha256": "c0a65e247fc47b263bd369a3d5ab505c4b323d4e6493be8a9ecbeb0e9dbb9107", + "agentFilesBinding": "registered-runtime-context-v1", + "systemInstructionDelivery": "COPILOT_HOME/copilot-instructions.md:replace-under-lifetime-lease-before-launch:v1", + "sharedRuntimeContract": "paperclip.acpx-runtime-contract.v1", + "innerDistributionSourceSha256": "9bb08347eed01f1ab44e065c8b06ead9be3a25973d6e6329dc4c0a099096279a", + "upstreamAppSha256": "7b48282a19b5b0814a0c96ad5e3a125173d792cc55f9525b2effea962f6063c0", + "patchedAppSha256": "61f8eebe0c30cc03b311c6818b2acb158a6d16ca58c8bfac5ebd42db095c2679", + "messageIdentityContract": "copilot-native-message-id-v1", + "ownedDistributionDelivery": "COPILOT_CLI_DIST_DIR:lease-owned-guarded-inner-distribution:v1", + "permissionContextContract": "copilot-edit-permission-context-v1", + "permissionContextSourceSha256": "de2d38d702b3c574161af0c0c04790fb6f6347916eb3916ce7a131a8c229121b", + "permissionAdapterSourceSha256": "ee5929ed2ee647ad0369465dee9d51a00b2dd28e1ff735b43e969ddc9c164668", + "permissionLocationsSourceSha256": "f39a81da395b6e23aa2a1a45572a28846e5e81bfd3d764d9a456f74a79c752e9", + "permissionRedactionSourceSha256": "ad6770c69269087951790e2e87e22ac088e3d432769adf17ebc865ed512e6c4f", + "permissionClassifierSourceSha256": "1161eabd82e26133e3b3db3ab567f65d52808b03f176e93497e02c8330349fb4", + "semanticToolReceiptContract": "paperclip.semantic_tool_receipt.v2", + "semanticReceiptSourceSha256": "b2795726b2234a663fec1a6ca11f0551ae69882fe9e82aa39686e952d036e791", + "semanticBridgeSourceSha256": "8764b8c9032fdc7d1415b01e75a9fc57c550e7a5e11da4bed7d9100082872ae5", + "toolEvidenceSourceSha256": "6eef5039afc75bb851eaf9cd82a6f099e4bd398ca2deb59dd66f5b2a81a7a706", + "semanticNormalizedInputContract": "validated-forwarded-input-commit-v1", + "semanticSidecarSourceSha256": "7d6e948b7cff4df52a9afc916db46ace0f42b4b2d3134ba0c21ee0a928857e62", + "semanticDirectDriverSourceSha256": "d683429427dd8a1a5efe50952018ea486f808bdbf0a92fc5946ee267fe2c72ee", + "semanticValidationSourceSha256": "02fbb92681c78c5de5d2f1aae25aabc25a0e5df6b0ff4baccac652192a8da975", + "semanticNormalizationSourceSha256": "8b628b82e8eacc91cdcc6fdca2210ba4cf282f01e9c6f74e95ff271655b5047e", + "semanticCompletionContractSourceSha256": "f49398b682a5623812f5d3c33d6bb9a749008e47cc0ed969072041615f66df59", + "semanticValidatorsSourceSha256": "f22d9caf2035b600abc1a6625ae23ca97f1409d7e29ed5b7264bb8f9784c7dc9", + "semanticSchemaBundleSourceSha256": "4cae579aac836762fdbf92c5447425d8b756e87a5766bc0c7a1885a90541c8d6", + "semanticSidecarProtocolSourceSha256": "26dd0cfbef8b2b401de6b7862a4af7f243fb703e23678a1a5ed8fa68d4784d42", + "semanticSidecarReceiptCommitContract": "correlated-tool-resolve-v1", + "displayEventSourceSha256": "d72f9a6035c247bc6a42c067c05753076c54322c4b58984e93922f3325dc6877", + "displayProjectionSourceSha256": "75952606b26dd93bbc582c1f0a4424487377272ea745fb3a27ceae031fd5dc7e", + "configuredEnvironmentSourceSha256": "5c9008d121979a3a8987e6b8a5752172eafdc4a2b5d725a73f3bbb8fb83be8ff", + "configuredEnvironmentRustSourceSha256": "85c87fae62b8a9377e4f0a4c00302c7519fae4397b1c48253f8f2b7ba7faf994", + "profileActivitySourceSha256": "4fdb6b0f9125fd8372ba6f27945fa5c8a3d3b2ec891df544092e3dc7a75d5282", + "profileExtensionsSourceSha256": "a80ddc95eb4505095bfdbbf85c758199949ab20be9c84b34c77f762eb7180571", + "profileInstallationSourceSha256": "3e61c4cdd4b78e38fcb16eaadae4f1fd65ce3c7127184ea7a80948c8e5d77f88", + "attachmentPolicySourceSha256": "4f7638b7cc64e18f6a4ba33fd46de2b3d49225e32aeab41f5c695b268fa4f8e2", + "singleReadEvidenceSourceSha256": "4069393405089bde613cecec04f5123c426594811da4c668bd96f0e7e283550f", + "nativeSessionRuntimeSourceSha256": "1fe6fa9e6b96d84d70432b85a57c473f833606d4a521563d5022b4d3c41576ce", + "nativeRuntimeContextSourceSha256": "71607f80dbb8dc5cc6c123d0b8f3995d6c5407277695a8c82cbedc90fb8f4842", + "nativeRuntimePromptsSourceSha256": "fff5755dba9b7dce8a5f10ff6ac6d5ed06b92ed079ce1e0a44ac72f2ce7869c1", + "nativeSessionContractSourceSha256": "22a8c96ce41ba4f03e976fc37f0cc7c5d05cee99fa789941da31ee1dab440b90", + "harnessSessionContractSourceSha256": "baba1a210b863518f01e0c875606ed37f9467c433222251438578bd999bc43b5", + "harnessBackendSourceSha256": "dc827e281ae6731545b166ee4d709a4f67f594a556dcbe63c70fe8f1aca26249", + "harnessSessionSourceSha256": "81361733e86b56dd5a66415aefc6f72e69be24acc8d3417370154ebb7fef6e94", + "transportContractSourceSha256": "03fca93d36a89b0cc0917335e293ad097fdf04356d21b307827bb0ea4c13f523", + "nativeBackendSourceSha256": "7495d9898d22dca1ebf6336361d332bafefc4498f6e49f1135025fad846c8586", + "nativeBackendFactorySourceSha256": "7e431c926b76d53c4c08e3a4105353dcd34bd940f6588a1a68f79631f6feeb44", + "nativeInstructionsSourceSha256": "82ac6e5f7275b4ef79fb166c7330efc504e80cab3d7e9081e4e40a8b570bc50b", + "providerLaunchSourceSha256": "c744235b3067ea75973523f78f1e13399337bc155f1775af694e34782d5c9bef", + "launchPreparationStoreSourceSha256": "ff6025568bbc0aa6ce545772d73dfca2d3da226740bd3a7b386b069e76cff296", + "runtimeRequestBridgeSourceSha256": "93a631b8439df1590b0cf442cf84a9b05f004f12bfee90291651ccfc0861effe", + "runtimeRequestFacadeSourceSha256": "d0ed28e0c5ddad34db8f437e704c8a755eb0b47f27e08e44185f66d1d2205295", + "runtimeRequestNotificationSourceSha256": "6196c6175f11c69295dc00fc773201f4fdeca26c500653870253c43b81e65211", + "guardedInstallationSourceSha256": "56a177d3e702914818dca4ca50e061f0d25f988bfc4c0582c36ce23c1696d82f", + "nativeExecutionSourceSha256": "d60b8ea63bad975c0144f0702b2749d7db9a7da50680a6147f20d4d11f2d5297", + "profileCompatibilitySourceSha256": "0832086e776ae28048f78ce87664bd196b73349899168746833d943e399bed76", + "installationSourceSha256": "a1c7c6f103b898216973e77ffbac0ef7572e250aa574d1a4065270e9dfa0dc2d", + "assetOwnershipSourceSha256": "1785e3f31e5dad5a13b53e25d8cf9482a9bd89a27b6c127830aed6b81acc605c", + "assetInstallerSourceSha256": "d3bd9514e5920d0cc3567aa50487fb534c57a6f66b8da0d3f22c5f6e499c841f", + "credentialBindingSourceSha256": "7586d44b47c83395498dc8806287cd3538ef58177ba7c5949f5766ed105a0161", + "admissionDiagnosticsSourceSha256": "df8f1aa4f2d1bc63b2635bdd09cb8165c8b7681ddcc2618435c7c08ca90475af", + "grokAdmissionSourceSha256": "f1e46cf2e1e36b7704bd797b678e3db8db2003f97bdb9d827cfa3d7662698777", + "sidecarErrorSourceSha256": "17948ad74350dc487ab20d24a0fb89daf0b426b98396454fff0e3856e922d82c", + "profileSourceSha256": "f5538b3dbefa7634a332b87d5ededcefb56aa7e5790cede1c7a0d948a548a456", + "metadataProbeSourceSha256": "be4c8416edd27061109bc1f33d0c26071b4facbb3f651beaae6b3c0275cce727", + "permissionToolIdentitySourceSha256": "770b47305e571a81f344883057d0e4a482a1cd59ac77d045836593a8e739deb1", + "semanticContinuationSourceSha256": "c9cc0fd516b3df92a6655f56ee9bef37615a889c371e2e137d8ac5c1176de811", + "runtimeHostSourceSha256": "c779acab9ae4bbab4a13002d2e12c8c1acb208d0c4b575e7c5697fc3e7bb44c3", + "runtimeAdapterSourceSha256": "7c47c05306ebe7d1ca6b5d046a96f5726a48adec4d27e7c16788f505f25b5db2", + "runtimeSandboxSourceSha256": "981449a4c6d10f682ecf86d834eed07faca30a7d9b419fb8db4973a442a343ec", + "cancellationSourceSha256": "a07d1a9e982dec0ae130c1ad1efa5bc75c6a39e013b730ba3ee45256f09ceeca", + "agentFilesSourceSha256": "fb32f6badc13dfa22f231479509b631b2f0eba0c99b1ab9f3cd5c0c601aa03ba", + "nativeDistributionSourceSha256": "aed327c389e957483a503cb1bb8b247ee7cd7ff4b528fb4567c39bcb4577690b" + } +} From a8bd012635dc0a7955ec45aae7479d0c1369b8ba Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:55:13 -0500 Subject: [PATCH 37/41] fix(evals): apply reviewed lock corrections to reports and qualification images (#17937) https://entire.com/gh/paperclipai/-/trails/98/changes/paperclip/98 ## Thinking Path > - Paperclip manages agents and records their work. > - Product E2E campaigns measure real agent behavior and model spend. > - A paid hosted run completed both cases and all ten model receipts. > - Its report job could not install dependencies from the committed lock. > - The trusted checkout needs a reviewed dependency correction. > - This change applies a reviewed lock correction before the frozen report install. ## Linked Issues or Issue Description Refs: #17928 **Description** The report in [run 40576181788](https://github.com/paperclipai/paperclip/actions/runs/40576181788) failed with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`. Both paid cases had passed. The worker and build jobs already restored the resolved source lock, but the report did not. **Steps to reproduce** Dispatch the bounded daily campaign on the default branch with a stale committed lock configuration. Workers use the resolved lock. The report uses the stale lock and fails before report generation. **Expected behavior** The report installs frozen dependencies from a verified lock for the exact trusted source. Evidence can be reported without repeating paid work. ## What Changed - Apply a reviewed lock correction from the trusted report checkout before its frozen install. Use the same correction for the qualification image build, which also requires a compatible frozen lock. - Pin the original lock, correction and output SHA-256. The trusted authorization job rejects unknown inputs before any paid work. - Apply the same correction to trusted manual history publication. Qualification image builds apply it only when their target equals the workflow revision; other targets keep their own lock. - Update existing Claude ACP and postgres patch identities and link the declared yaml dependency. Existing package tarball integrity entries stay unchanged; the declared yaml dependency adds exactly yaml@2.9.1 with pinned integrity. - Keep runtime registry resolution and worker target locks out of the credential-bearing report. - Test valid correction, idempotence and changed-lock, changed-patch and wrong-output rejection. ## Verification - Focused workflow and lock-correction tests: 36 passed. - `actionlint .github/workflows/runner-full-stack-e2e.yml` passed. - Original worker evidence: 2/2 passed, ten final model receipts, zero unresolved model requests. - The original report was regenerated from retained evidence with zero provider calls and saved privately with replay provenance. - Broader repository checks run in CI. Prior full Product checks passed for #17928. ## Risks The correction is tied to the reviewed committed lock. Dependency input changes need a reviewed correction update. No new registry version or lifecycle script can enter the privileged report through this fix. The original failed workflow stays failed. No root lockfile is committed. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/workflows/docker-runner-check.yml | 10 + .github/workflows/runner-full-stack-e2e.yml | 18 + tests/runner-e2e/OPERATIONS.md | 16 + tests/runner-e2e/README.md | 14 +- tests/runner-e2e/prepare-report-lock.d.mts | 1 + tests/runner-e2e/prepare-report-lock.mjs | 31 + tests/runner-e2e/prepare-report-lock.test.ts | 40 ++ tests/runner-e2e/report-lock.json | 6 + tests/runner-e2e/report-lock.patch | 612 +++++++++++++++++++ tests/runner-e2e/workflow-security.test.ts | 23 +- tests/runner-e2e/workflow-target.test.ts | 20 +- 11 files changed, 784 insertions(+), 7 deletions(-) create mode 100644 tests/runner-e2e/prepare-report-lock.d.mts create mode 100644 tests/runner-e2e/prepare-report-lock.mjs create mode 100644 tests/runner-e2e/prepare-report-lock.test.ts create mode 100644 tests/runner-e2e/report-lock.json create mode 100644 tests/runner-e2e/report-lock.patch diff --git a/.github/workflows/docker-runner-check.yml b/.github/workflows/docker-runner-check.yml index f9a2e44e408..5dfc132f576 100644 --- a/.github/workflows/docker-runner-check.yml +++ b/.github/workflows/docker-runner-check.yml @@ -164,6 +164,16 @@ jobs: with: node-version: 24 package-manager-cache: false + - name: Apply reviewed qualification lock corrections + if: inputs.release_run_id == '' && (inputs.build_eval_image || inputs.publish_eval_image) + env: + TARGET_SHA: ${{ needs.authorize_manual.outputs.target_sha }} + run: | + set -euo pipefail + if [ "$TARGET_SHA" = "$GITHUB_SHA" ]; then + node tests/runner-e2e/prepare-report-lock.mjs + fi + - name: Resolve source dependencies without lifecycle scripts if: inputs.release_run_id == '' run: | diff --git a/.github/workflows/runner-full-stack-e2e.yml b/.github/workflows/runner-full-stack-e2e.yml index bc285f4b695..c1a402d1a4c 100644 --- a/.github/workflows/runner-full-stack-e2e.yml +++ b/.github/workflows/runner-full-stack-e2e.yml @@ -187,6 +187,18 @@ jobs: echo '::notice title=Paid runner routing::Using a standard GitHub-hosted runner' fi + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + ref: ${{ github.sha }} + persist-credentials: false + + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 + with: + node-version: 24 + + - name: Preflight reviewed report dependencies before paid admission + run: node tests/runner-e2e/prepare-report-lock.mjs + target_lock: name: Resolve target pnpm lockfile needs: authorize @@ -1273,6 +1285,9 @@ jobs: with: version: 9.15.4 + - name: Apply reviewed reporting lock corrections + run: node tests/runner-e2e/prepare-report-lock.mjs + - name: Install pinned reporting dependencies run: pnpm install --frozen-lockfile --ignore-scripts --store-dir "${{ runner.temp }}/eval-report-store" @@ -1500,6 +1515,9 @@ jobs: # Use a job-local store and disable lifecycle scripts throughout install. # OIDC is available to every step, even before the AWS credential exchange. + - name: Apply reviewed publication lock corrections + run: node tests/runner-e2e/prepare-report-lock.mjs + - name: Install pinned publication dependencies run: pnpm install --frozen-lockfile --ignore-scripts --store-dir "${{ runner.temp }}/eval-publisher-store" diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index 53dfa5b570f..9ba8341a49c 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -951,3 +951,19 @@ this eval project. Keep it limited to the trusted harness/provider runtime; never expose it to test prompts or unrelated clients. The project check is not an attempt-isolation guarantee. Indexing verification is a separate retained or scheduled check; the manual launcher alone does not perform it. + +### Hosted report dependencies + +The trusted report job applies a reviewed `report-lock.patch` before its frozen +install. `report-lock.json` pins the input lock, patch and resulting lock hashes. +This updates existing patch identities and the already declared yaml dependency +without registry resolution. The worker target and its lock artifact never +control report dependencies. An unknown committed lock fails before install; +update and review this correction when dependency inputs change. The original +failed run remains failed when retained evidence gets a new presentation. + +The trusted authorization job checks this correction before catalog/build and +paid work. Unknown dependency locks fail before model admission. Manual history +publication applies the same correction before its frozen install. Qualification +image builds apply it only when target and workflow revisions match; a different +target keeps its own lock and does not require these reporting helper files. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index af2ba3758b4..5ef908f975f 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1272,9 +1272,17 @@ intentionally uncommitted workspace patch while keeping every target job on one identical dependency resolution. The shared-build job compiles the selected campaign's TypeScript outputs and native binaries once, then each paid cell verifies and extracts the immutable bundle. Remote native cells similarly reuse -one verified provider pack. Report -sanitization and AWS history publication do not consume the target lockfile; -they explicitly check out and install from the trusted workflow commit. The +one verified provider pack. Report sanitization and AWS history publication +explicitly check out the trusted workflow commit. The report applies the +reviewed, hash-pinned `report-lock.patch` to the committed lock before its +frozen install. It does not resolve registry versions at run time or accept a +worker target lock. The patch changes existing package patch identities and +links the already declared yaml dependency; existing package tarball integrity entries +stay unchanged. The correction adds exactly yaml@2.9.1 with pinned integrity. The +trusted authorization job rejects unknown locks before paid work. Manual history +publication uses the same reviewed correction. Qualification image builds apply +it only to a target equal to the workflow revision; other targets keep their own +lock. Unknown lock contents fail before installation. The workflow definition, runner-group permission, and protected-environment deployment still come from the default branch. Do not select the target branch in GitHub's **Use workflow from** control. diff --git a/tests/runner-e2e/prepare-report-lock.d.mts b/tests/runner-e2e/prepare-report-lock.d.mts new file mode 100644 index 00000000000..a6314304c3f --- /dev/null +++ b/tests/runner-e2e/prepare-report-lock.d.mts @@ -0,0 +1 @@ +export function prepareReportLock(root?: string): Promise<'already-reviewed' | 'reviewed-correction'>; diff --git a/tests/runner-e2e/prepare-report-lock.mjs b/tests/runner-e2e/prepare-report-lock.mjs new file mode 100644 index 00000000000..587a1428592 --- /dev/null +++ b/tests/runner-e2e/prepare-report-lock.mjs @@ -0,0 +1,31 @@ +import { createHash } from 'node:crypto'; +import { execFileSync } from 'node:child_process'; +import { readFile, writeFile, mkdtemp, rm } from 'node:fs/promises'; +import path from 'node:path'; +import { tmpdir } from 'node:os'; +import { pathToFileURL } from 'node:url'; + +const sha256 = value => createHash('sha256').update(value).digest('hex'); +export async function prepareReportLock(root = path.resolve(import.meta.dirname, '../..')) { + const policy = JSON.parse(await readFile(path.join(root, 'tests/runner-e2e/report-lock.json'), 'utf8')); + const lockPath = path.join(root, 'pnpm-lock.yaml'); + const bytes = await readFile(lockPath), current = sha256(bytes); + if (current === policy.outputSha256) return 'already-reviewed'; + if (current !== policy.inputSha256) throw new Error('Reporting lock changed; review the dependency correction before install'); + const patch = await readFile(path.join(root, 'tests/runner-e2e/report-lock.patch')); + if (sha256(patch) !== policy.patchSha256) throw new Error('Reviewed reporting lock patch changed'); + const result = execFileSync('git', ['apply', '--numstat', '-'], { cwd: root, input: patch, encoding: 'utf8' }); + if (!/^\d+\s+\d+\s+pnpm-lock\.yaml\n$/.test(result)) throw new Error('Reporting correction must change only the lockfile'); + const temporary = await mkdtemp(path.join(tmpdir(), 'eval-reviewed-lock-')); + try { + await writeFile(path.join(temporary, 'pnpm-lock.yaml'), bytes); + execFileSync('git', ['apply', '-'], { cwd: temporary, input: patch }); + const corrected = await readFile(path.join(temporary, 'pnpm-lock.yaml')); + if (sha256(corrected) !== policy.outputSha256) throw new Error('Reviewed reporting lock result changed'); + await writeFile(lockPath, corrected); + return 'reviewed-correction'; + } finally { await rm(temporary, { recursive: true, force: true }); } +} +if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) { + console.log(await prepareReportLock()); +} diff --git a/tests/runner-e2e/prepare-report-lock.test.ts b/tests/runner-e2e/prepare-report-lock.test.ts new file mode 100644 index 00000000000..d56c0d263d0 --- /dev/null +++ b/tests/runner-e2e/prepare-report-lock.test.ts @@ -0,0 +1,40 @@ +import { mkdtemp, mkdir, readFile, writeFile, copyFile, rm } from 'node:fs/promises'; +import { createHash } from 'node:crypto'; +import { execFileSync } from 'node:child_process'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { expect, it } from 'vitest'; +import { parse } from 'yaml'; +import { prepareReportLock } from './prepare-report-lock.mjs'; + +it.each(['valid', 'unknown-lock', 'changed-patch', 'wrong-output'])( + 'installs only the reviewed dependency correction: %s', async variant => { + const root = await mkdtemp(path.join(tmpdir(), 'eval-report-lock-test-')); + const here = import.meta.dirname, relative = path.join(root, 'tests/runner-e2e'); + try { + await mkdir(relative, { recursive: true }); + for (const file of ['report-lock.json', 'report-lock.patch']) await copyFile(path.join(here, file), path.join(relative, file)); + const input = execFileSync('git', ['show', 'HEAD:pnpm-lock.yaml'], { cwd: path.resolve(here, '../..') }); + const policy = JSON.parse(await readFile(path.join(relative, 'report-lock.json'), 'utf8')); + expect(createHash('sha256').update(input).digest('hex')).toBe(policy.inputSha256); + const lock = path.join(root, 'pnpm-lock.yaml'); + await writeFile(lock, variant === 'unknown-lock' ? 'unknown\n' : input); + if (variant === 'changed-patch') await writeFile(path.join(relative, 'report-lock.patch'), 'changed'); + if (variant === 'wrong-output') { policy.outputSha256 = '0'.repeat(64); await writeFile(path.join(relative, 'report-lock.json'), JSON.stringify(policy)); } + if (variant === 'valid') { + expect(await prepareReportLock(root)).toBe('reviewed-correction'); + expect(createHash('sha256').update(await readFile(lock)).digest('hex')).toBe(policy.outputSha256); + expect(await prepareReportLock(root)).toBe('already-reviewed'); + const before = parse(input.toString()).packages; + const after = parse(await readFile(lock, 'utf8')).packages; + expect(Object.keys(after).filter(key => !Object.hasOwn(before, key))).toEqual(['yaml@2.9.1']); + for (const [key, value] of Object.entries(before)) expect(after[key]).toEqual(value); + expect(after['yaml@2.9.1'].resolution.integrity).toBe('sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw=='); + + } else { + await expect(prepareReportLock(root)).rejects.toThrow(); + expect(await readFile(lock)).toEqual(variant === 'unknown-lock' ? Buffer.from('unknown\n') : input); + } + } finally { await rm(root, { recursive: true, force: true }); } + }, +); diff --git a/tests/runner-e2e/report-lock.json b/tests/runner-e2e/report-lock.json new file mode 100644 index 00000000000..4dd5d2bdeff --- /dev/null +++ b/tests/runner-e2e/report-lock.json @@ -0,0 +1,6 @@ +{ + "inputSha256": "893496dddc21c31bd812ea106ac30d1515dda305eb1dd183007d89cf5b1bab4d", + "outputSha256": "c7f23021335b60da0cff1dd4c75cf8b948333ef3165d7d1703eb79a589edb798", + "patchSha256": "c7c11e899c12e6a030fe1554a8ceee5f21689a3a4f19956746e852050049cc49", + "basis": "Reviewed corrections to existing patch hashes and the already declared yaml workspace dependency. Existing package tarball integrity entries are unchanged. The only added package is yaml@2.9.1 with its exact integrity pinned by this reviewed correction." +} diff --git a/tests/runner-e2e/report-lock.patch b/tests/runner-e2e/report-lock.patch new file mode 100644 index 00000000000..84a0bcac131 --- /dev/null +++ b/tests/runner-e2e/report-lock.patch @@ -0,0 +1,612 @@ +--- a/pnpm-lock.yaml ++++ b/pnpm-lock.yaml +@@ -16,7 +16,7 @@ + + patchedDependencies: + '@agentclientprotocol/claude-agent-acp@0.73.0': +- hash: riqgguivxjxydoydnzpedjy5au ++ hash: xjlrz3rllh2ztafbdiywpfh2rq + path: patches/@agentclientprotocol__claude-agent-acp@0.73.0.patch + '@agentclientprotocol/codex-acp@1.6.2': + hash: 2iynuypg76nxmydiylgtffhr7e +@@ -49,7 +49,7 @@ + hash: 55uhvnotpqyiy37rn3pqpukhei + path: patches/embedded-postgres@18.1.0-beta.16.patch + postgres@3.4.9: +- hash: sryle6qd5qfmnxvoebal4gashi ++ hash: 6q4cic6qawywsyeo2spgnjamda + path: patches/postgres@3.4.9.patch + + importers: +@@ -73,7 +73,10 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) ++ yaml: ++ specifier: ^2.8.1 ++ version: 2.9.1 + + cli: + dependencies: +@@ -133,7 +136,7 @@ + version: 17.4.2 + drizzle-orm: + specifier: 0.45.2 +- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)) ++ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)) + embedded-postgres: + specifier: ^18.1.0-beta.16 + version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei) +@@ -180,7 +183,7 @@ + dependencies: + '@agentclientprotocol/claude-agent-acp': + specifier: ^0.73.0 +- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5)) ++ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5)) + '@anthropic-ai/sdk': + specifier: 0.129.0 + version: 0.129.0(zod@4.6.5) +@@ -310,7 +313,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/adapters/hermes-gateway: + dependencies: +@@ -326,7 +329,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/adapters/kimi-local: + dependencies: +@@ -405,13 +408,13 @@ + version: link:../shared + drizzle-orm: + specifier: ^0.45.2 +- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)) ++ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)) + embedded-postgres: + specifier: ^18.1.0-beta.16 + version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei) + postgres: + specifier: ^3.4.9 +- version: 3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi) ++ version: 3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda) + devDependencies: + '@types/node': + specifier: ^24.0.0 +@@ -427,7 +430,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/google-sheets-mcp-server: + dependencies: +@@ -449,7 +452,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/kv-demo-mcp-server: + dependencies: +@@ -468,7 +471,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/mcp-server: + dependencies: +@@ -490,7 +493,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/paperclip-eval-kernel: + devDependencies: +@@ -505,7 +508,7 @@ + dependencies: + '@agentclientprotocol/claude-agent-acp': + specifier: 0.73.0 +- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5)) ++ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5)) + '@agentclientprotocol/codex-acp': + specifier: 1.6.2 + version: 1.6.2(patch_hash=2iynuypg76nxmydiylgtffhr7e) +@@ -561,7 +564,7 @@ + version: 19.2.5(@types/react@19.2.18) + '@vitejs/plugin-react': + specifier: ^4.7.0 +- version: 4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)) ++ version: 4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)) + axe-core: + specifier: ^4.12.1 + version: 4.13.0 +@@ -579,10 +582,10 @@ + version: 7.0.2 + vite: + specifier: ^6.1.0 +- version: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15) ++ version: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1) + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/plugins/create-paperclip-plugin: + dependencies: +@@ -629,7 +632,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/plugins/examples/plugin-file-browser-example: + dependencies: +@@ -747,7 +750,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/plugins/plugin-llm-wiki: + devDependencies: +@@ -786,7 +789,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/plugins/plugin-workspace-diff: + dependencies: +@@ -820,7 +823,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/plugins/sdk: + dependencies: +@@ -880,7 +883,7 @@ + version: 7.0.2 + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages/teams-catalog: + devDependencies: +@@ -895,7 +898,7 @@ + version: 0.5.0 + '@agentclientprotocol/claude-agent-acp': + specifier: 0.73.0 +- version: 0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3)) ++ version: 0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3)) + '@agentclientprotocol/codex-acp': + specifier: 1.6.2 + version: 1.6.2(patch_hash=2iynuypg76nxmydiylgtffhr7e) +@@ -1006,7 +1009,7 @@ + version: 2.1.0(@grpc/grpc-js@1.14.5)(nice-grpc-common@2.0.4)(nice-grpc@2.1.17) + '@vercel/connect': + specifier: 2.3.3 +- version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)))) ++ version: 2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)))) + acorn: + specifier: 8.18.0 + version: 8.18.0 +@@ -1024,7 +1027,7 @@ + version: 3.0.1(ajv@8.20.0) + better-auth: + specifier: 1.7.2 +- version: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))) ++ version: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))) + chat: + specifier: 4.39.0 + version: 4.39.0(ai@7.0.130(zod@4.4.3))(zod@4.4.3) +@@ -1045,7 +1048,7 @@ + version: 17.4.2 + drizzle-orm: + specifier: ^0.45.2 +- version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)) ++ version: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)) + embedded-postgres: + specifier: ^18.1.0-beta.16 + version: 18.1.0-beta.16(patch_hash=55uhvnotpqyiy37rn3pqpukhei) +@@ -1167,10 +1170,10 @@ + version: 7.0.2 + vite: + specifier: ^8.2.2 +- version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + ui: + dependencies: +@@ -1321,10 +1324,10 @@ + version: 10.6.0(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8)) + '@storybook/react-vite': + specifier: 10.5.10 +- version: 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + '@tailwindcss/vite': + specifier: ^4.3.3 +- version: 4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + '@types/node': + specifier: ^24.0.0 + version: 24.13.3 +@@ -1336,7 +1339,7 @@ + version: 19.2.5(@types/react@19.2.18) + '@vitejs/plugin-react': + specifier: ^6.1.1 +- version: 6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + storybook: + specifier: 10.6.0 + version: 10.6.0(@types/react@19.2.18)(react@19.2.8) +@@ -1348,10 +1351,10 @@ + version: 7.0.2 + vite: + specifier: ^8.2.2 +- version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ version: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + vitest: + specifier: ^5.0.3 +- version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ version: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + + packages: + +@@ -9256,6 +9259,11 @@ + yallist@4.0.0: + resolution: {integrity: sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==} + ++ yaml@2.9.1: ++ resolution: {integrity: sha512-3NxN8+78OdzbT7C/WjGsyfPAtJaN3FNDsWxv7Y7mcDsT/oOmgW8BpyQQFFBnvZE3j9Y2Sdz1ULFLezL7Eb2yFw==} ++ engines: {node: '>= 14.6'} ++ hasBin: true ++ + yargs-parser@21.1.1: + resolution: {integrity: sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==} + engines: {node: '>=12'} +@@ -9316,7 +9324,7 @@ + + '@adobe/css-tools@4.5.0': {} + +- '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))': ++ '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))': + dependencies: + '@agentclientprotocol/sdk': 1.4.0(zod@4.4.3) + '@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(zod@4.4.3) +@@ -9325,7 +9333,7 @@ + - '@anthropic-ai/sdk' + - '@modelcontextprotocol/sdk' + +- '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=riqgguivxjxydoydnzpedjy5au)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))': ++ '@agentclientprotocol/claude-agent-acp@0.73.0(patch_hash=xjlrz3rllh2ztafbdiywpfh2rq)(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))': + dependencies: + '@agentclientprotocol/sdk': 1.4.0(zod@4.4.3) + '@anthropic-ai/claude-agent-sdk': 0.3.286(@anthropic-ai/sdk@0.129.0(zod@4.6.5))(@modelcontextprotocol/sdk@1.31.0(zod@4.6.5))(zod@4.4.3) +@@ -9907,12 +9915,12 @@ + optionalDependencies: + '@opentelemetry/api': 1.9.1 + +- '@better-auth/drizzle-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))': ++ '@better-auth/drizzle-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))': + dependencies: + '@better-auth/core': 1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3) + '@better-auth/utils': 0.4.2 + optionalDependencies: +- drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)) ++ drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)) + + '@better-auth/kysely-adapter@1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(kysely@0.29.5)': + dependencies: +@@ -10974,11 +10982,11 @@ + '@img/sharp-win32-x64@0.35.4': + optional: true + +- '@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': ++ '@joshwooding/vite-plugin-react-docgen-typescript@0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))': + dependencies: + glob: 13.0.6 + react-docgen-typescript: 2.4.0(typescript@7.0.2) +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + optionalDependencies: + typescript: 7.0.2 + +@@ -13184,25 +13192,25 @@ + transitivePeerDependencies: + - '@types/react-dom' + +- '@storybook/builder-vite@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': +- dependencies: +- '@storybook/csf-plugin': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ '@storybook/builder-vite@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))': ++ dependencies: ++ '@storybook/csf-plugin': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8) + ts-dedent: 2.3.0 +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + transitivePeerDependencies: + - esbuild + - rollup + - webpack + +- '@storybook/csf-plugin@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': ++ '@storybook/csf-plugin@10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))': + dependencies: + storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8) + unplugin: 2.3.11 + optionalDependencies: + esbuild: 0.28.2 + rollup: 4.63.1 +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + + '@storybook/global@5.0.0': {} + +@@ -13228,11 +13236,11 @@ + '@types/react': 19.2.18 + '@types/react-dom': 19.2.5(@types/react@19.2.18) + +- '@storybook/react-vite@10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': +- dependencies: +- '@joshwooding/vite-plugin-react-docgen-typescript': 0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ '@storybook/react-vite@10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(esbuild@0.28.2)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))': ++ dependencies: ++ '@joshwooding/vite-plugin-react-docgen-typescript': 0.7.0(typescript@7.0.2)(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + '@rollup/pluginutils': 5.4.0(rollup@4.63.1) +- '@storybook/builder-vite': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ '@storybook/builder-vite': 10.5.10(esbuild@0.28.2)(rollup@4.63.1)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + '@storybook/react': 10.5.10(@types/react-dom@19.2.5(@types/react@19.2.18))(@types/react@19.2.18)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(storybook@10.6.0(@types/react@19.2.18)(react@19.2.8))(typescript@7.0.2) + empathic: 2.0.1 + magic-string: 0.30.21 +@@ -13242,7 +13250,7 @@ + resolve: 1.22.12 + storybook: 10.6.0(@types/react@19.2.18)(react@19.2.8) + tsconfig-paths: 4.2.0 +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + optionalDependencies: + typescript: 7.0.2 + transitivePeerDependencies: +@@ -13335,12 +13343,12 @@ + postcss-selector-parser: 6.0.10 + tailwindcss: 4.3.3 + +- '@tailwindcss/vite@4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': ++ '@tailwindcss/vite@4.3.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))': + dependencies: + '@tailwindcss/node': 4.3.3 + '@tailwindcss/oxide': 4.3.3 + tailwindcss: 4.3.3 +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + + '@tanstack/query-core@5.102.8': {} + +@@ -13758,14 +13766,14 @@ + dependencies: + execa: 5.1.1 + +- '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))))': ++ '@vercel/connect@2.3.3(@chat-adapter/slack@4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3))(@modelcontextprotocol/sdk@1.31.0(zod@4.4.3))(ai@7.0.130(zod@4.4.3))(better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))))': + dependencies: + '@vercel/oidc': 3.8.9 + optionalDependencies: + '@chat-adapter/slack': 4.39.0(patch_hash=226jj24akljccsfbw7bcxujz4u)(ai@7.0.130(zod@4.4.3))(zod@4.4.3) + '@modelcontextprotocol/sdk': 1.31.0(zod@4.4.3) + ai: 7.0.130(zod@4.4.3) +- better-auth: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))) ++ better-auth: 1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))) + + '@vercel/oidc@3.2.0': {} + +@@ -13775,7 +13783,7 @@ + '@vercel/cli-exec': 1.0.1 + jose: 5.10.0 + +- '@vitejs/plugin-react@4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))': ++ '@vitejs/plugin-react@4.7.0(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))': + dependencies: + '@babel/core': 7.29.7 + '@babel/plugin-transform-react-jsx-self': 7.29.7(@babel/core@7.29.7) +@@ -13783,14 +13791,14 @@ + '@rolldown/pluginutils': 1.0.0-beta.27 + '@types/babel__core': 7.20.5 + react-refresh: 0.17.0 +- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15) ++ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1) + transitivePeerDependencies: + - supports-color + +- '@vitejs/plugin-react@6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': ++ '@vitejs/plugin-react@6.1.1(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))': + dependencies: + '@rolldown/pluginutils': 1.0.1 +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + + '@vitest/expect@3.2.4': + dependencies: +@@ -13800,23 +13808,23 @@ + chai: 5.3.3 + tinyrainbow: 2.0.0 + +- '@vitest/mocker@5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15))': ++ '@vitest/mocker@5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1))': + dependencies: + '@jridgewell/trace-mapping': 0.3.31 + '@vitest/spy': 5.0.3 + estree-walker: 3.0.3 + magic-string: 1.4.3 + optionalDependencies: +- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15) +- +- '@vitest/mocker@5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))': ++ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1) ++ ++ '@vitest/mocker@5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))': + dependencies: + '@jridgewell/trace-mapping': 0.3.31 + '@vitest/spy': 5.0.3 + estree-walker: 3.0.3 + magic-string: 1.4.3 + optionalDependencies: +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + + '@vitest/pretty-format@3.2.4': + dependencies: +@@ -14042,10 +14050,10 @@ + + before-after-hook@4.0.0: {} + +- better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15))): ++ better-auth@1.7.2(@opentelemetry/api@1.9.1)(drizzle-kit@0.31.10)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)))(pg@8.18.0)(react-dom@19.2.8(react@19.2.8))(react@19.2.8)(vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1))): + dependencies: + '@better-auth/core': 1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3) +- '@better-auth/drizzle-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi))) ++ '@better-auth/drizzle-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda))) + '@better-auth/kysely-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2)(kysely@0.29.5) + '@better-auth/memory-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2) + '@better-auth/mongo-adapter': 1.7.2(@better-auth/core@1.7.2(@better-auth/utils@0.4.2)(@better-fetch/fetch@1.3.1)(@opentelemetry/api@1.9.1)(better-call@1.4.0(zod@4.4.3))(jose@6.2.10)(kysely@0.29.5)(nanostores@1.5.3))(@better-auth/utils@0.4.2) +@@ -14063,11 +14071,11 @@ + zod: 4.4.3 + optionalDependencies: + drizzle-kit: 0.31.10 +- drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)) ++ drizzle-orm: 0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)) + pg: 8.18.0 + react: 19.2.8 + react-dom: 19.2.8(react@19.2.8) +- vitest: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ vitest: 5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + transitivePeerDependencies: + - '@cloudflare/workers-types' + - '@opentelemetry/api' +@@ -14654,12 +14662,12 @@ + esbuild: 0.25.12 + tsx: 4.23.15 + +- drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi)): ++ drizzle-orm@0.45.2(@opentelemetry/api@1.9.1)(kysely@0.29.5)(pg@8.18.0)(postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda)): + optionalDependencies: + '@opentelemetry/api': 1.9.1 + kysely: 0.29.5 + pg: 8.18.0 +- postgres: 3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi) ++ postgres: 3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda) + + dunder-proto@1.0.1: + dependencies: +@@ -16542,7 +16550,7 @@ + dependencies: + xtend: 4.0.2 + +- postgres@3.4.9(patch_hash=sryle6qd5qfmnxvoebal4gashi): {} ++ postgres@3.4.9(patch_hash=6q4cic6qawywsyeo2spgnjamda): {} + + powershell-utils@0.1.0: {} + +@@ -17587,7 +17595,7 @@ + '@types/unist': 3.0.3 + vfile-message: 4.0.3 + +- vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15): ++ vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1): + dependencies: + esbuild: 0.25.12 + fdir: 6.5.0(picomatch@4.0.7) +@@ -17601,8 +17609,9 @@ + jiti: 2.7.0 + lightningcss: 1.33.0 + tsx: 4.23.15 +- +- vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15): ++ yaml: 2.9.1 ++ ++ vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1): + dependencies: + lightningcss: 1.33.0 + picomatch: 4.0.7 +@@ -17615,11 +17624,12 @@ + fsevents: 2.3.3 + jiti: 2.7.0 + tsx: 4.23.15 +- +- vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)): ++ yaml: 2.9.1 ++ ++ vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)): + dependencies: + '@types/chai': 5.2.3 +- '@vitest/mocker': 5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)) ++ '@vitest/mocker': 5.0.3(vite@6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1)) + chai: 6.3.0 + es-module-lexer: 2.3.2 + expect-type: 1.4.0 +@@ -17630,7 +17640,7 @@ + tinybench: 6.2.0 + tinyexec: 1.3.1 + tinyglobby: 0.2.17 +- vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15) ++ vite: 6.4.3(@types/node@24.13.3)(jiti@2.7.0)(lightningcss@1.33.0)(tsx@4.23.15)(yaml@2.9.1) + why-is-node-running: 3.2.1 + optionalDependencies: + '@opentelemetry/api': 1.9.1 +@@ -17639,10 +17649,10 @@ + transitivePeerDependencies: + - msw + +- vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)): ++ vitest@5.0.3(@opentelemetry/api@1.9.1)(@types/node@24.13.3)(jsdom@30.0.1(@noble/hashes@2.4.0))(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)): + dependencies: + '@types/chai': 5.2.3 +- '@vitest/mocker': 5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)) ++ '@vitest/mocker': 5.0.3(vite@8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1)) + chai: 6.3.0 + es-module-lexer: 2.3.2 + expect-type: 1.4.0 +@@ -17653,7 +17663,7 @@ + tinybench: 6.2.0 + tinyexec: 1.3.1 + tinyglobby: 0.2.17 +- vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15) ++ vite: 8.2.2(@types/node@24.13.3)(esbuild@0.28.2)(jiti@2.7.0)(tsx@4.23.15)(yaml@2.9.1) + why-is-node-running: 3.2.1 + optionalDependencies: + '@opentelemetry/api': 1.9.1 +@@ -17748,6 +17758,8 @@ + yallist@3.1.1: {} + + yallist@4.0.0: {} ++ ++ yaml@2.9.1: {} + + yargs-parser@21.1.1: {} + diff --git a/tests/runner-e2e/workflow-security.test.ts b/tests/runner-e2e/workflow-security.test.ts index bc740e62bd3..d68df923c6a 100644 --- a/tests/runner-e2e/workflow-security.test.ts +++ b/tests/runner-e2e/workflow-security.test.ts @@ -56,6 +56,10 @@ describe("public repository paid workflow security", () => { expect(manual).toContain('pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile'); expect(manual).toContain('PAPERCLIP_RUNNER_LOCK_SHA256=$lock_sha'); expect(manual).toContain('docker logout ghcr.io'); + expect(manual.indexOf('Apply reviewed qualification lock corrections')).toBeLessThan(manual.indexOf('Resolve source dependencies without lifecycle scripts')); + expect(manual).toContain('node tests/runner-e2e/prepare-report-lock.mjs'); + expect(manual).toContain('if [ "$TARGET_SHA" = "$GITHUB_SHA" ]; then'); + }); it("uses the reviewed master branch for the first-party trusted PR workflow", async () => { @@ -275,7 +279,9 @@ describe("public repository paid workflow security", () => { "target_ref: ${{ steps.target.outputs.ref }}", ); expect(authorizeJob).toContain('echo "ref=refs/heads/$TARGET_BRANCH"'); - expect(authorizeJob).not.toContain("actions/checkout@"); + expect(authorizeJob.match(/actions\/checkout@/gu)).toHaveLength(1); + expect(authorizeJob).toContain("ref: ${{ github.sha }}"); + expect(authorizeJob).not.toContain("ref: ${{ needs.authorize.outputs.target_sha }}"); expect(authorizeJob).not.toContain("pnpm install"); expect(targetLockJob).toContain("name: Resolve target pnpm lockfile"); expect(targetLockJob).toContain("needs: authorize"); @@ -537,8 +543,8 @@ describe("public repository paid workflow security", () => { /ref: \$\{\{ needs\.authorize\.outputs\.target_sha \}\}/g, ), ).toHaveLength(6); - expect(fullStack.match(/ref: \$\{\{ github\.sha \}\}/g)).toHaveLength(2); - expect(fullStack.match(/persist-credentials: false/g)).toHaveLength(8); + expect(fullStack.match(/ref: \$\{\{ github\.sha \}\}/g)).toHaveLength(3); + expect(fullStack.match(/persist-credentials: false/g)).toHaveLength(9); expect(fullStack).not.toContain("ref: ${{ inputs.target_branch }}"); expect(fullStack).toContain( "PAPERCLIP_RUNNER_SOURCE_REVISION=${TARGET_SHA}", @@ -553,11 +559,22 @@ describe("public repository paid workflow security", () => { "ref: ${{ needs.authorize.outputs.target_sha }}", ); expect(reportJob).not.toContain("Download resolved target lockfile"); + expect(reportJob).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs"); + expect(reportJob.indexOf("Apply reviewed reporting lock corrections")).toBeLessThan(reportJob.indexOf("Install pinned reporting dependencies")); + expect(reportJob).not.toContain("needs.target_lock.outputs"); + expect(historyJob).toContain("ref: ${{ github.sha }}"); expect(historyJob).not.toContain( "ref: ${{ needs.authorize.outputs.target_sha }}", ); expect(historyJob).not.toContain("Download resolved target lockfile"); + expect(historyJob).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs"); + const authorization = fullStack.split(" authorize:\n")[1]!.split(" target_lock:\n")[0]!; + expect(authorization).toContain("ref: ${{ github.sha }}"); + expect(authorization).toContain("Preflight reviewed report dependencies before paid admission"); + expect(authorization).toContain("run: node tests/runner-e2e/prepare-report-lock.mjs"); + expect(authorization).not.toContain("environment:"); + expect(fullStack).toContain( "if: always() && !cancelled() && needs.catalog.result == 'success'", ); diff --git a/tests/runner-e2e/workflow-target.test.ts b/tests/runner-e2e/workflow-target.test.ts index 88aa5b6cb5c..8abc9f95e3e 100644 --- a/tests/runner-e2e/workflow-target.test.ts +++ b/tests/runner-e2e/workflow-target.test.ts @@ -1,5 +1,5 @@ import { spawnSync } from "node:child_process"; -import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { mkdtemp, readFile, rm, access } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; import { describe, expect, it } from "vitest"; @@ -47,3 +47,21 @@ describe("paid workflow target authorization", () => { expect(result.output).toContain("ref=refs/heads/development\n"); }); }); + +describe("qualification image dependency correction", () => { + it.each([true, false])("applies the workflow correction only to its own source: %s", async sameSource => { + const workflow = await readFile(new URL("../../.github/workflows/docker-runner-check.yml", import.meta.url), "utf8"); + const step = workflow.split(" - name: Apply reviewed qualification lock corrections\n")[1]!.split("\n - ")[0]!; + const script = step.split(" run: |\n")[1]!.replace(/^ {10}/gm, ""); + const directory = await mkdtemp(path.join(os.tmpdir(), "eval-image-lock-target-")); + try { + const marker = path.join(directory, "called"); + const result = spawnSync("bash", ["-c", `node() { touch "$CALLED"; }\n${script}`], { + cwd: directory, encoding: "utf8", env: { ...process.env, GITHUB_SHA: queuedSha, + TARGET_SHA: sameSource ? queuedSha : advancedSha, CALLED: marker }, + }); + expect(result.status, result.stderr).toBe(0); + expect(await access(marker).then(() => true, () => false)).toBe(sameSource); + } finally { await rm(directory, { recursive: true, force: true }); } + }); +}); From e4b30163bba629914904b068511b45bafa21d93b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:55:18 -0500 Subject: [PATCH 38/41] fix(evals): preserve the complete trusted controller artifact inventory (#17943) https://entire.com/gh/paperclipai/-/trails/103/changes/paperclip/103 ## Thinking Path > - Paperclip manages real agent work. > - Controlled Runner evaluations record model spend and traces. > - A trusted build creates one portable controller with a complete file inventory. > - GitHub artifact upload omitted one hidden dependency metadata file. > - The next job correctly refused the incomplete inventory before any paid worker started. > - This change preserves all inventoried bytes without relaxing verification. ## Linked Issues or Issue Description Refs #17889, #17929. **What happened?** [Run 40585138437](https://github.com/paperclipai/paperclip/actions/runs/40585138437) failed at copied controller verification. Its artifact contains 294 files; its signed inventory expects 295. The only missing file is `server/node_modules/protobufjs/node_modules/@protobufjs/pool/.npmignore`. No extra files exist. Every paid worker was skipped and no model request ran. **Expected behavior** The trusted controller artifact preserves its full inventory, including hidden package metadata. The strict verifier rejects any actual change or missing file. **Steps to reproduce** Build the portable runtime, upload the directory with the action's default hidden-file exclusion, then verify the downloaded artifact against the original receipt. ## What Changed - Enable hidden files only on the fresh trusted controller code upload. - Download that exact upload and run a real artifact-service round-trip test before any private preparation. The test fails if the upload drops hidden files again; it tests the actual transport instead of matching a YAML flag. - Keep the strict byte inventory and trusted source/receipt checks. - Document why this code artifact differs from private attempt evidence, which still uses the protected archive. ## Verification - Exact retained artifact comparison: one missing `.npmignore`, zero extra files. - `actionlint .github/workflows/runner-protocol-live-evals.yml` passed. - Protocol workflow security tests: 14 passed. - The artifact-service integration test is explicit-only locally. The controlled workflow supplies the real downloaded artifact, trusted source and receipt digest. - No product runtime, provider routing or grading code changed. CI runs broader checks. Fresh controlled workflow qualification follows merge. ## Risks Hidden files are allowed only in the code-only output of the provider-credential-free build. Raw private evidence and credentials do not enter that upload. The artifact verifier still requires every file and SHA-256 from the trusted receipt. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository inspection and code execution. The exact serving model ID and context window are not exposed to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .../runner-controller-artifact-transport.test.mjs | 12 ++++++++++++ .github/workflows/runner-protocol-live-evals.yml | 15 +++++++++++++++ .../docs/runner-protocol-live-evals.md | 6 ++++++ tests/runner-e2e/workflow-security.test.ts | 6 ++++++ 4 files changed, 39 insertions(+) create mode 100644 .github/tests/runner-controller-artifact-transport.test.mjs diff --git a/.github/tests/runner-controller-artifact-transport.test.mjs b/.github/tests/runner-controller-artifact-transport.test.mjs new file mode 100644 index 00000000000..2c144cae80a --- /dev/null +++ b/.github/tests/runner-controller-artifact-transport.test.mjs @@ -0,0 +1,12 @@ +import test from 'node:test'; +import { verifyControllerRuntime } from '../../packages/paperclip-runner/scripts/controlled-evals/verify-runtime.mjs'; + +test('the uploaded controller preserves every inventoried runtime file', { + skip: !process.env.PAPERCLIP_CONTROLLER_TRANSPORT_DIRECTORY, +}, async () => { + await verifyControllerRuntime({ + directory: process.env.PAPERCLIP_CONTROLLER_TRANSPORT_DIRECTORY, + sourceRevision: process.env.GITHUB_SHA, + receiptSha256: process.env.PAPERCLIP_CONTROLLER_RECEIPT_SHA256, + }); +}); diff --git a/.github/workflows/runner-protocol-live-evals.yml b/.github/workflows/runner-protocol-live-evals.yml index 3fd5f41d0d2..ba3bd0d14c0 100644 --- a/.github/workflows/runner-protocol-live-evals.yml +++ b/.github/workflows/runner-protocol-live-evals.yml @@ -513,10 +513,25 @@ jobs: uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: runner-protocol-runtime-${{ github.run_id }}-${{ github.run_attempt }} + include-hidden-files: true path: ${{ runner.temp }}/runner-protocol-runtime/ retention-days: 30 if-no-files-found: error + - name: Download controller artifact for a credential-free transport test + if: needs.authorize.outputs.accounting_mode == 'controlled' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8 + with: + artifact-ids: ${{ steps.upload.outputs.artifact-id }} + path: ${{ runner.temp }}/runner-controller-roundtrip + + - name: Verify the real uploaded controller inventory before private preparation + if: needs.authorize.outputs.accounting_mode == 'controlled' + env: + PAPERCLIP_CONTROLLER_TRANSPORT_DIRECTORY: ${{ runner.temp }}/runner-controller-roundtrip + PAPERCLIP_CONTROLLER_RECEIPT_SHA256: ${{ steps.build.outputs.runtime_receipt_sha256 }} + run: node --test .github/tests/runner-controller-artifact-transport.test.mjs + prepare_controller: name: Prepare one shared cost and trace manifest needs: [authorize, catalog, build_runner, build_controller] diff --git a/packages/paperclip-runner/docs/runner-protocol-live-evals.md b/packages/paperclip-runner/docs/runner-protocol-live-evals.md index 478eaf869cc..94801d17540 100644 --- a/packages/paperclip-runner/docs/runner-protocol-live-evals.md +++ b/packages/paperclip-runner/docs/runner-protocol-live-evals.md @@ -952,3 +952,9 @@ For this policy version, `operatingReconciliation` is always `pending`; the immutable controller summary has no operating invoice settlement step. Mixed histories retain that warning when any campaign uses the policy. A shared `accountingReadiness` label appears only when every campaign uses it. + + +The trusted controller artifact preserves hidden dependency metadata as well as +code. Its inventory verifier requires every recorded byte, including package +`.npmignore` files. Only the fresh provider-credential-free controller build uses +this upload setting; private attempts still use the protected archive path. diff --git a/tests/runner-e2e/workflow-security.test.ts b/tests/runner-e2e/workflow-security.test.ts index d68df923c6a..698f7e71575 100644 --- a/tests/runner-e2e/workflow-security.test.ts +++ b/tests/runner-e2e/workflow-security.test.ts @@ -981,6 +981,12 @@ it("keeps dependency resolution and shared caches out of Runner protocol archive } const builder = jobs.find(job => job.startsWith(" build_controller:"))!; expect(builder).not.toMatch(/id-token: write|environment:|secrets\./u); + const upload = builder.split(" - name: Retain the trusted controller code for every worker\n")[1]!.split("\n - ")[0]!; + expect(upload).toContain("include-hidden-files: true"); + expect(upload).toContain("id: upload"); + expect(builder).toContain("artifact-ids: ${{ steps.upload.outputs.artifact-id }}"); + expect(builder).toContain("run: node --test .github/tests/runner-controller-artifact-transport.test.mjs"); + expect(builder).toContain("PAPERCLIP_CONTROLLER_RECEIPT_SHA256: ${{ steps.build.outputs.runtime_receipt_sha256 }}"); const preparer = jobs.find(job => job.startsWith(" prepare_controller:"))!; expect(preparer).not.toMatch(/pnpm install/u); expect(preparer.indexOf("verify-runtime.mjs")).toBeLessThan(preparer.indexOf('hosted.mjs" prepare')); From 38d08847b8d3d5a78bf864c355868d5420b8a2fb Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:57:19 -0500 Subject: [PATCH 39/41] fix(runner): keep OpenRouter upstream keys in the parent proxy (#17949) ## Thinking Path > - Paperclip manages agents and their work. > - Paperclip Runner starts OpenCode with the selected provider and model. > - The native OpenRouter route passed the reusable upstream key into the child environment. > - A persisted-state scan found that key in an OpenCode database WAL. The exact write operation is not known. > - This pull request reuses the existing parent-held provider proxy for that route. > - The child receives a temporary loopback credential instead of the upstream key. ## Linked Issues or Issue Description **What happened?** The native OpenRouter path passed `OPENROUTER_API_KEY` into the OpenCode child. A real Product E2E persisted-state scan detected the key in the child database WAL. The scan does not establish which operation wrote it. **Expected behavior** The runner keeps the reusable upstream key in its own process. OpenCode can call only the selected model through the session proxy. **Steps to reproduce** 1. Start the OpenCode driver with an explicit or ambient synthetic OpenRouter key. 2. Inspect the fake child configuration, environment, and persisted provider values. 3. Send a request through the real loopback proxy and verify the upstream authorization and exact model. **Paperclip version or commit** The regression is present on merged master `813e808af462b6a6f9db38b6791e3bfe8465b51e`. This PR is rebased onto merged master `d40f9ac377`. **Deployment mode** Built from source. The finding came from an AWS Product E2E job. The regression tests use local synthetic servers and a fake child, with no provider calls. Related issue search found #16643 about general inherited environment scope. This change addresses the narrower native OpenRouter path. No duplicate custody PR was found. This PR must remain unmerged until separately approved; it does not change the eval baseline or qualify a new live result. ## What Changed - Reuse the existing parent-held proxy for native `openrouter/` models. - Remove the upstream key from the child environment and retain it in diagnostic redaction. - Keep the selected provider and exact model. Reject alternate-routing fields before the upstream request. Use a session-lived loopback token. - Preserve the effective explicit or ambient proxy bypass list when adding loopback entries. - Add explicit and ambient key regressions, upstream routing checks, and child persistence checks. - Document the boundary and its limits. ## Verification - Final rebased head: full OpenCode driver suite, 64 synthetic tests passed, and runner source TypeScript check passed. Original author run had 61 passing tests before the new master tests. - The two new explicit and ambient regressions fail on the old implementation. - Independent source review passed. The two new synthetic regressions also passed independently. A separately filtered legacy pair hit fixture selection/chmod ordering and is not claimed as a passing run. - Tests use the real driver and proxy with a fake OpenCode child and synthetic upstream transport. They check key exclusion, exact URL/model/authentication, wrong-model and alternate-routing rejection, proxy bypass preservation, redirect rejection, and proxy shutdown. - The two advisory regressions failed before the correction: fallback routing reached upstream, and ambient NO_PROXY was lost. The full 64-test driver suite and source TypeScript check pass after the correction. - No real OpenCode SDK/provider qualification was run. The original WAL write mechanism remains unknown. - Repo-wide typecheck, test:run, and build were not run locally because local native/DB/model execution is excluded. Normal PR CI passed on head `3e1903272ce65b92055ea70f8dc586d9aa5e0205`: typecheck/release registry, general and serialized suites, runner static/Rust/Vitest checks, E2E shards, build, and canary dry run. Greptile is 5/5 on that head. Paperclip Review is neutral with no blocking findings; its two earlier findings are fixed and resolved. ## Risks The native OpenRouter route now uses the existing OpenAI-compatible proxy transport. Synthetic tests do not prove every live SDK response shape. The loopback token remains a credential, is redacted, and expires with the session. This change does not claim isolation of every child secret or persisted value. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted source investigation, implementation, synthetic tests, and review with tool use. No provider model call was used for verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- packages/paperclip-runner/README.md | 7 ++ .../opencode/opencode-server-driver.test.ts | 91 ++++++++++++++++++- .../opencode/opencode-server-driver.ts | 24 +++-- .../test/fixtures/fake-opencode-server.mjs | 5 + 4 files changed, 116 insertions(+), 11 deletions(-) diff --git a/packages/paperclip-runner/README.md b/packages/paperclip-runner/README.md index 488c97584b3..4003cb8da6f 100644 --- a/packages/paperclip-runner/README.md +++ b/packages/paperclip-runner/README.md @@ -388,6 +388,13 @@ support disabling reasoning; other runner providers reject an explicit selection before starting work. Native sessions expose support through the `perTurnReasoning` capability. +For the native `openrouter/` route, the parent-held provider proxy retains the +upstream OpenRouter key. The child config keeps the selected `openrouter` provider +and exact model, using a random loopback token restricted to that model's chat +completions endpoint. The proxy and token expire when the session closes. The +local token remains a credential and is redacted from diagnostics; this does not +claim isolation for every child credential or persisted value. + OpenCode's proxy sends agent instructions once per turn through the system prompt. The task envelope retains task-specific constraints and the completion contract, without repeating the system instructions. This applies to all users of the diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts index c8e2ed1c292..5f171927977 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.test.ts @@ -19,7 +19,7 @@ import { EventEmitter } from "node:events"; import { promisify } from "node:util"; import { getCACertificates } from "node:tls"; import { basename, dirname, join, resolve } from "node:path"; -import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; import { NATIVE_RUNTIME_ASSET_SCHEMA, @@ -37,6 +37,18 @@ import { openCodeServerDriverInternals, } from "./opencode-server-driver.js"; +const upstreamTransport = vi.hoisted(() => ({ endpoint: "", urls: [] as string[] })); +vi.mock("node:https", async importOriginal => { + const actual = await importOriginal(); + const http = await import("node:http"); + return { ...actual, request: (...args: Parameters) => { + if (!upstreamTransport.endpoint) return actual.request(...args); + upstreamTransport.urls.push(String(args[0])); + const options = args[1] as import("node:http").RequestOptions; + return http.request(upstreamTransport.endpoint, { ...options, agent: undefined }, args[2]); + } }; +}); + const roots: string[] = []; let fixtureRoot: string | undefined; let fixture: string; @@ -704,6 +716,81 @@ describe("OpenCodeServerDriver", () => { } }); + it.each(["explicit", "ambient"])("keeps OpenRouter upstream credentials out of the child while preserving exact provider/model routing (%s)", async source => { + await chmod(fixture, 0o755); + const key = "synthetic-upstream-openrouter-key"; + const model = "deepseek/deepseek-v4.1-flash-1017"; + const received: Array<{ authorization?: string; body: unknown; referer?: string }> = []; + let redirect = false; + const upstream = createHttpServer((request, response) => { + void (async () => { + const chunks = []; + for await (const chunk of request) chunks.push(Buffer.from(chunk)); + received.push({ authorization: request.headers.authorization, body: JSON.parse(Buffer.concat(chunks).toString("utf8")), referer: request.headers["http-referer"] as string }); + if (redirect) response.writeHead(302, { location: "https://must-not-follow.invalid/" }).end(); + else response.writeHead(200, { "content-type": "text/event-stream" }).end("data: [DONE]\n\n"); + })().catch(() => response.destroy()); + }); + await new Promise(resolve => upstream.listen(0, "127.0.0.1", resolve)); + const address = upstream.address(); + if (!address || typeof address === "string") throw new Error("Missing fixture address"); + upstreamTransport.endpoint = `http://127.0.0.1:${address.port}/chat/completions`; + upstreamTransport.urls = []; + const root = await mkdtemp(join(tmpdir(), "paperclip-openrouter-custody-")); + roots.push(root); + let session: Awaited> | undefined; + try { + if (source === "ambient") { + vi.stubEnv("OPENROUTER_API_KEY", key); + vi.stubEnv("no_proxy", undefined); + vi.stubEnv("NO_PROXY", "custody-bypass.invalid"); + } + const driver = new OpenCodeServerDriver({ model: `openrouter/${model}`, runtimeDirectory: root, command: fixture, + ...(source === "explicit" ? { environment: { PATH: process.env.PATH, OPENROUTER_API_KEY: key, NO_PROXY: "custody-bypass.invalid" } } : {}) }); + session = await driver.openSession({ runId: "custody", normalizedSessionId: "custody", workingDirectory: root }); + const configText = await readFile(join(root, "custody", "config", "opencode", "opencode.json"), "utf8"); + const config = JSON.parse(configText); + expect(config.model).toBe(`openrouter/${model}`); + expect(config.small_model).toBe(`openrouter/${model}`); + expect(Object.keys(config.provider)).toEqual(["openrouter"]); + expect(Object.keys(config.provider.openrouter.models)).toEqual([model]); + const { baseURL, apiKey } = config.provider.openrouter.options; + expect(configText).not.toContain(key); + const environmentText = await readFile(join(root, "custody", "data", "fake-environment.json"), "utf8"); + expect(JSON.parse(environmentText).keys).not.toContain("OPENROUTER_API_KEY"); + expect(environmentText).not.toContain(key); + expect(JSON.parse(environmentText).noProxy).toBe("custody-bypass.invalid,127.0.0.1,localhost"); + const persisted = await readFile(join(root, "custody", "data", "fake-provider-custody.json"), "utf8"); + expect(persisted).not.toContain(key); + expect(JSON.parse(persisted)).toMatchObject({ providerOptions: { baseURL, apiKey } }); + expect(baseURL).toMatch(/^http:\/\/127\.0\.0\.1:\d+\/v1$/); + expect(apiKey).not.toBe(key); + const url = `${baseURL}/chat/completions`; + const headers = { Authorization: `Bearer ${apiKey}`, "Content-Type": "application/json" }; + const payload = { model, messages: [{ role: "user", content: "synthetic" }] }; + expect((await fetch(url, { method: "POST", headers, body: JSON.stringify({ ...payload, model: "wrong-model" }) })).status).toBe(400); + expect(received).toHaveLength(0); + for (const [field, value] of Object.entries({ models: ["wrong-model"], route: "fallback", plugins: [{ id: "web" }], transforms: ["middle-out"] })) { + expect((await fetch(url, { method: "POST", headers, body: JSON.stringify({ ...payload, [field]: value }) })).status).toBe(400); + expect(received).toHaveLength(0); + } + expect((await fetch(url, { method: "POST", headers, body: JSON.stringify(payload) })).status).toBe(200); + expect(received).toEqual([{ authorization: `Bearer ${key}`, body: payload, referer: "https://paperclip.ing" }]); + expect(upstreamTransport.urls).toEqual(["https://openrouter.ai/api/v1/chat/completions"]); + redirect = true; + expect((await fetch(url, { method: "POST", headers, body: JSON.stringify(payload) })).status).toBe(502); + expect(upstreamTransport.urls).toHaveLength(2); + await session.close({ reason: "test" }); + session = undefined; + await expect(fetch(url, { method: "POST", headers, body: JSON.stringify(payload) })).rejects.toThrow(); + } finally { + await session?.close({ reason: "test" }); + upstreamTransport.endpoint = ""; + vi.unstubAllEnvs(); + await new Promise(resolve => { upstream.close(() => resolve()); upstream.closeAllConnections(); }); + } + }); + it.each([["HTTP_PROXY", "file"], ["ALL_PROXY", "directory"]] as const)("uses the runtime's %s, %s trust, and NO_PROXY without changing global transport", async (proxySetting, trust) => { const requests: string[] = []; const outgoingProxy = createHttpServer((request, response) => { @@ -910,7 +997,7 @@ describe("OpenCodeServerDriver", () => { "utf8", ), ); - expect(environment.keys).toContain("OPENROUTER_API_KEY"); + expect(environment.keys).not.toContain("OPENROUTER_API_KEY"); expect(environment.keys).not.toContain("PAPERCLIP_API_KEY"); expect(environment.keys).not.toContain("UNRELATED_SECRET"); expect(environment.keys).not.toContain("PAPERCLIP_PROVIDER_TRACE_PATH"); diff --git a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts index 4505b29d687..825812a0d09 100644 --- a/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts +++ b/packages/paperclip-runner/src/drivers/opencode/opencode-server-driver.ts @@ -2216,7 +2216,8 @@ async function startOpenCodeProviderProxy(baseUrl: string, key: string, model: s let payload: Record; try { payload = JSON.parse(body.toString("utf8")); } catch { response.writeHead(400).end(); return; } - if (!payload || typeof payload !== "object" || payload.model !== model) { + if (!payload || typeof payload !== "object" || payload.model !== model || + ["models", "route", "plugins", "transforms"].some(field => Object.hasOwn(payload, field))) { response.writeHead(400).end(); return; } @@ -2323,14 +2324,15 @@ async function startRuntime(input: { const identityKey = input.options.environment?.PAPERCLIP_AGENT_PRIVATE_KEY; const identityValues = identityKey ? [identityKey, JSON.stringify(identityKey).slice(1, -1), ...identityKey.split(/\r?\n/).filter(line => line && !line.startsWith("-----"))] : []; + const providerEnvironment = input.options.environment ?? process.env; const sensitiveValues = [ ...identityValues, password, authHeader, bridge.secret, assignedMcp?.token, - input.options.environment?.OPENROUTER_API_KEY, - input.options.environment?.PAPERCLIP_AI_PROVIDER_KEY, + providerEnvironment.OPENROUTER_API_KEY, + providerEnvironment.PAPERCLIP_AI_PROVIDER_KEY, ].filter((value): value is string => Boolean(value)); input.trace?.addSensitiveValues(sensitiveValues); const instructionRoot = @@ -2346,8 +2348,13 @@ async function startRuntime(input: { if (instructionRoot) externalDirectories[`${instructionRoot}/**`] = "allow"; const [modelProvider, ...modelIdParts] = input.options.model.split("/"); const providerModelId = modelIdParts.join("/"); - const providerProxy = modelProvider === "paperclip" && input.options.environment?.PAPERCLIP_AI_PROVIDER_URL - ? await startOpenCodeProviderProxy(input.options.environment.PAPERCLIP_AI_PROVIDER_URL, input.options.environment.PAPERCLIP_AI_PROVIDER_KEY ?? "", providerModelId, input.options.environment).catch(async error => { + const providerRoute = modelProvider === "openrouter" + ? { baseUrl: "https://openrouter.ai/api/v1", key: providerEnvironment.OPENROUTER_API_KEY ?? "" } + : modelProvider === "paperclip" && input.options.environment?.PAPERCLIP_AI_PROVIDER_URL + ? { baseUrl: input.options.environment.PAPERCLIP_AI_PROVIDER_URL, key: input.options.environment.PAPERCLIP_AI_PROVIDER_KEY ?? "" } + : null; + const providerProxy = providerRoute + ? await startOpenCodeProviderProxy(providerRoute.baseUrl, providerRoute.key, providerModelId, providerEnvironment).catch(async error => { await bridge.close().catch(() => {}); await removeIsolatedHome(isolatedHome).catch(() => {}); throw error; @@ -2397,8 +2404,9 @@ async function startRuntime(input: { ...(modelProvider === "openrouter" ? { options: { headers: OPENROUTER_ATTRIBUTION_HEADERS } } : {}), ...(providerProxy ? { npm: "@ai-sdk/openai-compatible", - name: "Paperclip connection", + name: modelProvider === "openrouter" ? "OpenRouter" : "Paperclip connection", options: { + ...(modelProvider === "openrouter" ? { headers: OPENROUTER_ATTRIBUTION_HEADERS } : {}), baseURL: providerProxy.baseURL, apiKey: providerProxy.token, }, @@ -2470,7 +2478,7 @@ async function startRuntime(input: { OPENCODE_SERVER_USERNAME: username, OPENCODE_SERVER_PASSWORD: password, ...performanceDiagnostics?.environment, - ...(providerProxy ? { NO_PROXY: [input.options.environment?.no_proxy ?? input.options.environment?.NO_PROXY, "127.0.0.1", "localhost"].filter(Boolean).join(",") } : {}), + ...(providerProxy ? { NO_PROXY: [providerEnvironment.no_proxy ?? providerEnvironment.NO_PROXY, "127.0.0.1", "localhost"].filter(Boolean).join(",") } : {}), }, ); const isolateProcessGroup = input.options.isolateProcessGroup ?? true; @@ -2960,7 +2968,6 @@ function sanitizedEnvironment( "ALL_PROXY", "SSL_CERT_FILE", "SSL_CERT_DIR", - "OPENROUTER_API_KEY", ]; const result: NodeJS.ProcessEnv = {}; const candidate = source ?? process.env; @@ -2982,7 +2989,6 @@ function sanitizedEnvironmentKeys(): string[] { "ALL_PROXY", "SSL_CERT_FILE", "SSL_CERT_DIR", - "OPENROUTER_API_KEY", ]; } diff --git a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs index 412d602171a..cde9d5fbc87 100644 --- a/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs +++ b/packages/paperclip-runner/test/fixtures/fake-opencode-server.mjs @@ -93,10 +93,15 @@ let pendingPermission = await readFile( let permissionStyle = "v2"; await mkdir(process.env.XDG_DATA_HOME, { recursive: true }); +await writeFile(join(process.env.XDG_DATA_HOME, "fake-provider-custody.json"), JSON.stringify({ + openrouterKey: process.env.OPENROUTER_API_KEY, + providerOptions: runtimeConfig.provider?.openrouter?.options, +})); await writeFile( join(process.env.XDG_DATA_HOME, "fake-environment.json"), JSON.stringify({ keys: Object.keys(process.env).sort(), + noProxy: process.env.NO_PROXY, home: process.env.HOME, configHome: process.env.XDG_CONFIG_HOME, projectConfigDisabled: process.env.OPENCODE_DISABLE_PROJECT_CONFIG, From 2cd747d1b6e4c118007fe3d60c757da440657f7d Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 20:59:01 -0500 Subject: [PATCH 40/41] perf(server): batch connection install capability checks (#17945) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip manages AI agents and their access to external connections. > - The connection grants screen describes which agents can receive an installation. > - Each candidate currently repeats authorization for every retained installation. > - A company installation makes this work grow with the square of the roster size. > - This pull request checks retained targets once inside the existing service owner. > - The benefit is fewer authorization calls with the same scoped capability results. ## Linked Issues or Issue Description **What happened?** The grants GET repeats connection preparation and retained-target checks for each candidate agent. A local 100-agent company-install fixture made 50,300 SQL calls and 10,000 target lookups in one capability-description operation. Open PR searches found no matching capability fan-out fix. **Expected behavior** The description must preserve retained-installation obligations and each candidate's permission. It should evaluate retained targets once per description. Writes must still authorize fresh. **Steps to reproduce** Run the committed benchmark with `--baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4`. It creates disposable synthetic PostgreSQL fixtures and loads the actual pre-fix service from Git. It compares every positional capability, reason, and operation. **Paperclip version or commit** Baseline `f6bc653f7f745adaafa26393852fb2f5a99256a4`. **Deployment mode** Built from source. Local real PostgreSQL proof. No hosted rollout. ## What Changed - Add `describeAgentInstallCapabilities` to connection management. Ordinary authorization and the batch share preparation, target checks, and fallback completion. - Use the batch only for direct saved non-AI connections and board actors. Complex resources, delegated actors, and unusual saved target IDs keep the ordinary evaluator. Fallback consumes the first preparation once. - Replace only the grants route's editable-agent loop. Keep configure/use checks, identity filtering, audience disclosure, and write authority on their existing paths. - Add scoped-grant, retained-target, error-order, freshness, fallback, full HTTP response, and real mutation regression coverage. Add a reproducible benchmark and evidence notes. ## Verification Real PostgreSQL 18.6 Linux Docker, Node 26.4.0. Two warmups and five measured samples per arm, in alternating order. These are service-component measurements with the same SQL-count instrumentation in both arms. They do not measure the whole hosted endpoint. | Fixture | Old median | Batch median | SQL calls old → batch | Target lookups old → batch | | --- | ---: | ---: | ---: | ---: | | 100 agents, company install | 24.753 s | 246.39 ms | 50,300 → 503 | 10,000 → 100 | | 100 agents, partial install | 6.339 s | 236.37 ms | 13,075 → 502 | 2,575 → 100 | | 1 agent, company install | 5.13 ms | 4.82 ms | 8 → 8 | 1 → 1 | All eight fixture shapes and every measured iteration had complete positional equality. One-agent timing ranges overlap. Full ranges, database settings, and limits are in `doc/plans/2026-10-27-connection-grants-batch-verification.md`. - **Passed:** `pnpm -r typecheck` and `pnpm build`, using pinned pnpm 9.15.4. - **Passed:** 116 focused tests: all 83 connection-management integration cases use real PostgreSQL and HTTP; 33 batch regressions execute real policy over synthetic query transport. The integration cases keep existing database creation, migrations, clients, services, and assembled HTTP server, with disposable Docker PostgreSQL. - **Passed:** complete grants GET parity across six human-role fixtures; real install PUT succeeds before revocation, then fresh GET and actual PUT deny after revocation without changing saved installs. - **Passed:** a real PostgreSQL red/green regression for empty saved target selectors, including candidate-specific decisions and later reached infrastructure faults. - **Passed:** independent old/new evaluator comparisons, including recursive and delegated fallback, run-context reuse, continuation, failure order, and principal/company isolation. - **Local full-suite limit:** `pnpm test:run` was attempted and stopped after native PostgreSQL fixtures could not initialize. An isolated failed native case reproduced that startup error. The host cannot allocate PostgreSQL SysV semaphores, including outside the sandbox. The focused suites use an existing constructor seam to start disposable Docker PostgreSQL. This is not a complete local full-suite pass. - **Passed:** all 16 native-runtime cases that failed to start PostgreSQL on the host, rerun with the disposable Docker provider. The full local workflow remains incomplete. - `git diff --check` passes. Greptile is 5/5 with no actionable findings on final head `41dcf3ae029b6c60999ca6208136399b0af663d0`. All final-head CI gates passed, including the full PR workflow. The two Storybook checks are expected skips because this change has no UI changes. Normal reproduction: ```sh node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4 cd server pnpm exec vitest run src/__tests__/connection-install-capabilities.test.ts src/__tests__/connection-management.integration.test.ts ``` ## Risks Retained facts are observed once during one descriptive call. A mid-call revoke or deletion can leave a stale displayed choice. The next description and every mutation or human continuation read fresh authority. The batch returns no reusable authorization context. The bound applies to target evaluations in the eligible shape. Scoped grant payloads and hierarchy reads can still grow inside each policy decision. Complex resources and agent callers keep existing costs. Hosted latency improvement remains unverified. There is no migration, provider protocol change, new worker, persistent cache, or UI contract change. ## Model Used OpenAI Codex with code editing, execution, and test tools. Design, test, and independent review agents used exact model IDs `gpt-6.1-sol` and `gpt-6-astra` with xhigh reasoning. The host did not expose the primary session's exact model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- ...27-connection-grants-batch-verification.md | 127 ++++++ ...0-27-connection-grants-capability-batch.md | 220 +++++++++ doc/plans/2026-10-27-next-bottlenecks.md | 254 +++++++++++ ...nchmark-connection-install-capabilities.ts | 181 ++++++++ .../connection-install-capabilities.test.ts | 417 ++++++++++++++++++ .../connection-management.integration.test.ts | 170 ++++++- server/src/routes/tool-access.ts | 6 +- server/src/services/connection-management.ts | 185 ++++++-- 8 files changed, 1523 insertions(+), 37 deletions(-) create mode 100644 doc/plans/2026-10-27-connection-grants-batch-verification.md create mode 100644 doc/plans/2026-10-27-connection-grants-capability-batch.md create mode 100644 doc/plans/2026-10-27-next-bottlenecks.md create mode 100644 scripts/benchmark-connection-install-capabilities.ts create mode 100644 server/src/__tests__/connection-install-capabilities.test.ts diff --git a/doc/plans/2026-10-27-connection-grants-batch-verification.md b/doc/plans/2026-10-27-connection-grants-batch-verification.md new file mode 100644 index 00000000000..04b3d3de940 --- /dev/null +++ b/doc/plans/2026-10-27-connection-grants-batch-verification.md @@ -0,0 +1,127 @@ +# Connection-grants batch verification + +Date: 2026-10-27. Implements the +[selected design](2026-10-27-connection-grants-capability-batch.md). + +## Change and authority boundary + +`connectionManagementService.describeAgentInstallCapabilities` prepares a direct +saved connection once, checks its retained installation targets once, and checks +each additional candidate in order. It uses the same private target evaluator as +ordinary authorization. The grants GET replaces only its editable-agent loop. + +The optimized shape is board actors, non-AI saved connections, canonical retained +and candidate IDs, and no contextual or proposed-resource selectors. Delegated +agents and +complex resources finish the first already prepared evaluation and then use the +ordinary evaluator. Preparation is not restarted. Configure/use capabilities, +grant identity filtering, audience disclosure, writes, and human continuations +retain their existing authority paths. + +Reuse lasts one descriptive call. It is not an atomic snapshot: a retained +permission or target removed mid-call can leave a stale displayed choice. The +next description and actual authorization must read fresh authority. No context, +mutation authority, or reusable cache escapes the batch. + +## Reproduction and measurements + +Pinned baseline: `f6bc653f7f745adaafa26393852fb2f5a99256a4`. +Treatment service SHA-256: +`f5fca127e8a669d343b61455187e102dc6705d87389544ff4edfa5a066185656`. +The benchmark loads the actual baseline service from Git, verifies that its other +runtime dependencies and lockfile match, and compares it with the new production +batch. Every positional capability, reason, and operation must match. + +```sh +node --import ./server/node_modules/tsx/dist/esm/index.mjs \ + scripts/benchmark-connection-install-capabilities.ts \ + --baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4 +``` + +The fixture is a Cloud board member who is not the creator, with a target-scoped +grant covering the synthetic roster. Company installs retain the whole roster; +partial installs retain one quarter, with at least one retained agent. Each arm +has two warmups and five measured samples in alternating order. Counts below are +per invocation. Full positional equality passed in every iteration. Individual +SQL shapes are unchanged; the benchmark saves statement counts, parameters, rows, +and treatment `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` outside timed samples. + +| Agents / installation | Old median ms (range) | Batch median ms (range) | SQL calls old → batch | Target lookups old → batch | +| --- | ---: | ---: | ---: | ---: | +| 1 / company | 5.13 (4.53–7.45) | 4.82 (4.38–5.08) | 8 → 8 | 1 → 1 | +| 1 / partial | 5.16 (4.01–5.54) | 5.26 (4.83–6.13) | 7 → 7 | 1 → 1 | +| 8 / company | 231.79 (221.63–238.96) | 31.11 (24.79–35.86) | 344 → 43 | 64 → 8 | +| 8 / partial | 128.84 (114.42–160.09) | 43.07 (39.36–45.69) | 126 → 42 | 22 → 8 | +| 32 / company | 2,804.04 (2,516.50–4,928.42) | 77.19 (75.56–132.76) | 5,216 → 163 | 1,024 → 32 | +| 32 / partial | 697.51 (676.38–732.57) | 73.92 (73.21–77.78) | 1,464 → 162 | 280 → 32 | +| 100 / company | 24,753.29 (23,849.21–25,019.72) | 246.39 (234.94–278.72) | 50,300 → 503 | 10,000 → 100 | +| 100 / partial | 6,338.58 (6,239.68–6,526.36) | 236.37 (228.40–288.54) | 13,075 → 502 | 2,575 → 100 | + +At 100 agents, returned Drizzle rows fell from 60,200 to 602 for company installs +and from 15,475 to 526 for partial installs. The one-agent controls are neutral, +including a small overlapping timing regression for the partial case. The table +retains every sample range; do not extrapolate a latency curve from these medians. + +The host could not start native PostgreSQL because SysV semaphore allocation +failed, including an unsandboxed low-resource initdb attempt. Local proof used +the existing test constructor seam with a scratch Docker provider: PostgreSQL +18.6 Debian/aarch64 Linux, Node 26.4.0, loopback-only networking, tmpfs storage, +and no simulated database delay. Each provider instance removed only its own +container. Existing database creation, migrations, clients, service code and +cleanup remained in use. The normal embedded package targets Darwin PostgreSQL +18.1; minor version, platform, networking, and compute differ. Defaults included +128 MiB shared buffers, 4 MiB work memory, and JIT enabled. + +## Correctness evidence and limits + +- Focused real-policy regressions cover scoped grants without installs, retained + missing/denied targets, mixed and duplicate candidates, 403 versus reached + 409/500 faults, earlier denial masking an unreachable fault, forced mutation + mode, fresh calls and reauthorization, actor/company isolation, + deliberate within-call staleness, and ordinary fallback preparation counts. +- Real PostgreSQL differential coverage includes company and partial installs, + empty/duplicate candidates, foreign targets, uppercase UUID comparisons, + profile/proposed-provider fallback, and an injected SQL division-by-zero fault + that must escape only when reached. +- The assembled grants GET compares complete old-loop and batch responses across + creator, manager, member, viewer, inactive, and administrator fixtures. It + verifies personal identity filtering, audience disclosure, ordered editable + IDs, and an actual allowed install PUT followed by revocation, fresh denied + capabilities, and a denied PUT with saved installs unchanged. +- Independent review compared the pinned and extracted ordinary evaluator in + 18 context/query/lock-order cases and 18 complete batch cases. Additional + checks covered delegated fallback, recursive profiles/gateways/AI pools, run + capture reuse, a succeeded-run continuation, and fresh revocation. This + supplemental proof uses synthetic database transport, not real lock contention. + +Initial test failures were incorrect fixture expectations: inactive access hides +the resource with 404, database roster order is not insertion order, and the +existing final install mutation gate requires an unscoped agent-configuration +grant. These were corrected in tests. Complete old/new GET equality passed before +these extra assertions. No product behavior was changed to satisfy them. + +Final guard review also found a real defect before merge: saved target IDs are +text, and an empty saved target makes the legacy check depend on the current +candidate. An executable regression returned `[false, true]` in the original +service and `[false, false]` in the first batch. Real PostgreSQL reproduced it. +The guard now requires canonical, nonempty retained IDs as well as candidates; +unusual saved selectors keep ordinary evaluation. The fixed path matches both +the complete capabilities and a later reached infrastructure fault. No stored +target is dropped or normalized, and legacy selector semantics stay unchanged. + +The bound is retained-plus-candidate target evaluations, not total CPU, SQL rows, +or endpoint latency. Scoped grant arrays and company hierarchy reads can still +grow inside each policy decision. Recursive and delegated fallback retain their +existing costs. The benchmark measures the descriptive service component; +hosted request attribution and production latency improvement remain unverified. + +Local coverage contains 33 batch regressions and 83 connection-management +integration cases, using disposable Docker PostgreSQL. +The required workspace typecheck and build passed using pinned pnpm 9.15.4. +The standard full Vitest workflow was attempted, then stopped after native +database fixtures failed to initialize. A separately isolated failing native +runner case reproduced the same PostgreSQL startup failure. This is not a full +local test pass; the critical connection suites use the disposable Docker +provider described above, and the complete CI suite remains a separate gate. +Final test/CI outcomes are recorded in [PR #17945](https://github.com/paperclipai/paperclip/pull/17945); this document does not +substitute for a final-head check or deployed acceptance. diff --git a/doc/plans/2026-10-27-connection-grants-capability-batch.md b/doc/plans/2026-10-27-connection-grants-capability-batch.md new file mode 100644 index 00000000000..84cdaac6caa --- /dev/null +++ b/doc/plans/2026-10-27-connection-grants-capability-batch.md @@ -0,0 +1,220 @@ +# Connection-grants capability fan-out: design and safety proof + +Date: 2026-10-27. This records the selected design before implementation. +See [implementation verification](2026-10-27-connection-grants-batch-verification.md) +for measured results and completed acceptance checks. + +## Recommendation + +Add one narrow, service-owned batch operation for describing which agents may +receive a connection installation. Factor the common installed-agent checks out +of the per-candidate loop. Keep authorization policy in its existing owner and +keep actual mutations on the existing fresh authorization path. + +Start with board actors describing a direct, saved, non-AI connection. Preserve +the ordinary evaluator for delegated agents and complex resources until their +additional obligations have executable parity evidence. This is a bounded first +fix, not a general authorization cache or a rewrite of connection permissions. + +The endpoint currently asks, for each candidate agent, whether the actor may +change the connection's installations. Each answer checks **all retained +installed agents**, followed by the candidate if it is not already installed. +A company installation expands to the company roster. For 100 candidates with +100 retained targets, this means 10,000 target evaluations before the other route +work. The proposed eligible path needs 100 retained-target evaluations, with +extra evaluations only for candidates outside that set. + +Those retained checks are real policy obligations. The fix removes their repeated +execution; it must not remove their authority requirements. + +## What the evidence establishes + +The prior observation window contains two successful grants GETs lasting +56.506 and 66.388 seconds. There is no matching request trace attributing their +duration to this loop. See [the measurement record](2026-10-27-next-bottlenecks.md). +The source and local probes establish amplification, not the fraction of hosted +latency it causes or the eventual improvement. + +The production path is: + +- `server/src/routes/tool-access.ts`, `describeConnectionCapabilities`: selects + the company roster and calls `connections:profiles` once per agent, with + mutation authority and `readOnly: false`. +- `server/src/services/connection-management.ts`, `authorize`: resolves the + saved connection, expands its retained installations, then checks the ordered + target set. `capability` converts only errors with status 403 into denials; + other faults escape. +- `server/src/services/authorization.ts`: owns membership and scoped permission + decisions. Some subtree grants can read the whole company hierarchy for each + target, so linear target evaluations do not guarantee linear rows read. + +Three independent designs were compared: structural factoring, general lazy +request caching, and a narrow guarded batch. A separate judge selected the narrow +batch. The final design incorporates the structural candidate's preparation +continuation and differential test approach, plus the cache candidate's careful +first-visit ordering and separate work counters. + +| Local experiment | Observed result | Limit | +| --- | --- | --- | +| Existing exported service, all-installed fixtures | 4 agents caused 16 target reads; 8 caused 64 | Synthetic database transport, not PostgreSQL | +| Handwritten factored terminal phase compared with the current service | Six static output/fault cases matched; target reads fell 16 to 4 and 64 to 8 | Treatment omits preparation and is not the proposed production implementation | +| General read-reuse experiment, 32 agents | Physical transport reads fell 3,168 to 36, but 1,024 target invocations remained | Demonstrates why caching alone can retain quadratic work | +| Existing policy with a grant scoped to one agent and no installations | Named candidate allowed; targetless request denied | Rules out an extra targetless authorization preflight | +| Existing service with an unavailable retained target | Normal capability denied; `mutation: false` shortcut allowed | Rules out bypassing retained installation checks | +| Existing continuation after membership removal | Fresh `reauthorize` denied | Does not prove a future batch's freshness | + +All three local proof scripts were rerun successfully by the primary agent and +the independent judge. They execute unchanged exported services over synthetic +Drizzle transport. They are level-4 evidence for the exercised current-code +facts, not proof of the unimplemented batch, assembled HTTP equivalence, +PostgreSQL performance, or delegated-actor parity. + +## Owning API and eligibility + +The new operation belongs in `connectionManagementService`, beside `capability`. +Its only initial consumer replaces the `editableAgentIds` loop. Use a fixed +`connections:profiles` action and mutation-mode semantics rather than exposing +general action or read-only knobs. Accept the original actor, base connection +resource, ordered candidate IDs, and existing read context. Return positional +capability descriptions; the route derives the existing editable-ID list. + +Do not return an authorization context, audit authority, reusable session, cache +handle, or continuation. Do not change the wire schema. Keep manager checks, +configure/use capabilities, membership gates, grant identity filtering, and +audience disclosure on their existing paths. + +The initial optimized shape is deliberately explicit: + +- Board actor; one direct saved connection whose actual saved purpose is not AI. +- No non-null profile, gateway, routine, project, or issue selector. +- No own `requestedProvider` or `requestedInstallCompanyIds` property, and no + `selfServiceAgentInstalls: true` request. +- Validated, nonempty roster IDs in their original spelling and input order. + Retained target IDs must also be canonical and nonempty. Unusual identifiers + use the ordinary path rather than normalization: a saved empty target is a + candidate-dependent selector in the legacy evaluator. + +Use the saved connection's provider, purpose, and creator derivation, including +its ordinary personal-grant handling. Caller labels cannot establish eligibility. +Unsupported shapes retain the existing per-candidate evaluator. In particular, +AI pools, profile bindings and selectors, current/proposed gateway resources, +and delegated actor contexts keep all recursive checks. + +## Evaluation algorithm + +Extract private preparation, target evaluation, and completion primitives from +the existing evaluator. Both the ordinary and batch paths must consume these +same primitives; do not copy policy into the route or implement a second policy +engine. Ordinary authorization always starts with fresh private state. + +1. An empty candidate list returns an empty result without authorization, as + the current loop does. +2. Begin preparation for the first actual candidate in the current order. + Discover eligibility during the normal saved-connection lookup. The private + preparation frame is consumed exactly once. If ineligible, finish that same + evaluation through the ordinary path, then evaluate the remaining candidates + normally. Do not restart preparation to fall back. +3. For eligible inputs, form the same ordered set of retained installation + targets, including the existing company-install expansion. Evaluate each + retained target once, using the original check order: connection policy, + applicable proposed-provider/resource trust checks, target existence, and + principal-specific target authority. The shared primitive preserves these + obligations even where the initial eligibility guard excludes some branches. +4. A retained-target 403 denies every candidate with the same current denial. + A reached non-403 fault aborts the operation. Do not inspect later targets to + discover a fault the original evaluator would not reach. +5. After the retained prefix succeeds, visit candidates in input order. A + candidate already in the retained set uses the final retained decision, + matching the current ordered-set behavior and success reason. An additional + candidate uses the shared target primitive. Its 403 denies that candidate; + a reached non-403 fault aborts. Return one result per input position. +6. With no retained installations, the candidate is the policy target. Do not + add a targetless preflight: a grant scoped to that candidate may correctly + allow it while denying an unspecified target. + +For N candidate positions and S distinct retained targets, the target phase is +bounded by S + N evaluations, and often fewer when candidates are retained. +This is not a promise of S + N SQL statements. Authorization may still read +membership, grants, and hierarchy for each distinct target; complex fallback +keeps its present cost. Measure those separately before widening the change. + +Avoid eager prefetch and `Promise.all` here. A denied retained target currently +prevents later target lookups. Loading those targets first changes which faults +are reached. Parallelizing the old loop also retains its total work. + +## Two safety facts and the blast radius + +**1. A displayed permission must retain every existing installation obligation +and its own target scope.** The batch cannot turn a denied or absent retained +target into editable siblings, substitute a targetless decision, or reuse an +allowed candidate's decision for a different candidate. Preserve positional +results, reason, operation, ordered deduplication, and the 403/fatal-fault boundary. +Existing-code counterexamples prove these shortcuts are unsafe. Actual old/new +service and route comparisons remain required to prove the proposed change. + +**2. A description must never become reusable mutation authority.** Batch state +exists only inside one call and cannot cross actors, companies, connections, +requests, operations, or human continuations. Actual writes and `reauthorize` +start fresh. The batch returns descriptions only; identity disclosure does not +consume them as authority. + +There is a deliberate consistency change: reused facts and retained decisions +are observed once during the description rather than reread for every candidate. +This is neither a database snapshot nor an atomic revocation boundary. A revoke +or target deletion during the call can leave a briefly stale displayed choice; +a second GET and an actual mutation must observe fresh authority. Sequential +first visits preserve static-fact failure order, not every possible failure or +revocation between removed duplicate reads. Document and test this honestly. + +Delegated agents remain on the ordinary path initially. `captureRunIdentity` +locks and can reconcile pending steering identity; it is not a pure read. A +future extension must prove run ownership, work mode, API-key status, responsible +human and external-human ceilings, and pending-identity behavior. Preserve the +original actor reference because existing read-context reuse checks identity by +reference. Do not memoize capture or attach batch state to that context. + +The intended change touches the connection-management owner, its route consumer, +and focused tests. Existing connection policy is the authority. There is no +schema migration, provider protocol change, background worker, persistent cache, +UI contract change, or blanket creator/manager allow rule in this design. + +## Implementation and acceptance gates + +1. **Shared-helper refactor:** preserve ordinary evaluation before enabling the + batch. Prove preparation runs exactly once for the first candidate on both + eligible and fallback paths. Keep its continuation private and consumed. +2. **Actual service differential:** compare the new production batch against the + old full evaluator with a disposable PostgreSQL fixture. Compare every + positional `allowed`, `reason`, and `operation`, not just editable counts. + Cover empty/duplicate candidates; no, partial, and company installations; + scoped grants with no installations; missing/foreign retained and candidate + targets; local board and Cloud creator, owner, administrator, operator, + ordinary member, viewer, and inactive membership. +3. **Failures and fallback:** retained denial before an unreachable fault; + candidate denial followed by a reached later fault; preparation/target 500 + and relevant reconciliation 409; only 403 converted. Exercise every excluded + shape, including empty AI pools, explicit null selectors, proposed providers, + company-install changes, profile selectors, and linked gateway obligations. +4. **Full grants GET:** compare complete assembled responses, including order, + editable IDs, configure/use flags, personal identity filtering, manager + visibility, audience members, and route access gates. Exercise the existing + connection Permissions and agent installation UI consumers. +5. **Freshness and isolation:** controlled mid-batch revoke/delete; fresh second + GET; GET then revoke then real mutation; delayed human/OAuth continuation; + concurrent actors, companies, and connections. Agent fast-path enablement + additionally requires real run/key/steering and lock-contention fixtures. +6. **Performance:** at roster sizes 1, 8, 32, and 100 or larger, count preparation, + target evaluations, policy decisions, SQL calls, returned rows, and wall time + separately. Test subtree grants and unrelated-request responsiveness. Measure + assembled route phases and database/pool delay before attributing savings. + +After implementation, run the relevant regression suite and the repository's +required typecheck, tests, and build before PR hand-off. Require passing final-head +CI and review before hosted rollout. Repeat acceptance on the exact serving +revision, verify both response behavior and latency with enough samples, and +retain a way to restore the ordinary evaluator if parity fails. + +This design pass changed no application behavior and performed no hosted mutation. +It establishes a preferred implementation and executable current-code safety +constraints. Complete batch parity and production latency improvement remain +unproved until the implementation and acceptance gates above pass. diff --git a/doc/plans/2026-10-27-next-bottlenecks.md b/doc/plans/2026-10-27-next-bottlenecks.md new file mode 100644 index 00000000000..063bca09e4c --- /dev/null +++ b/doc/plans/2026-10-27-next-bottlenecks.md @@ -0,0 +1,254 @@ +# Next bottleneck batch + +## Scope and observation + +Find the next bounded batch after the three merged slow-query fixes. This is a +diagnosis pass: rank fresh costs, identify actual callers and existing work, +and define the smallest performance and correctness experiments. No application +code, schema, hosted configuration or deployment changes are part of this pass. + +The reported workspace's Cloud, Neon database/runtime role, and Railway +deployment were revalidated from the live registry. The staging app served +`8548173fc9efdc0fa07644fb92dcc0b6851a95e5` throughout the query window. Upstream +`master` was refreshed to `50b932a142216331b8d476042b454bc123739c40`. Private +resource mapping, normalized SQL and raw logs remain outside Git. + +After that stable measurement window, the final serving check at 18:49:08 UTC +observed revision `6e781e0ab20cdaeb6570cebdb5f14dbb8557ba04`. +The grants route/policy/authorization paths, wake-notice owner and skills +route/service are byte-identical between that source and the measured source. +The counts below describe the earlier window, not a post-rollout measurement. +Re-baseline the assembled application before accepting a later performance fix. + +The fresh query window is 2026-10-27 **18:32:51.270–18:38:07.464 UTC**, or +316.194 seconds. Both snapshots read every statement for the exact application +database OID and runtime-role OID through the existing monitoring view, using +bounded read-only transactions. There is no top-100 or query-family filter. + +- Statistics reset stayed at 14:21:16.446 UTC; eviction count stayed zero. +- There were 4,573 starting and 4,590 ending entries, with no missing entries or + invalid matched counters. Entry-level epochs were checked. +- The 1,147 matched entries with positive call deltas account for 121,299 calls + and 596.418 seconds of summed SQL execution time. They are all top-level + entries, so this total does not add nested statement timings. +- Seventeen new entries contribute 25 calls and 75.941 ms as separate lower + bounds. They are excluded from the matched ranking. +- I/O timing is disabled. Zero timing fields do not establish zero I/O latency. + Point-in-time wait samples cannot establish sustained pressure or its absence. +- The probe comparator passed eight controlled cases, including global/entry + resets, decreasing counters, missing/new entries, evictions and scope changes. + An initial diagnostic assertion about a pooler's default read-only setting + was rejected before capture; the successful probe checks the actual transaction. + +The matched HTTP read contains 997 completed log records from the same deployment +and exact time window, below its 2,000-record cap. One WebSocket connection is +excluded from ordinary request latency. The remaining 996 include 603 HTTP 304 +responses and one HTTP 500 from issue summaries; this is an observed traffic +sample, not a clean acceptance run. Duration units follow +[Railway's log contract](https://docs.railway.com/observability/logs). + +SQL execution time can overlap across requests and background workers. Its sum +is neither elapsed time nor endpoint latency. Honeycomb queries using the +verified host, forwarded host and stack fields found no matching request spans +in the checked dataset/window; the SQL share of these HTTP requests is unproved. + +## Fresh rankings + +| SQL caller | Calls | Mean SQL time | Total SQL time | Returned/affected rows | Disposition | +| --- | ---: | ---: | ---: | ---: | --- | +| Chat lifecycle milestone candidates | 167 | 1,497.65 ms | 250.108 s | 0 | Exact caller overlaps open draft #17606 | +| Terminal workspace last activity | 245 | 591.64 ms | 144.951 s | 245 | No qualified open fix for these reads found | +| Inbound wake-notice candidates | 647 | 48.99 ms | 31.694 s | 0 | Exact duty overlaps open draft #17647 | +| GitHub stale-review repair UPDATE | 12 | 1,658.12 ms | 19.897 s | 0 | Exact duty overlaps open draft #17664 | +| GitHub review source-run fallback | 82 | 230.25 ms | 18.880 s | 1 | Distinct query in GitHub maintenance owner | +| GitHub stopped-response recovery | 12 | 1,340.53 ms | 16.086 s | 120 | Exact duty overlaps open draft #17661 | + +The first two families still account for **66.24%** of matched SQL time after +the earlier query improvements. Their existing optimized SQL shapes are active. +The historical pre-fix fingerprints had no new calls in this window. + +Guarded membership upserts used **41.660 ms across 84 calls**. The advisory-lock +statement with a large historical maximum used **0.950 ms across 174 calls**. +Neither historical cumulative total belongs on the new bottleneck shortlist. + +| Ordinary HTTP route | Requests | Mean | Sample p95 | Maximum | Relevant result | +| --- | ---: | ---: | ---: | ---: | --- | +| GET connection grants | 2 | 61.447 s | 66.388 s | 66.388 s | Both 200; minimum 56.506 s; very small sample | +| GET company skills | 20 | 2.733 s | 6.015 s | 13.945 s | All 304 | +| GET company issues | 135 | 1.079 s | 2.957 s | 9.671 s | 112 returned 304 | +| GET sidebar badges | 197 | 482 ms | 1.440 s | 5.851 s | All 304 | +| GET run events | 203 | 346 ms | 1.063 s | 4.273 s | 137 returned 304 | + +Percentiles use nearest rank within the observed sample. Upstream request +duration is almost identical to total duration for these routes. That locates +the observed delay behind the edge, but does not distinguish application CPU, +database round trips, pool wait or other dependencies. + +## Recommended next work + +### 1. Connection-grants capability fan-out + +This is the largest newly observed individual request delay. The route is +`server/src/routes/tool-access.ts:1754`. Listing grants itself uses batched +database reads. The route also describes connection capabilities by inspecting +every company agent. Each inspection asks the connection-management policy for +a capability; its default mutation-authority calculation expands existing +company-wide installations to their agents and checks each target serially. +This creates quadratic authorization fan-out in the reproduced service path. No provider HTTP call +was found in this listing path. The relevant route and policy paths are +unchanged between the measured source, later serving source and refreshed +upstream master. The per-agent helper is `tool-access.ts:665–692`; the repeated +install expansion and target checks are `connection-management.ts:143–158` and +`:225–249`. It deliberately checks retained installed targets as well as the +candidate target. + +The fresh SQL window contains thousands of individually fast membership, +project, issue and run-authority reads. Their counts alone do not attribute them +to these two requests. A scratch local reproduction called the real exported +`connectionManagementService.capability` → `authorize` → authorization service +with a synthetic database transport and an implicit local board actor. Four +agents caused 16 real target probes / 28 transport queries; eight agents caused +64 probes / 88 queries. The outer iteration follows the private route helper's +source; the helper itself was not exported or executed by the harness. This +proves the policy amplification through real code (blast-radius evidence level +4), not its share of the observed hosted request latency or a faster treatment. +The Cloud connection-creator case also reproduced 16 → 64 target probes, with +60 → 216 transport queries and 32 → 128 membership reads. A missing installed +target negative case denied all capabilities. These are synthetic transport +counts through real policy code, not real PostgreSQL or HTTP measurements. +Next measure the assembled route with representative installed scopes and +controlled database round-trip latency. The fix should make capability description share equivalent +request-local facts while retaining each target's authorization decision. + +**Safety fact to prove:** any reused authority fact belongs to the same actor, +company, connection and request; a denied or absent target cannot inherit an +allowed sibling's capability. Cover board, user and agent callers, personal and +company installations, project/resource restrictions, removed grants, missing +agents and mixed allowed/denied targets. Preserve current membership/revocation +reads between requests. Do not make a one-minute route appear fast by skipping +its checks. Do not bypass installed-target checks by passing `mutation:false` or +`readOnly:true`; a missing installed target currently denies every candidate. + +### 2. Qualify existing inbound wake-notice work + +The measured query is owned by +`server/src/services/chat-channels/inbound/wakeup-notices.ts`, in +`createInboundWakeNotices.enqueueInboundWakeupPublications`. This selects +historical notice eligibility; it is not the execution-admission gate. It ran +647 times at approximately 2.05 calls per second and returned no rows. Its +31.694 seconds of SQL time included 12,973,945 shared-buffer hits and no +shared-block reads or temporary blocks in this window. The deployed and current +upstream query paths are identical. + +Direct overlap is confirmed in +[#17647](https://github.com/paperclipai/paperclip/pull/17647), draft head +`ca8f2d17c64fc7cecff4526817d1ac6e1d479ff4`. It changes this exact selector and +removes its normal publication-maintenance call. Its coordinator reconstructs +durable scopes at startup, retains cursors, and consumes committed-change +notifications. Explicit/manual publication drains still call the selector, so +eliminating all 647 observed calls is not established. The PR is stacked on +`codex/sleep-cancel-contention` rather than master. Advance and qualify this +existing work, its dependencies and remaining callers before creating a +competing implementation. This pass did not run its tests or qualify its CI. + +For remaining query work, capture the actual SQL and parameters from this service and +reproduce long retained action/publication/comment history with no pending +notice, followed by a single real eligible notice. Compare plans and all output +columns before deciding whether to narrow the query or wake it from concrete +committed obligations. A quiet path must still reconstruct unresolved work at +startup and recover missed notifications. + +**Safety fact to prove:** a narrower candidate path still finds every notice +authorized by the original action/publication relationship, with unchanged +company scope and deduplication. Cover legacy/native lineage, malformed or +missing context, delayed commits, resolved/expired actions, failed publication, +duplicate delivery and drain/resume. Removing the recurring call requires proof +for every producer and restart/recovery path, not only an empty-queue test. + +### 3. Skills-list reconciliation and repeated resolution + +`server/src/routes/company-skills.ts:467` calls the service list at +`server/src/services/company-skills.ts:3269`. Every request first awaits +`ensureSkillInventoryCurrent` at line 3235: bundled imports, bundled-release +verification, folder reconciliation and local-source inspection. The shared +promise coalesces concurrent work but is removed when it finishes, so a later +request repeats the refresh. A GET can therefore perform filesystem and +database reconciliation before listing any skills. + +The list then resolves every agent's desired skills again for each displayed +skill (`company-skills.ts:3351`). Reference resolution performs searches over +the skill list at line 2152. This is a concrete repeated-work lead, but its share +of the measured 2.733-second HTTP mean remains unproved. These service/route +files are identical in the deployed source, local integration and current +upstream master. + +First profile the real service with representative skill and agent counts, +separating inventory refresh, database reads, desired-skill resolution and +serialization. Compute each agent's desired-key set once per request as the +first small candidate, comparing complete results. Treat any longer-lived +inventory cache as a separate change requiring freshness proof. + +**Safety fact to prove:** each agent contributes at most once to each selected +skill, with identical exact-ID/key/slug resolution and ambiguous/missing alias +handling. Preserve visibility, folder/subtree/search filtering, sort order, +last-editor results and changed local-source discovery. A cached authorized +inventory cannot be reused for another principal's visibility. + +## Existing work and deferred candidates + +- [#17606](https://github.com/paperclipai/paperclip/pull/17606) owns the exact chat + lifecycle milestone path. Its draft branch moves lifecycle work to committed + changes and reconstructs pending work. Audit its producer/recovery proof and + drive its checks before starting a competing polling rewrite. Its draft + status and source design are not proof that it is safe or merge-ready. +- [#17664](https://github.com/paperclipai/paperclip/pull/17664) and + [#17661](https://github.com/paperclipai/paperclip/pull/17661) overlap the two + measured GitHub maintenance duties. The source-run fallback is a distinct + delivery/link/run lookup in `chat-github-checks.ts:181–202`, used when the + review lacks a resolvable run. It still merits a local plan/equality experiment, + coordinated with those owner changes. A previous scalar delivery lookup fix + in #17669 is already deployed and must not be proposed again. +- Workspace activity is called at + `server/src/services/execution-workspaces.ts:3576` through the shared terminal + workspace worker, already migrated by #16282. Its query is at + `server/src/services/git-workspace-expiration.ts:153–160`. + [#17505](https://github.com/paperclipai/paperclip/pull/17505) adjusts cursor and + retry deadlines but leaves this query/call intact. A second pass could batch + repeated activity reads; fresh destructive eligibility, live leases and + concurrent reopen checks must remain authoritative. No qualified open fix + reducing these activity reads was found in the scoped check. +- [#17896](https://github.com/paperclipai/paperclip/pull/17896), + [#17894](https://github.com/paperclipai/paperclip/pull/17894), and + [#17893](https://github.com/paperclipai/paperclip/pull/17893) are already merged + after the measured deployment. They narrow explicit-continuation run reads, + avoid duplicate GitHub webhook body transfer and narrow review-wait JSON. + The final 18:49:08 serving source contains all three. Evaluate their impact in + a new post-rollout window before duplicating those changes. +- [#17907](https://github.com/paperclipai/paperclip/pull/17907) keeps the skill + cache lock alive during slow preparation. Its documented scope does not + explain why preparation is slow or remove the skills-list work above. + +The prior Slack continuation's 353.059-second delay is confirmed by an exact +deployed-version `heartbeat.queue` span without children. Executor startup begins +after that wait. A periodic recovery chain stayed in flight for at least +240.086 seconds and suppressed eight ticks, but an earlier recovery pass reached +its post-resume stage while the continuation was still queued. Source cleanup +had completed before the answer arrived. This narrows the failure to queue / +admission or recovery scheduling; it does not establish the blocking gate, a +lost wake, or SQL as the cause. Instrument gate decisions and recovery stages +before choosing a repair. No task was replayed or re-woken for diagnosis. + +## Finish conditions for the next implementation batch + +Take at most the three priorities above, advancing existing work where it +overlaps. Pin the deployed baseline, prove +the suspected cost with the real caller, define the performance criterion, then +compare complete behavior and adversarial authority/recovery cases. Keep +independent changes in independent PRs with green final-head checks. No migration +or boot script is justified by this discovery pass. Repeat hosted acceptance and +a fresh comparable counter/request window after an authorized rollout. + +The private evidence set retains all raw inputs, comparator validation, exact +source inventories, request summaries and query snapshots. This document is the +sanitized discovery handoff. Correctness of proposed optimizations is unproved +until their real-code comparisons and regressions run. diff --git a/scripts/benchmark-connection-install-capabilities.ts b/scripts/benchmark-connection-install-capabilities.ts new file mode 100644 index 00000000000..0e6f13e8bc9 --- /dev/null +++ b/scripts/benchmark-connection-install-capabilities.ts @@ -0,0 +1,181 @@ +import assert from "node:assert/strict"; +import { execFileSync } from "node:child_process"; +import { createHash, randomUUID } from "node:crypto"; +import { readFile, mkdtemp, rm, writeFile } from "node:fs/promises"; +import { createRequire } from "node:module"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { + agents, companies, companyMemberships, createDb, principalPermissionGrants, + startEmbeddedPostgresTestDatabase, toolApplications, toolConnections, toolConnectionInstalls, type Db, +} from "../packages/db/src/index.js"; +import { connectionManagementService } from "../server/src/services/connection-management.js"; +import type { AuthorizationActor } from "../server/src/services/authorization.js"; + +// Run explicitly; this benchmark never reads DATABASE_URL or contacts a hosted DB. +// node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --baseline-ref= +const baselineRef = process.argv.find(arg => arg.startsWith("--baseline-ref="))?.slice(15); +assert.ok(baselineRef, "Supply an explicit pre-fix --baseline-ref"); +const repository = fileURLToPath(new URL("../", import.meta.url)); +const git = (...args: string[]) => execFileSync("git", args, { cwd: repository, encoding: "utf8" }).trim(); +const baselineRevision = git("rev-parse", "--verify", `${baselineRef}^{commit}`); +const treatmentRevision = git("rev-parse", "HEAD"); +const servicePath = "server/src/services/connection-management.ts"; +const routePath = "server/src/routes/tool-access.ts"; +const runtimeChanges = git("diff", baselineRevision, "--name-only", "--", "server/src", "packages/db/src", "packages/shared/src") + .split("\n").filter(Boolean).filter(name => !name.includes("/__tests__/") && !/\.test\.[cm]?ts$/.test(name)); +assert.deepEqual(runtimeChanges.filter(name => name !== servicePath && name !== routePath), [], + "Baseline imports current dependencies: isolate any additional changed runtime dependencies first"); +assert.equal(git("diff", baselineRevision, "--", "pnpm-lock.yaml"), "", "Dependency lock must match the pinned baseline"); +const oldSource = execFileSync("git", ["show", `${baselineRevision}:${servicePath}`], { cwd: repository, encoding: "utf8" }); +const requireFromServer = createRequire(new URL("../server/package.json", import.meta.url)); +const serviceUrl = new URL(`../${servicePath}`, import.meta.url); +const rewritten = oldSource.replace(/from "([^"]+)"/g, (_match, specifier: string) => { + const resolved = specifier.startsWith(".") + ? fileURLToPath(new URL(specifier, serviceUrl)) + : requireFromServer.resolve(specifier).replace(/\.cjs$/, ".js"); + return `from ${JSON.stringify(resolved)}`; +}); +const baselineDirectory = await mkdtemp(path.join(tmpdir(), "connection-install-baseline-")); +const baselineFile = path.join(baselineDirectory, "connection-management.mts"); +await writeFile(baselineFile, rewritten); +const baselineModule = await import(pathToFileURL(baselineFile).href) as typeof import("../server/src/services/connection-management.js"); + +type Statement = { sql: string; params: unknown[]; calls: number; rows: number }; +type Counts = { calls: number; rows: number; statements: Map }; +let activeCounts: Counts | undefined; +function instrument(db: Db): Db { + return new Proxy(db, { get(target, key, receiver) { + if (key !== "select") return Reflect.get(target, key, receiver); + return (...fields: Parameters) => { + const builder = target.select(...fields); + const from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const query = from(...args), execute = query.execute.bind(query); + query.execute = (async (...executeArgs: Parameters) => { + const counts = activeCounts; + const compiled = counts ? query.toSQL() : undefined; + const result = await execute(...executeArgs); + if (counts && compiled) { + counts.calls++; + const rows = Array.isArray(result) ? result.length : 0; + counts.rows += rows; + const statement = counts.statements.get(compiled.sql) ?? { ...compiled, calls: 0, rows: 0 }; + statement.calls++; + statement.rows += rows; + counts.statements.set(compiled.sql, statement); + } + return result; + }) as typeof execute; + return query; + }) as typeof builder.from; + return builder; + }; + } }); +} +function summary(samples: number[]) { + const sorted = [...samples].sort((a, b) => a - b); + return { samplesMs: samples, medianMs: sorted[Math.floor(sorted.length / 2)], minMs: sorted[0], maxMs: sorted.at(-1) }; +} + +const keepAlive = setInterval(() => {}, 1_000); +let database: Awaited> | undefined; +let db: Db | undefined; +try { + database = await startEmbeddedPostgresTestDatabase("connection-install-benchmark-"); + db = createDb(database.connectionString); + const observedDb = instrument(db); + const old = baselineModule.connectionManagementService(observedDb); + const treatment = connectionManagementService(observedDb); + assert.equal(typeof treatment.describeAgentInstallCapabilities, "function", "Implement the production batch first"); + const results = []; + for (const n of [1, 8, 32, 100]) { + for (const installs of ["company", "partial"] as const) { + const companyId = randomUUID(), connectionId = randomUUID(), applicationId = randomUUID(), userId = randomUUID(); + const candidateIds = Array.from({ length: n }, () => randomUUID()); + await db.insert(companies).values({ id: companyId, name: `Capability benchmark ${n} ${installs}`, issuePrefix: `B${results.length}` }); + await db.insert(agents).values(candidateIds.map((id, index) => ({ id, companyId, name: `Agent ${index}`, adapterType: "process" }))); + await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "member" }); + await db.insert(principalPermissionGrants).values({ companyId, principalType: "user", principalId: userId, + permissionKey: "tools:manage_profiles", scope: { targetAgentIds: candidateIds } }); + await db.insert(toolApplications).values({ id: applicationId, companyId, name: "Synthetic benchmark", type: "mcp_http" }); + await db.insert(toolConnections).values({ id: connectionId, applicationId, companyId, uid: connectionId, + name: "Synthetic benchmark", transport: "mcp_remote", authKind: "none", config: { provider: "fixture" } }); + await db.insert(toolConnectionInstalls).values(installs === "company" + ? [{ companyId, connectionId, targetType: "company", targetId: companyId }] + : candidateIds.slice(0, Math.max(1, Math.floor(n / 4))).map(targetId => ({ companyId, connectionId, targetType: "agent" as const, targetId }))); + await db.$client.unsafe("ANALYZE"); + const actor: AuthorizationActor = { type: "board", source: "cloud_tenant", userId }; + const resource = { type: "connection_management" as const, companyId, connectionId, operation: "configure", readOnly: false }; + const baseline = async () => { + const capabilities = []; + for (const targetAgentId of candidateIds) capabilities.push(await old.capability(actor, "connections:profiles", { ...resource, targetAgentId })); + return capabilities; + }; + const batch = () => treatment.describeAgentInstallCapabilities(actor, resource, candidateIds); + const samples = { baseline: [] as number[], treatment: [] as number[] }; + const counters: Record = {}; + let captured: Record = {}; + for (let iteration = 0; iteration < 7; iteration++) { + const order = iteration % 2 ? ["treatment", "baseline"] as const : ["baseline", "treatment"] as const; + const outputs: Record = {}; + for (const name of order) { + activeCounts = { calls: 0, rows: 0, statements: new Map() }; + const start = performance.now(); + outputs[name] = await (name === "baseline" ? baseline() : batch()); + const elapsed = performance.now() - start; + const counts = activeCounts; + activeCounts = undefined; + if (iteration >= 2) samples[name].push(elapsed); + const previous = counters[name]; + const current = { calls: counts.calls, rows: counts.rows }; + if (previous) assert.deepEqual(current, previous, "Stable fixture must generate stable work counts"); + counters[name] = current; + if (iteration === 6) captured[name] = [...counts.statements.values()]; + } + assert.deepEqual(outputs.treatment, outputs.baseline, `${installs} N=${n}: full positional capabilities`); + assert.equal((outputs.baseline as Array<{ allowed: boolean }>).length, n); + assert.ok((outputs.baseline as Array<{ allowed: boolean }>).every(capability => capability.allowed), "The benchmark must exercise every authorized retained target"); + } + assert.ok(counters.treatment.calls < counters.baseline.calls || n === 1, "Batch must remove repeated SQL"); + // This is a call-fan-out change. Individual statement plans are controls, + // not the explanation of the gain; capture them outside measured samples. + const plans = []; + for (const statement of captured.treatment) { + const [plan] = await db.$client.unsafe(`EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON) ${statement.sql}`, statement.params as never[]); + plans.push({ ...statement, plan: plan["QUERY PLAN"] }); + } + const before = summary(samples.baseline), after = summary(samples.treatment); + const targetLookups = (statements: Statement[]) => statements.filter(statement => statement.sql.includes('"agents"."id" =')).reduce((sum, statement) => sum + statement.calls, 0); + const expectedRetained = installs === "company" ? n : Math.max(1, Math.floor(n / 4)); + assert.equal(targetLookups(captured.baseline), n * expectedRetained + n - expectedRetained); + assert.equal(targetLookups(captured.treatment), n); + const result = { n, installs, principal: "cloud noncreator member with target-scoped grant", completePositionalEquality: true, + baseline: { ...before, ...counters.baseline, targetLookups: targetLookups(captured.baseline) }, + treatment: { ...after, ...counters.treatment, targetLookups: targetLookups(captured.treatment) }, + speedup: before.medianMs! / after.medianMs!, statements: captured, treatmentPlans: plans }; + results.push(result); + console.error(JSON.stringify({ n, installs, baselineMs: before.medianMs, treatmentMs: after.medianMs, counts: counters })); + } + } + const [version] = await db.$client.unsafe("SELECT version() AS version"); + const settings = await db.$client.unsafe("SELECT name, setting, unit FROM pg_settings WHERE name IN ('shared_buffers','work_mem','effective_cache_size','jit') ORDER BY name"); + console.log(JSON.stringify({ baselineRevision, treatmentRevision, + benchmarkSourceSha256: createHash("sha256").update(await readFile(fileURLToPath(import.meta.url))).digest("hex"), + treatmentSourceSha256: createHash("sha256").update(await readFile(new URL(`../${servicePath}`, import.meta.url))).digest("hex"), + uncommittedDiffSha256: createHash("sha256").update(git("diff", "HEAD")).digest("hex"), + dependencyLockfileSha256: createHash("sha256").update(await readFile(new URL("../pnpm-lock.yaml", import.meta.url))).digest("hex"), + baselineIsolation: { copiedModule: servicePath, imports: "Absolute current paths; all other server/shared/db runtime sources and lockfile verified unchanged from baseline", runtimeChanges }, + node: process.version, postgres: version.version, settings, warmups: 2, samples: 5, alternatingOrder: true, + measurement: "Real service evaluation with real local PostgreSQL; select statement executions and returned Drizzle rows; no simulated transport latency", + limits: "Descriptive service batch only; no hosted latency attribution, agent fast path, route timing, concurrent revocation, or subtree hierarchy scaling claim", + results }, null, 2)); +} finally { + activeCounts = undefined; + try { + await db?.$client.end({ timeout: 0 }); + await database?.cleanup(); + await rm(baselineDirectory, { recursive: true, force: true }); + } finally { clearInterval(keepAlive); } +} diff --git a/server/src/__tests__/connection-install-capabilities.test.ts b/server/src/__tests__/connection-install-capabilities.test.ts new file mode 100644 index 00000000000..dc330ee227b --- /dev/null +++ b/server/src/__tests__/connection-install-capabilities.test.ts @@ -0,0 +1,417 @@ +import { describe, expect, it } from "vitest"; +import { PgDialect } from "drizzle-orm/pg-core"; +import type { SQL } from "drizzle-orm"; +import { + agents, + aiConnectionPools, + companyMemberships, + connectionGrants, + principalPermissionGrants, + routines, + toolConnections, + toolConnectionInstalls, + toolMcpGateways, + toolProfiles, + type Db, +} from "@paperclipai/db"; +import { connectionManagementService } from "../services/connection-management.js"; +import type { AuthorizationActor } from "../services/authorization.js"; +import type { ConnectionManagementResource } from "../services/connection-management-policy.js"; + +const companyId = "00000000-0000-4000-8000-000000000001"; +const connectionId = "00000000-0000-4000-8000-000000000002"; +const userId = "capability-test-user"; +const agentId = (index: number) => `00000000-0000-4000-8000-${String(index + 10).padStart(12, "0")}`; +const actor: AuthorizationActor = { type: "board", source: "cloud_tenant", userId }; +const localActor: AuthorizationActor = { type: "board", source: "local_implicit" }; +const resource: ConnectionManagementResource = { + type: "connection_management", companyId, connectionId, operation: "configure", readOnly: false, +}; +const dialect = new PgDialect(); + +/** Only the query transport is synthetic. Both paths use the real services and policy evaluator. */ +function fixture(count = 3) { + const agentRows = Array.from({ length: count }, (_, index) => ({ + id: agentId(index), companyId, status: "idle", role: "engineer", permissions: {}, reportsTo: null, + })); + const state = { + activeMembership: true, + membershipRole: "member", + creatorId: userId, + connectionPurpose: "tool", + provider: null as string | null, + connectionExists: true, + missingConnections: new Set(), + agentStatus: "idle", + installs: [] as Array<{ targetType: string; targetId: string }>, + missingAgents: new Set(), + targetErrors: new Map(), + grants: new Map | null>(), + poolMembers: [] as Array<{ binding: { connectionId: string } }>, + afterTargetRead: undefined as ((targetId: string) => void) | undefined, + }; + const reads = { total: 0, targetIds: [] as string[], connections: 0, pools: 0, profiles: 0, gateways: 0 }; + const db = { + select() { + return { + from(table: unknown) { + return { + async where(condition: SQL) { + reads.total++; + const query = dialect.sqlToQuery(condition); + if (table === toolConnections) { + reads.connections++; + const selectedId = String(query.params[0]); + return state.connectionExists && query.params[1] === companyId && !state.missingConnections.has(selectedId) ? [{ + id: selectedId, companyId, createdByUserId: state.creatorId, + connectionPurpose: selectedId === connectionId ? state.connectionPurpose : "tool", + credentialPolicy: "shared", config: state.provider ? { provider: state.provider } : {}, transportConfig: {}, + }] : []; + } + if (table === toolConnectionInstalls) return state.installs; + if (table === companyMemberships) return state.activeMembership && query.params[0] === companyId + && query.params[2] === userId ? [{ + companyId, principalType: "user", principalId: userId, + status: "active", membershipRole: state.membershipRole, + }] : []; + if (table === principalPermissionGrants) { + const permissionKey = String(query.params[3]); + return state.grants.has(permissionKey) ? [{ + companyId, principalType: "user", principalId: userId, + permissionKey, scope: state.grants.get(permissionKey), + }] : []; + } + if (table === connectionGrants) return []; + if (table === aiConnectionPools) { + reads.pools++; + return [{ id: connectionId, config: { members: state.poolMembers } }]; + } + if (table === toolProfiles) { reads.profiles++; return []; } + if (table === toolMcpGateways) { reads.gateways++; return []; } + if (table === routines) return []; + if (table === agents) { + if (!query.sql.includes('"agents"."id" =')) return agentRows; + const targetId = String(query.params[0]); + reads.targetIds.push(targetId); + const error = state.targetErrors.get(targetId); + if (error) throw error; + const result = state.missingAgents.has(targetId) ? [] : agentRows + .filter(row => row.id === targetId) + .map(row => ({ ...row, status: state.agentStatus })); + state.afterTargetRead?.(targetId); + return result; + } + throw new Error(`Unexpected capability fixture query: ${query.sql}`); + }, + }; + }, + }; + }, + } as unknown as Db; + const management = connectionManagementService(db); + const candidates = agentRows.map(row => row.id); + const baseline = async ( + ids = candidates, + probeResource = resource, + probeActor = actor, + ) => { + const results = []; + for (const targetAgentId of ids) { + results.push(await management.capability(probeActor, "connections:profiles", { + ...probeResource, readOnly: false, targetAgentId, + })); + } + return results; + }; + return { state, reads, management, candidates, baseline }; +} + +describe("connection agent-install capability batches", () => { + it("does no database work for an empty candidate list", async () => { + const f = fixture(); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, [])).toEqual([]); + expect(f.reads.total).toBe(0); + }); + + it("preserves candidate order, duplicate results and target-scoped grants without retained installs", async () => { + const f = fixture(); + f.state.creatorId = "another-user"; + f.state.grants.set("tools:manage_profiles", { targetAgentIds: [agentId(1)] }); + const candidates = [agentId(1), agentId(0), agentId(1), agentId(2)]; + const expected = await f.baseline(candidates); + expect(expected.map(result => result.allowed)).toEqual([true, false, true, false]); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, candidates)).toEqual(expected); + }); + + it("preserves a mixture of existing and missing candidates", async () => { + const f = fixture(); + const candidates = [agentId(1), agentId(99), agentId(0)]; + const expected = await f.baseline(candidates); + expect(expected.map(result => result.allowed)).toEqual([true, false, true]); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, candidates)).toEqual(expected); + }); + + it("denies every candidate when a retained installed target is unavailable", async () => { + const f = fixture(); + f.state.installs = [{ targetType: "agent", targetId: agentId(99) }]; + const expected = await f.baseline(); + expect(expected.every(result => !result.allowed)).toBe(true); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(expected); + }); + + it("does not hide a retained-target policy denial behind candidate-specific permission", async () => { + const f = fixture(); + f.state.creatorId = "another-user"; + f.state.grants.set("tools:manage_profiles", { targetAgentIds: [agentId(0)] }); + f.state.installs = [{ targetType: "agent", targetId: agentId(1) }]; + const expected = await f.baseline([agentId(0)]); + expect(expected[0]).toMatchObject({ allowed: false }); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)])).toEqual(expected); + }); + + it("preserves candidate-dependent policy checks for an empty retained agent selector", async () => { + const f = fixture(); + f.state.creatorId = "another-user"; + f.state.grants.set("tools:manage_profiles", { targetAgentIds: [agentId(1)] }); + f.state.installs = [{ targetType: "agent", targetId: "" }]; + const expected = await f.baseline(); + expect(expected.map(result => result.allowed)).toEqual([false, true, false]); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(expected); + const laterFault = new Error("later allowed candidate database fault"); + f.state.targetErrors.set(agentId(1), laterFault); + await expect(f.baseline()).rejects.toBe(laterFault); + await expect(f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).rejects.toBe(laterFault); + }); + + it("preserves output for retained candidates, new candidates and duplicates in the same batch", async () => { + const f = fixture(); + f.state.installs = [{ targetType: "agent", targetId: agentId(0) }]; + const candidates = [agentId(0), agentId(2), agentId(0), agentId(99), agentId(1)]; + const expected = await f.baseline(candidates); + expect(expected.map(result => result.allowed)).toEqual([true, true, true, false, true]); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, candidates)).toEqual(expected); + }); + + it("stops at an earlier retained denial without visiting a later infrastructure fault", async () => { + const f = fixture(); + f.state.installs = [ + { targetType: "agent", targetId: agentId(99) }, + { targetType: "agent", targetId: agentId(1) }, + ]; + f.state.targetErrors.set(agentId(1), new Error("later database fault")); + const expected = await f.baseline(); + f.reads.targetIds.length = 0; + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(expected); + expect(f.reads.targetIds).not.toContain(agentId(1)); + }); + + it("propagates an earlier retained infrastructure fault before any later denial", async () => { + const f = fixture(); + f.state.installs = [ + { targetType: "agent", targetId: agentId(0) }, + { targetType: "agent", targetId: agentId(99) }, + ]; + const fault = new Error("first database fault"); + f.state.targetErrors.set(agentId(0), fault); + await expect(f.baseline()).rejects.toBe(fault); + f.reads.targetIds.length = 0; + await expect(f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).rejects.toBe(fault); + expect(f.reads.targetIds).toEqual([agentId(0)]); + }); + + it("converts only 403 faults to denied descriptions and continues in candidate order", async () => { + const f = fixture(); + f.state.targetErrors.set(agentId(1), Object.assign(new Error("synthetic forbidden target"), { status: 403 })); + const expected = await f.baseline(); + expect(expected.map(result => result.allowed)).toEqual([true, false, true]); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(expected); + }); + + it.each([409, 500])("propagates a %s fault without evaluating later candidates", async status => { + const f = fixture(); + const fault = Object.assign(new Error("synthetic infrastructure failure"), { status }); + f.state.targetErrors.set(agentId(1), fault); + await expect(f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).rejects.toBe(fault); + expect(f.reads.targetIds).not.toContain(agentId(2)); + }); + + it("always probes mutation authority even when a caller supplies readOnly or mutation:false", async () => { + const f = fixture(); + f.state.installs = [{ targetType: "agent", targetId: agentId(99) }]; + expect(await f.management.capability(localActor, "connections:profiles", { + ...resource, targetAgentId: agentId(0), + }, { mutation: false })).toMatchObject({ allowed: true }); + const extraCallerOptions = { mutation: false, readContext: undefined }; + expect(await f.management.describeAgentInstallCapabilities(localActor, { ...resource, readOnly: true }, + [agentId(0)], extraCallerOptions)).toEqual(await f.baseline([agentId(0)], resource, localActor)); + }); + + it("ignores a caller's targetAgentId when evaluating each supplied candidate", async () => { + const f = fixture(); + f.state.creatorId = "another-user"; + f.state.grants.set("tools:manage_profiles", { targetAgentIds: [agentId(0)] }); + expect(await f.management.describeAgentInstallCapabilities(actor, { ...resource, targetAgentId: agentId(0) }, + [agentId(1)])).toEqual(await f.baseline([agentId(1)])); + }); + + it("keeps later batches and actual authorization fresh after membership revocation", async () => { + const f = fixture(); + const description = await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)]); + expect(description[0]).toMatchObject({ allowed: true }); + expect(description[0]).not.toHaveProperty("reauthorize"); + const context = await f.management.authorize(actor, "connections:profiles", { ...resource, targetAgentId: agentId(0) }); + f.state.activeMembership = false; + expect(await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)])) + .toEqual(await f.baseline([agentId(0)])); + expect((await f.baseline([agentId(0)]))[0]).toMatchObject({ allowed: false }); + await expect(f.management.authorize(actor, "connections:profiles", { ...resource, targetAgentId: agentId(0) })) + .rejects.toMatchObject({ status: 403 }); + await expect(context.reauthorize()).rejects.toMatchObject({ status: 403 }); + }); + + it("reloads target-scoped grants on the next batch", async () => { + const f = fixture(); + f.state.creatorId = "another-user"; + f.state.grants.set("tools:manage_profiles", { targetAgentIds: [agentId(0)] }); + expect((await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)]))[0].allowed).toBe(true); + f.state.grants.delete("tools:manage_profiles"); + expect((await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)]))[0].allowed).toBe(false); + }); + + it("reloads a deleted connection on the next batch", async () => { + const f = fixture(); + expect((await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)]))[0].allowed).toBe(true); + f.state.connectionExists = false; + const result = await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)]); + expect(result).toEqual(await f.baseline([agentId(0)])); + expect(result[0]).toMatchObject({ allowed: false, reason: "Connection is unavailable in this company" }); + }); + + it("does not reuse a prior actor's or company's descriptions", async () => { + const f = fixture(); + expect((await f.management.describeAgentInstallCapabilities(actor, resource, [agentId(0)]))[0].allowed).toBe(true); + const otherActor: AuthorizationActor = { ...actor, userId: "another-user" }; + const otherResource = { ...resource, companyId: agentId(98) }; + const actorResult = await f.management.describeAgentInstallCapabilities(otherActor, resource, [agentId(0)]); + const companyResult = await f.management.describeAgentInstallCapabilities(actor, otherResource, [agentId(0)]); + expect(actorResult).toEqual(await f.baseline([agentId(0)], resource, otherActor)); + expect(companyResult).toEqual(await f.baseline([agentId(0)], otherResource)); + expect(actorResult[0].allowed).toBe(false); + expect(companyResult[0].allowed).toBe(false); + }); + + it("allows descriptive staleness after a retained check while fresh authorization observes deletion", async () => { + const f = fixture(2); + f.state.installs = [{ targetType: "agent", targetId: agentId(0) }]; + const targetResource = { ...resource, targetAgentId: agentId(1) }; + const context = await f.management.authorize(actor, "connections:profiles", targetResource); + f.state.afterTargetRead = targetId => { + if (targetId === agentId(0)) f.state.missingAgents.add(targetId); + }; + const described = await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates); + expect(described.map(result => result.allowed)).toEqual([true, true]); + await expect(f.management.authorize(actor, "connections:profiles", targetResource)) + .rejects.toMatchObject({ status: 403 }); + await expect(context.reauthorize()).rejects.toMatchObject({ status: 403 }); + const freshDescriptions = await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates); + expect(freshDescriptions.every(result => !result.allowed)).toBe(true); + }); + + it.each([4, 8])("checks company-installed targets once instead of N squared for N=%s", async count => { + const f = fixture(count); + f.state.installs = [{ targetType: "company", targetId: companyId }]; + const expected = await f.baseline(); + expect(f.reads.targetIds).toHaveLength(count * count); + f.reads.targetIds.length = 0; + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(expected); + expect(f.reads.targetIds).toHaveLength(count); + expect(new Set(f.reads.targetIds)).toEqual(new Set(f.candidates)); + }); + + it.each([ + "AAAAAAAA-0000-4000-8000-000000000001", + "", + "not-a-uuid", + ])("uses ordinary evaluation without repeating preparation for candidate %j", async candidate => { + const f = fixture(2); + f.state.installs = [{ targetType: "company", targetId: companyId }]; + const candidates = [agentId(0), candidate, agentId(1)]; + const expected = await f.baseline(candidates); + // The ordinary evaluator treats an empty selector as omitted. + expect(expected.map(result => result.allowed)).toEqual([true, candidate === "", true]); + f.reads.connections = 0; + expect(await f.management.describeAgentInstallCapabilities(actor, resource, candidates)).toEqual(expected); + expect(f.reads.connections).toBe(candidates.length); + }); + + it.each([ + "AAAAAAAA-0000-4000-8000-000000000001", + "not-a-uuid", + ])("uses ordinary evaluation without repeating preparation for retained selector %j", async targetId => { + const f = fixture(); + f.state.installs = [{ targetType: "agent", targetId }]; + const expected = await f.baseline(); + expect(expected.every(result => !result.allowed)).toBe(true); + f.reads.connections = 0; + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(expected); + expect(f.reads.connections).toBe(f.candidates.length); + }); + + it.each(["profileId", "gatewayId", "routineId"] as const)("retains ordinary denial behavior for a missing %s selector", async selector => { + const f = fixture(); + const selected = { ...resource, [selector]: agentId(99) }; + const expected = await f.baseline(f.candidates, selected); + expect(expected.every(result => !result.allowed)).toBe(true); + expect(await f.management.describeAgentInstallCapabilities(actor, selected, f.candidates)).toEqual(expected); + }); + + it("retains recursive AI-pool checks through the ordinary path", async () => { + const f = fixture(2); + f.state.connectionPurpose = "ai"; + f.state.poolMembers = [{ binding: { connectionId: agentId(99) } }]; + const expected = await f.baseline(); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(expected); + expect(f.reads.pools).toBeGreaterThan(0); + f.state.missingConnections.add(agentId(99)); + const denied = await f.baseline(); + expect(denied.every(result => !result.allowed)).toBe(true); + expect(await f.management.describeAgentInstallCapabilities(actor, resource, f.candidates)).toEqual(denied); + }); + + it("retains proposed-provider scope checks through the ordinary path", async () => { + const f = fixture(); + f.state.creatorId = "another-user"; + f.state.provider = "current-provider"; + f.state.grants.set("tools:manage_profiles", { providers: ["current-provider"] }); + const selected = { ...resource, requestedProvider: "replacement-provider" }; + const expected = await f.baseline(f.candidates, selected); + expect(expected.every(result => !result.allowed)).toBe(true); + expect(await f.management.describeAgentInstallCapabilities(actor, selected, f.candidates)).toEqual(expected); + f.state.grants.set("tools:manage_profiles", { providers: ["current-provider", "replacement-provider"] }); + const allowed = await f.baseline(f.candidates, selected); + expect(allowed.every(result => result.allowed)).toBe(true); + expect(await f.management.describeAgentInstallCapabilities(actor, selected, f.candidates)).toEqual(allowed); + }); + + it("retains configure escalation for a proposed company installation", async () => { + const f = fixture(); + f.state.creatorId = "another-user"; + f.state.grants.set("tools:manage_profiles", null); + expect((await f.baseline())[0].allowed).toBe(true); + const selected = { ...resource, requestedInstallCompanyIds: [companyId] }; + const expected = await f.baseline(f.candidates, selected); + expect(expected.every(result => !result.allowed)).toBe(true); + expect(await f.management.describeAgentInstallCapabilities(actor, selected, f.candidates)).toEqual(expected); + }); + + it("retains inactive-agent rejection through the ordinary path", async () => { + const f = fixture(); + f.state.agentStatus = "paused"; + const agentActor: AuthorizationActor = { + type: "agent", source: "agent_key", companyId, agentId: agentId(0), onBehalfOfUserId: userId, + }; + const expected = await f.baseline(f.candidates, resource, agentActor); + expect(expected.every(result => !result.allowed)).toBe(true); + expect(await f.management.describeAgentInstallCapabilities(agentActor, resource, f.candidates)).toEqual(expected); + }); +}); diff --git a/server/src/__tests__/connection-management.integration.test.ts b/server/src/__tests__/connection-management.integration.test.ts index 2a91b321306..bbf43adfc18 100644 --- a/server/src/__tests__/connection-management.integration.test.ts +++ b/server/src/__tests__/connection-management.integration.test.ts @@ -6,7 +6,7 @@ import { join } from "node:path"; import { createServer } from "node:http"; import { and, eq, sql } from "drizzle-orm"; import { beforeAll, afterAll, describe, expect, it, vi } from "vitest"; -import { createDb, activityLog, agentApiKeys, companies, companySecrets, companySecretVersions, chatConversations, chatDeliveries, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, toolPolicies, toolRateLimitCounters, connectionGrants, connectionGrantDelegations, toolAccessAuditEvents, toolOauthStates, issueThreadInteractions, connectionIntentDeliveries, agents, companyMemberships, heartbeatRuns, issues, principalPermissionGrants, projects, toolConnections, toolMcpGateways, toolMcpGatewayTokens, toolActionRequests, toolCatalogEntries, toolInvocations, toolCallEvents, toolProfileBindings, plugins, adapterAuthSessions, environments, routines, chatGitHubReviews } from "@paperclipai/db"; +import { createDb, authUsers, activityLog, agentApiKeys, companies, companySecrets, companySecretVersions, chatConversations, chatDeliveries, chatEndpoints, chatExternalPrincipals, chatIdentityLinks, toolPolicies, toolRateLimitCounters, connectionGrants, connectionGrantMembers, connectionGrantDelegations, toolAccessAuditEvents, toolOauthStates, issueThreadInteractions, connectionIntentDeliveries, agents, companyMemberships, heartbeatRuns, issues, principalPermissionGrants, projects, toolApplications, toolConnections, toolConnectionInstalls, toolMcpGateways, toolMcpGatewayTokens, toolActionRequests, toolCatalogEntries, toolInvocations, toolCallEvents, toolProfiles, toolProfileBindings, plugins, adapterAuthSessions, environments, routines, chatGitHubReviews } from "@paperclipai/db"; import { findSatisfiedToolConnection, satisfiedConnectionIntentService } from "../services/satisfied-connection-intents.js"; import { connectionManagementService, continuingConnectionManagement } from "../services/connection-management.js"; import { redactRunnerApiPayloads } from "../services/native-runtime/runner-api-log-redaction.js"; @@ -25,6 +25,8 @@ import { secretService } from "../services/secrets.js"; import { localEncryptedProvider } from "../secrets/local-encrypted-provider.js"; import { instanceSettingsService } from "../services/instance-settings.js"; import { PaperclipRunnerToolAuthority } from "../services/native-runtime/paperclip-runner-tool-authority.js"; +import type { AuthorizationActor } from "../services/authorization.js"; +import type { ConnectionManagementResource } from "../services/connection-management-policy.js"; vi.mock("../services/agent-lifecycle-driver.js", () => ({ createLifecycleDriver: () => ({ requiredPluginIds: async () => [], runPlugin: async () => "complete", runHost: async () => "complete" }), @@ -85,6 +87,172 @@ describe("connection management with production default agents", () => { authority: new PaperclipRunnerToolAuthority(server.db, { companyId: base.companyId, agentId: agent.id, issueId: base.issueId, runId: base.runId, apiUrl: server.apiUrl, storage: server.storage }) }; } + it("matches ordinary install capabilities on real PostgreSQL for scoped, retained and fallback resources", async () => { + const f = await server.fixture({ connectionScenario: "fresh" }); + const candidateIds = [f.agentId, randomUUID(), randomUUID()]; + await server.db.insert(agents).values(candidateIds.slice(1).map((id, index) => ({ id, companyId: f.companyId, name: `Capability target ${index}`, adapterType: "process" }))); + const [application] = await server.db.insert(toolApplications).values({ companyId: f.companyId, name: "Capability differential", type: "mcp_http" }).returning(); + const [connection] = await server.db.insert(toolConnections).values({ companyId: f.companyId, applicationId: application.id, + uid: randomUUID(), name: "Capability differential", transport: "mcp_remote", authKind: "none", createdByUserId: "different-creator" }).returning(); + await server.db.insert(principalPermissionGrants).values({ companyId: f.companyId, principalType: "user", principalId: f.responsibleUserId!, + permissionKey: "tools:manage_profiles", scope: { targetAgentIds: candidateIds.slice(0, 2) } }); + const service = connectionManagementService(server.db); + const actor: AuthorizationActor = { type: "board", userId: f.responsibleUserId, source: "cloud_tenant" }; + const resource = { type: "connection_management" as const, companyId: f.companyId, connectionId: connection.id, operation: "configure", readOnly: false }; + const compare = async (base: ConnectionManagementResource = resource, candidates: readonly string[] = candidateIds, principal = actor) => { + const baseline = []; + for (const targetAgentId of candidates) baseline.push(await service.capability(principal, "connections:profiles", { ...base, targetAgentId })); + const batch = await service.describeAgentInstallCapabilities(principal, base, candidates); + expect(batch).toEqual(baseline); + return batch; + }; + expect((await compare()).map(result => result.allowed)).toEqual([true, true, false]); + expect(await service.capability(actor, "connections:profiles", resource)).toMatchObject({ allowed: false }); + expect(await compare(resource, [])).toEqual([]); + expect((await compare(resource, [candidateIds[1], candidateIds[0], candidateIds[1]])).map(result => result.allowed)).toEqual([true, true, true]); + // A saved empty text target leaves the candidate ID in the ordinary + // evaluator's resolved policy input, so it is not a shared target prefix. + await server.db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection.id, targetType: "agent", targetId: "" }); + const emptyTargetCandidates = [candidateIds[2], candidateIds[0]]; + const emptyTargetBaseline = []; + for (const targetAgentId of emptyTargetCandidates) { + emptyTargetBaseline.push(await service.capability(actor, "connections:profiles", { ...resource, targetAgentId })); + } + expect(emptyTargetBaseline.map(capability => capability.allowed)).toEqual([false, true]); + expect(await service.describeAgentInstallCapabilities(actor, resource, emptyTargetCandidates)).toEqual(emptyTargetBaseline); + await server.db.delete(toolConnectionInstalls).where(eq(toolConnectionInstalls.connectionId, connection.id)); + await server.db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection.id, targetType: "agent", targetId: candidateIds[0] }); + expect((await compare()).map(result => result.allowed)).toEqual([true, true, false]); + await server.db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection.id, targetType: "agent", targetId: candidateIds[2] }); + expect((await compare()).every(result => !result.allowed)).toBe(true); + await server.db.delete(toolConnectionInstalls).where(eq(toolConnectionInstalls.connectionId, connection.id)); + await server.db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection.id, targetType: "company", targetId: f.companyId }); + expect((await compare()).every(result => !result.allowed)).toBe(true); + // JavaScript target sets distinguish UUID spelling; PostgreSQL's uuid + // lookup does not. Noncanonical candidates must retain ordinary evaluation. + for (const spellings of [candidateIds.map(id => id.toUpperCase()), + [candidateIds[0].toUpperCase(), candidateIds[1], candidateIds[0], candidateIds[1].toUpperCase(), candidateIds[0].toUpperCase()]]) { + expect((await compare(resource, spellings, { type: "board", source: "local_implicit" })).map(result => result.allowed)) + .toEqual(spellings.map(() => true)); + } + await server.db.delete(toolConnectionInstalls).where(eq(toolConnectionInstalls.connectionId, connection.id)); + await server.db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection.id, targetType: "agent", targetId: randomUUID() }); + expect((await compare(resource, candidateIds, { type: "board", source: "local_implicit" })).map(result => result.reason)) + .toEqual(candidateIds.map(() => "Target agent is outside this company")); + await server.db.delete(toolConnectionInstalls).where(eq(toolConnectionInstalls.connectionId, connection.id)); + const [foreign] = await server.db.insert(agents).values({ companyId: f.foreignCompanyId, name: "Foreign capability target", adapterType: "process" }).returning(); + expect((await compare(resource, [candidateIds[0], foreign.id], { type: "board", source: "local_implicit" })).map(result => result.allowed)) + .toEqual([true, false]); + const [profile] = await server.db.insert(toolProfiles).values({ companyId: f.companyId, name: "Fallback profile", profileKey: randomUUID(), defaultAction: "deny" }).returning(); + await compare({ ...resource, profileId: profile.id }); + // Proposed changes also stay on the full evaluator, including own-property presence. + await compare({ ...resource, requestedProvider: null }); + const faultingDb = new Proxy(server.db, { get(target, key, receiver) { + if (key !== "select") return Reflect.get(target, key, receiver); + return (...fields: Parameters) => { + const builder = target.select(...fields), from = builder.from.bind(builder); + builder.from = ((...args: Parameters) => { + const query = from(...args), execute = query.execute.bind(query); + query.execute = (async (...executeArgs: Parameters) => { + const compiled = query.toSQL(); + if (compiled.sql.includes('"agents"."id" =') && compiled.params[0] === candidateIds[1]) { + // A real database failure at the visited target boundary, rather + // than a fake permission denial or a mocked query result. + await server.db.$client.unsafe("SELECT 1 / 0"); + } + return execute(...executeArgs); + }) as typeof execute; + return query; + }) as typeof builder.from; + return builder; + }; + } }); + const faulting = connectionManagementService(faultingDb); + const localActor: AuthorizationActor = { type: "board", source: "local_implicit" }; + await expect(faulting.capability(localActor, "connections:profiles", { ...resource, targetAgentId: candidateIds[1] })).rejects.toThrow(/division by zero/); + await expect(faulting.describeAgentInstallCapabilities(localActor, resource, [candidateIds[1]])).rejects.toThrow(/division by zero/); + await server.db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection.id, targetType: "agent", targetId: randomUUID() }); + const denied = await faulting.capability(localActor, "connections:profiles", { ...resource, targetAgentId: candidateIds[1] }); + expect(denied).toMatchObject({ allowed: false, reason: "Target agent is outside this company" }); + expect(await faulting.describeAgentInstallCapabilities(localActor, resource, [candidateIds[1]])).toEqual([denied]); + }); + + it("preserves the complete grants GET across human roles and denies a real install mutation after revocation", async () => { + const f = await server.fixture({ connectionScenario: "fresh" }); + const candidateIds = [f.agentId, randomUUID(), randomUUID()]; + await server.db.insert(agents).values(candidateIds.slice(1).map((id, index) => ({ id, companyId: f.companyId, name: `Grant target ${index}`, adapterType: "process" }))); + const users = { creator: f.responsibleUserId!, manager: randomUUID(), member: randomUUID(), viewer: randomUUID(), inactive: randomUUID(), + admin: randomUUID(), owner: randomUUID(), operator: randomUUID() }; + for (const [name, userId] of Object.entries(users)) { + if (name !== "creator") { + await server.db.insert(authUsers).values({ id: userId, name, email: `${userId}@fixture.invalid`, emailVerified: true, createdAt: new Date(), updatedAt: new Date() }); + await server.db.insert(companyMemberships).values({ companyId: f.companyId, principalType: "user", principalId: userId, + membershipRole: ["viewer", "admin", "owner", "operator"].includes(name) ? name : "member", status: name === "inactive" ? "inactive" : "active" }); + } + } + for (const permissionKey of ["tools:manage_connections", "tools:manage_profiles", "tools:use", "agents:configure"]) { + await server.db.insert(principalPermissionGrants).values([users.manager, users.admin].map(principalId => ({ companyId: f.companyId, principalType: "user", principalId, permissionKey }))); + } + await server.db.insert(principalPermissionGrants).values(["tools:manage_profiles", "agents:configure"].map(permissionKey => ({ companyId: f.companyId, + principalType: "user", principalId: users.member, permissionKey, + scope: permissionKey === "tools:manage_profiles" ? { targetAgentIds: candidateIds.slice(0, 2) } : null }))); + const [application] = await server.db.insert(toolApplications).values({ companyId: f.companyId, name: "Grants response parity", type: "mcp_http" }).returning(); + const [connection] = await server.db.insert(toolConnections).values({ companyId: f.companyId, applicationId: application.id, + uid: randomUUID(), name: "Grants response parity", transport: "mcp_remote", authKind: "none", createdByUserId: users.creator }).returning(); + const grants = await server.db.insert(connectionGrants).values([ + { companyId: f.companyId, connectionId: connection.id, kind: "organization", createdByUserId: users.creator, isDefault: true }, + ...[users.creator, users.manager, users.member].map((subjectUserId, index) => ({ companyId: f.companyId, connectionId: connection.id, kind: "user" as const, + subjectUserId, createdByUserId: subjectUserId, updatedAt: new Date(`2026-10-2${index + 1}T12:00:00.000Z`) })), + ]).returning(); + await server.db.insert(connectionGrantMembers).values({ companyId: f.companyId, grantId: grants[0].id, subjectType: "user", subjectId: users.member }); + await server.db.insert(toolConnectionInstalls).values({ companyId: f.companyId, connectionId: connection.id, targetType: "agent", targetId: candidateIds[0] }); + const cookies = Object.fromEntries(await Promise.all(Object.entries(users).map(async ([name, userId]) => [name, await server.humanSession(userId)]))); + const request = async (name: keyof typeof users, method = "GET", body?: unknown) => { + const response = await fetch(`${server.apiUrl}/api/tool-connections/${connection.id}/${method === "GET" ? "grants" : "installs"}`, { + method, headers: { cookie: cookies[name], origin: server.apiUrl, "Content-Type": "application/json" }, + ...(body === undefined ? {} : { body: JSON.stringify(body) }), + }); + return { status: response.status, body: await response.json() }; + }; + const original = connectionManagementModule.connectionManagementService; + const legacy = vi.spyOn(connectionManagementModule, "connectionManagementService").mockImplementation(db => { + const service = original(db); + return { ...service, describeAgentInstallCapabilities: async (actor, base, candidates, options) => { + const result = []; + for (const targetAgentId of candidates) result.push(await service.capability(actor, "connections:profiles", { ...base, targetAgentId, readOnly: false }, options)); + return result; + } }; + }); + const baseline = new Map>>(); + try { + for (const name of Object.keys(users) as Array) baseline.set(name, await request(name)); + } finally { legacy.mockRestore(); } + const actual = new Map>>(); + for (const name of Object.keys(users) as Array) { + actual.set(name, await request(name)); + expect(actual.get(name), name).toEqual(baseline.get(name)); + } + expect(actual.get("manager")).toMatchObject({ status: 200, body: { capabilities: { canConfigure: true, canViewOtherPersonalIdentities: true, editableAgentIds: expect.arrayContaining(candidateIds) } } }); + expect(actual.get("manager")!.body.grants).toHaveLength(4); + expect(actual.get("creator")!.body.capabilities).toMatchObject({ canConfigure: true, canViewOtherPersonalIdentities: false }); + expect(actual.get("creator")!.body.members.length).toBeGreaterThan(0); + expect(actual.get("member")!.body.capabilities).toMatchObject({ canConfigure: false, canViewOtherPersonalIdentities: false, editableAgentIds: expect.arrayContaining(candidateIds.slice(0, 2)) }); + expect(actual.get("member")!.body.capabilities.editableAgentIds).toHaveLength(2); + expect(actual.get("member")!.body.grants.map((grant: { id: string }) => grant.id)).toEqual([grants[0].id, grants[3].id]); + expect(actual.get("member")!.body.members).toEqual([]); + expect(actual.get("viewer")!.body.capabilities).toMatchObject({ canConfigure: false, canManageAgentInstalls: false, editableAgentIds: [] }); + expect(actual.get("inactive")!.status).toBe(404); + const installs = [{ targetType: "agent", targetId: candidateIds[0] }, { targetType: "agent", targetId: candidateIds[1] }]; + const beforeRevoke = await request("member", "PUT", { installs }); + expect(beforeRevoke, JSON.stringify(beforeRevoke)).toMatchObject({ status: 200 }); + await server.db.delete(principalPermissionGrants).where(and(eq(principalPermissionGrants.companyId, f.companyId), eq(principalPermissionGrants.principalId, users.member))); + expect(await request("member")).toMatchObject({ status: 200, body: { capabilities: { editableAgentIds: [], canManageAgentInstalls: false } } }); + expect(await request("member", "PUT", { installs: [{ targetType: "agent", targetId: candidateIds[0] }] })).toMatchObject({ status: 403 }); + const savedInstalls = await server.db.select().from(toolConnectionInstalls).where(eq(toolConnectionInstalls.connectionId, connection.id)); + expect(savedInstalls.map(install => install.targetId)).toEqual(expect.arrayContaining(candidateIds.slice(0, 2))); + expect(savedInstalls).toHaveLength(2); + }); + it("creates and configures tool connections and profiles with fresh grants and actual key/JWT authentication", async () => { const f = await fixture(); const grants = await server.db.select().from(principalPermissionGrants).where(eq(principalPermissionGrants.principalId, f.agentId)); diff --git a/server/src/routes/tool-access.ts b/server/src/routes/tool-access.ts index 1218ff4809d..3f3bb760e17 100644 --- a/server/src/routes/tool-access.ts +++ b/server/src/routes/tool-access.ts @@ -687,9 +687,9 @@ function connectorEnrollmentPrincipal(req: Request): string { const editableAgentIds: string[] = []; if (mutationCapable) { const companyAgents = await db.select({ id: agents.id }).from(agents).where(eq(agents.companyId, connection.companyId)); - for (const agent of companyAgents) { - const capability = await management.capability(req.actor, "connections:profiles", { ...resource, targetAgentId: agent.id }, { readContext: connectionRequestContext(req) }); - if (capability.allowed) editableAgentIds.push(agent.id); + const capabilities = await management.describeAgentInstallCapabilities(req.actor, resource, companyAgents.map(agent => agent.id), { readContext: connectionRequestContext(req) }); + for (let i = 0; i < companyAgents.length; i++) { + if (capabilities[i].allowed) editableAgentIds.push(companyAgents[i].id); } } return { diff --git a/server/src/services/connection-management.ts b/server/src/services/connection-management.ts index 0c1740f04ba..cf65d2afb34 100644 --- a/server/src/services/connection-management.ts +++ b/server/src/services/connection-management.ts @@ -20,6 +20,22 @@ export type ConnectionManagementContext = { type CapturedConnectionRun = RunIdentityCapture<"task_authorization">; const connectionReadRuns = new WeakMap(); +type AuthorizationOptions = { targetAgentIds?: string[]; mutation?: boolean; afterHumanAction?: boolean; expandedProfile?: boolean; gatewayContextChecked?: boolean; readContext?: ConnectionManagementContext; capabilityProbe?: boolean }; +type PreparedAuthorization = { + actor: AuthorizationActor; + action: ConnectionManagementAction; + resource: ConnectionManagementResource; + options: AuthorizationOptions; + responsibleUserId: string | null; + workContext: ConnectionManagementContext["workContext"]; + capturedRun: CapturedConnectionRun | undefined; + expandedTargets: Set; + referencedConnections: string[]; + currentGatewayResource: ConnectionManagementResource | undefined; + savedNonAiConnection: boolean; +}; +type ConnectionCapability = { allowed: boolean; reason: string; operation: string }; + export function responsibleConnectionUser(actor: AuthorizationActor): string | null { return (actor.type === "agent" ? actor.onBehalfOfUserId : actor.userId) ?? null; } @@ -28,12 +44,12 @@ export function responsibleConnectionUser(actor: AuthorizationActor): string | n export function connectionManagementService(db: Db) { const authorization = authorizationService(db); - async function authorize( + async function prepareAuthorization( actor: AuthorizationActor, action: ConnectionManagementAction, resource: ConnectionManagementResource, - options: { targetAgentIds?: string[]; mutation?: boolean; afterHumanAction?: boolean; expandedProfile?: boolean; gatewayContextChecked?: boolean; readContext?: ConnectionManagementContext; capabilityProbe?: boolean } = {}, - ): Promise { + options: AuthorizationOptions, + ): Promise { const responsibleUserId = responsibleConnectionUser(actor); let workContext: ConnectionManagementContext["workContext"] = { runId: null, issueId: null, projectId: null }; let capturedRun: CapturedConnectionRun | undefined; @@ -72,6 +88,7 @@ export function connectionManagementService(db: Db) { const expandedTargets = new Set(options.targetAgentIds ?? []); const referencedConnections: string[] = []; let currentGatewayResource: ConnectionManagementResource | undefined; + let savedNonAiConnection = false; if (resource.gatewayId && !options.gatewayContextChecked) { const [gateway] = await db.select().from(toolMcpGateways).where(and(eq(toolMcpGateways.id, resource.gatewayId), eq(toolMcpGateways.companyId, resource.companyId))); if (!gateway) throw forbidden("Gateway is unavailable in this company"); @@ -142,6 +159,7 @@ export function connectionManagementService(db: Db) { if (resource.connectionId) { const [connection] = await db.select().from(toolConnections).where(and(eq(toolConnections.id, resource.connectionId), eq(toolConnections.companyId, resource.companyId))); if (!connection) throw forbidden("Connection is unavailable in this company"); + savedNonAiConnection = connection.connectionPurpose !== "ai"; resource = { ...resource, selfServiceUserId: connection.createdByUserId, purpose: connection.connectionPurpose === "ai" ? "ai" : connection.connectionPurpose === "channel" ? "chat" : "tool", provider: typeof connection.config.sourceTemplateKey === "string" ? connection.config.sourceTemplateKey @@ -222,34 +240,47 @@ export function connectionManagementService(db: Db) { if (issue.workMode === "skill_test" || ((options.mutation ?? action !== "connections:read") && issue.workMode !== "standard")) throw forbidden("Connection changes require a standard task"); } } + return { actor, action, resource, options, responsibleUserId, workContext, capturedRun, + expandedTargets, referencedConnections, currentGatewayResource, savedNonAiConnection }; + } + + async function evaluateTarget(prepared: PreparedAuthorization, targetAgentId: string | undefined): Promise { + const { actor, action, resource, responsibleUserId } = prepared; + const resolved = { ...resource, ...(targetAgentId ? { targetAgentId } : {}) }; + const decision = await authorization.decide({ actor, action, resource: resolved, scope: { ...resolved } }); + if (!decision.allowed) throw forbidden(decision.explanation, { reason: decision.reason, code: decision.code, operation: resource.operation }); + if (Object.hasOwn(resource, "requestedProvider") && resource.requestedProvider !== resource.provider) { + const proposed = { ...resolved, provider: resource.requestedProvider }; + const proposedDecision = await authorization.decide({ actor, action, resource: proposed, scope: { ...proposed } }); + if (!proposedDecision.allowed) throw forbidden(proposedDecision.explanation, { reason: proposedDecision.reason, code: proposedDecision.code, operation: resource.operation }); + } + if (actor.type === "agent" && actor.runId && (resolved.projectId || resolved.issueId)) { + const resourceTrust = await authorization.decide({ actor: { ...actor, runId: undefined }, action, resource: resolved, scope: { ...resolved } }); + if (!resourceTrust.allowed) throw forbidden(resourceTrust.explanation); + } + if (targetAgentId) { + const [target] = await db.select({ id: agents.id }).from(agents).where(and(eq(agents.id, targetAgentId), eq(agents.companyId, resource.companyId))); + if (!target) throw forbidden("Target agent is outside this company"); + // Explicitly enforce both principals even when unrelated authorization is in shadow mode. + // Existing human routes retain their domain-specific target authority. + // Delegation additionally checks both the acting agent and its human. + const principals: AuthorizationActor[] = actor.type === "agent" || resource.selfServiceAgentInstalls ? [actor] : []; + if (actor.type === "agent") principals.push({ type: "board", userId: responsibleUserId, source: "session", ignoreInstanceAdmin: true }); + for (const principal of principals) { + const targetDecision = await authorization.decide({ actor: principal, action: "agent_config:update", resource: { type: "agent", companyId: resource.companyId, agentId: targetAgentId }, scope: { targetAgentId } }); + if (!targetDecision.allowed) throw forbidden(targetDecision.explanation); + } + } + return decision; + } + + async function finishAuthorization(prepared: PreparedAuthorization): Promise { + const { actor, action, resource, options, responsibleUserId, workContext, capturedRun, + expandedTargets, referencedConnections, currentGatewayResource } = prepared; const targets = [...new Set([...expandedTargets, ...(resource.targetAgentId ? [resource.targetAgentId] : [])])]; let decision: AuthorizationDecision | undefined; for (const targetAgentId of targets.length ? targets : [undefined]) { - const resolved = { ...resource, ...(targetAgentId ? { targetAgentId } : {}) }; - decision = await authorization.decide({ actor, action, resource: resolved, scope: { ...resolved } }); - if (!decision.allowed) throw forbidden(decision.explanation, { reason: decision.reason, code: decision.code, operation: resource.operation }); - if (Object.hasOwn(resource, "requestedProvider") && resource.requestedProvider !== resource.provider) { - const proposed = { ...resolved, provider: resource.requestedProvider }; - const proposedDecision = await authorization.decide({ actor, action, resource: proposed, scope: { ...proposed } }); - if (!proposedDecision.allowed) throw forbidden(proposedDecision.explanation, { reason: proposedDecision.reason, code: proposedDecision.code, operation: resource.operation }); - } - if (actor.type === "agent" && actor.runId && (resolved.projectId || resolved.issueId)) { - const resourceTrust = await authorization.decide({ actor: { ...actor, runId: undefined }, action, resource: resolved, scope: { ...resolved } }); - if (!resourceTrust.allowed) throw forbidden(resourceTrust.explanation); - } - if (targetAgentId) { - const [target] = await db.select({ id: agents.id }).from(agents).where(and(eq(agents.id, targetAgentId), eq(agents.companyId, resource.companyId))); - if (!target) throw forbidden("Target agent is outside this company"); - // Explicitly enforce both principals even when unrelated authorization is in shadow mode. - // Existing human routes retain their domain-specific target authority. - // Delegation additionally checks both the acting agent and its human. - const principals: AuthorizationActor[] = actor.type === "agent" || resource.selfServiceAgentInstalls ? [actor] : []; - if (actor.type === "agent") principals.push({ type: "board", userId: responsibleUserId, source: "session", ignoreInstanceAdmin: true }); - for (const principal of principals) { - const targetDecision = await authorization.decide({ actor: principal, action: "agent_config:update", resource: { type: "agent", companyId: resource.companyId, agentId: targetAgentId }, scope: { targetAgentId } }); - if (!targetDecision.allowed) throw forbidden(targetDecision.explanation); - } - } + decision = await evaluateTarget(prepared, targetAgentId); } if (action === "connections:profiles" && resource.requestedInstallCompanyIds && !resource.selfServiceAgentInstalls) { await authorize(actor, "connections:configure", resource, options); @@ -275,6 +306,15 @@ export function connectionManagementService(db: Db) { return context; } + async function authorize( + actor: AuthorizationActor, + action: ConnectionManagementAction, + resource: ConnectionManagementResource, + options: AuthorizationOptions = {}, + ): Promise { + return finishAuthorization(await prepareAuthorization(actor, action, resource, options)); + } + async function authorizeBindingTarget(actor: AuthorizationActor, action: ConnectionManagementAction, resource: ConnectionManagementResource, binding: { targetType: string; targetId: string }, options: NonNullable[3]> = {}): Promise { @@ -303,17 +343,96 @@ export function connectionManagementService(db: Db) { return authorize(actor, action, resolved, checkOptions); } - async function capability(...args: Parameters) { + function deniedCapability(error: unknown, operation: string): ConnectionCapability { + // Database/provider faults are not authorization denials. + if (!(error instanceof Error && "status" in error && error.status === 403)) throw error; + return { allowed: false, reason: error.message, operation }; + } + + async function capability(...args: Parameters): Promise { try { const context = await authorize(args[0], args[1], args[2], { ...args[3], capabilityProbe: true }); return { allowed: true, reason: context.decision.reason, operation: args[2].operation }; } catch (error) { - // Database/provider faults are not authorization denials. - if (!(error instanceof Error && "status" in error && error.status === 403)) throw error; - return { allowed: false, reason: error.message, operation: args[2].operation }; + return deniedCapability(error, args[2].operation); } } - return { authorize, authorizeBindingTarget, capability }; + + /** Descriptions only: retained installation authority is checked once per + * call. Mutations and human continuations always start a fresh authorization. + * Complex resources and delegated actors retain the ordinary evaluator. */ + async function describeAgentInstallCapabilities( + actor: AuthorizationActor, + resource: ConnectionManagementResource, + candidateAgentIds: readonly string[], + options: { readContext?: ConnectionManagementContext } = {}, + ): Promise { + if (!candidateAgentIds.length) return []; + const candidates = [...candidateAgentIds]; + const base = { ...resource, readOnly: false }; + const probeOptions = { readContext: options.readContext, capabilityProbe: true, mutation: true }; + const results: ConnectionCapability[] = []; + const describeRemaining = async (start: number) => { + for (let i = start; i < candidates.length; i++) { + results.push(await capability(actor, "connections:profiles", { ...base, targetAgentId: candidates[i] }, probeOptions)); + } + return results; + }; + + let prepared: PreparedAuthorization; + try { + prepared = await prepareAuthorization(actor, "connections:profiles", { ...base, targetAgentId: candidates[0] }, probeOptions); + } catch (error) { + results.push(deniedCapability(error, base.operation)); + return describeRemaining(1); + } + const resolved = prepared.resource; + const isCanonicalAgentId = (id: string) => isUuidLike(id) && id === id.toLowerCase(); + const eligible = actor.type === "board" && prepared.savedNonAiConnection + && [resolved.profileId, resolved.gatewayId, resolved.routineId, resolved.projectId, resolved.issueId].every(id => id == null) + && !Object.hasOwn(resolved, "requestedProvider") && !Object.hasOwn(resolved, "requestedInstallCompanyIds") + && !resolved.selfServiceAgentInstalls + && !prepared.referencedConnections.length && !prepared.currentGatewayResource + && candidates.every(isCanonicalAgentId) + // A saved empty selector resolves against the current candidate in the + // ordinary evaluator. Such retained checks cannot be shared. + && [...prepared.expandedTargets].every(isCanonicalAgentId); + if (!eligible) { + // Consume the already prepared first evaluation. Restarting it can repeat + // recursive checks or run-identity reconciliation and its locking effects. + try { + const context = await finishAuthorization(prepared); + results.push({ allowed: true, reason: context.decision.reason, operation: base.operation }); + } catch (error) { + results.push(deniedCapability(error, base.operation)); + } + return describeRemaining(1); + } + + let retainedDecision: AuthorizationDecision | undefined; + try { + for (const targetAgentId of prepared.expandedTargets) { + retainedDecision = await evaluateTarget(prepared, targetAgentId); + } + } catch (error) { + const denied = deniedCapability(error, base.operation); + return candidates.map(() => ({ ...denied })); + } + for (const targetAgentId of candidates) { + try { + // With no retained targets, the named candidate is the first policy + // target: a targetless preflight can incorrectly deny scoped grants. + const decision = prepared.expandedTargets.has(targetAgentId) + ? retainedDecision! + : await evaluateTarget(prepared, targetAgentId); + results.push({ allowed: true, reason: decision.reason, operation: base.operation }); + } catch (error) { + results.push(deniedCapability(error, base.operation)); + } + } + return results; + } + return { authorize, authorizeBindingTarget, capability, describeAgentInstallCapabilities }; } /** A pending provider login can finish after a successful yielding run. This From 39e811c9c5226979fce42e052a27cfb28301018a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 27 Oct 2026 21:00:05 -0500 Subject: [PATCH 41/41] fix(runner): prepare diagnostic proxy and reduce bundled skill queries (#17941) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native turns reconcile company skills and select verified remote runtime packs. > - Catalog imports repeat folder transactions for skills in the same bundled category. > - Diagnostic requests can also reuse an older image proxy that lacks the phase collector. > - This pull request reuses categories within one import and selects the current proxy for fresh diagnostics. > - Each import has its own category map, and recovery keeps its admitted runtime identity. > - The result is less preparation work and a runtime that can provide the requested measurements. ## Linked Issues or Issue Description Refs #17918. **What happened?** A hosted run retained `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` but emitted no provider phase batch. Its saved pack authority selected an older image proxy. The deployed server proxy contained the collector. The image proxy did not. Bundled inventory reconciliation also repeats folder transactions for skills in the same category. The paired local benchmark exposes that preparation cost. **Expected behavior** A fresh diagnostic request must use a verified proxy that supports the current collector. Bundled skill reconciliation should resolve each category once per import. **Steps to reproduce** 1. Use a protected image pack with the same OpenCode interface revision and an older proxy hash. 2. Enable `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` for a fresh remote run. 3. Observe that the current backend accepts the older proxy and cannot emit the requested phase batch. ## What Changed - Reuse each bundled category during one import. Every skill still runs its existing validation, access lookup, value comparison, and write path. - Compare the installed proxy hash with the current build when a fresh run requests diagnostics. - Keep compatible-pack reuse for ordinary runs and exact admitted identity for recovery. - Cover selection through the real backend, including recovery after diagnostics are enabled. - Document runtime preparation costs and admission limits for diagnostic requests. The skill importer owns category reuse. The provider verifier owns the artifact comparison. The native backend passes the resolved diagnostic requirement at the existing admission seam. The changes stay in those existing owners and share one latency investigation and rollout. ## Verification - Before the fix, the new verifier and backend cases failed because both accepted a stale proxy. - After the fix, 63 focused cases passed in 8.10 seconds. They cover normal reuse, current diagnostic proxy selection, stale proxy rejection, selected ACPX profiles, and admitted recovery identity. - 110 skill and folder service cases passed against isolated real PostgreSQL databases. Only cluster bootstrap used the existing local PostgreSQL service. Queries, migrations, authorization, file handling, and transactions ran on their real paths. - A paired benchmark used the same immutable skill snapshot, authenticated actor, 170-skill inventory, and 3.5 ms delay in each network direction. It checked every delivered file and executable bit. Warm median fell from 3.319 s to 2.482 s (25.2%). SQL calls fell by 81 in each matched sample, from 241–243 to 160–162. The bundled phase median fell from 2.596 s to 1.721 s. The final measurement uses commit `37eac8d8de`; only the skill importer differs from baseline `7951455073`. These local results are not hosted acceptance. - `pnpm -r --workspace-concurrency=1 typecheck` passed with the pinned pnpm 9.15.4 and a 6 GiB Node heap. Earlier attempts exposed a subprocess toolchain mismatch, a local IPC restriction, and a fixture type error. Each was corrected before the passing run. - Module boundary and whitespace checks passed. - `pnpm build` passed for all workspaces. - `pnpm test:run` did not complete. Embedded PostgreSQL initialization failed after repeated startup attempts. The owned test subtree was stopped after preserving the failure log. This is not a full local test pass. On final commit `37eac8d8de`, all 53 successful check contexts passed; two optional Storybook checks were skipped. Greptile is 5/5 with no unresolved findings. The draft awaits merge approval. - The catalog change has a measured local gain. The proxy change fixes missing phase capture. Real hosted response latency remains above the target and needs a new matched measurement after delivery. ## Risks The category map belongs to one import and one company. The importer still resolves current skill data on every invocation. Fresh diagnostic runs can prepare a current verified pack, which can add startup time. Admission can fail when matching runtime assets are unavailable. Ordinary runs keep the existing fast compatibility path. Recovery keeps the already admitted identity, even if diagnostics are enabled later. The operator endpoint gate, collector limits, and closed span fields are unchanged. ## Model Used OpenAI GPT-6 through Codex, with repository inspection, shell execution, browser tools, and Honeycomb queries. The session does not expose a verifiable backend deployment ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/observability.md | 8 +++++ server/src/services/company-skills.ts | 9 +++-- .../native-session-executor.test.ts | 36 ++++++++++++------- .../native-runtime/native-session-executor.ts | 2 ++ .../remote-provider-pack-verification.test.ts | 14 ++++++-- .../remote-provider-pack-verification.ts | 11 ++++-- 6 files changed, 60 insertions(+), 20 deletions(-) diff --git a/doc/observability.md b/doc/observability.md index 324273807ec..e8004fe4f93 100644 --- a/doc/observability.md +++ b/doc/observability.md @@ -354,6 +354,14 @@ phase names, wall timestamps, and success/failure, with a 1 MiB request limit, 128 retained measurements, and 16 measurements per phase. No operator OTLP endpoint or ingestion credential is passed to OpenCode. +For fresh remote diagnostic runs, the selected pack must contain the current +build's OpenCode proxy. An otherwise compatible image pack can predate the +collector. Such a pack cannot satisfy the diagnostic request. The existing +verified pack preparation path selects or stages the current proxy; it can add +startup time or fail admission when matching runtime assets are unavailable. +Ordinary runs retain compatible image-pack reuse. Recovery retains the pack +identity already admitted for that session. + Available phases cover context/instruction loading, skill discovery, tool and model resolution, model streaming (`ai.streamText`, including stream consumption), Git discovery/default branch, snapshots, dependency installation, and plugin startup. diff --git a/server/src/services/company-skills.ts b/server/src/services/company-skills.ts index d2819de5714..1a874631923 100644 --- a/server/src/services/company-skills.ts +++ b/server/src/services/company-skills.ts @@ -6385,6 +6385,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo database: DbOrTransaction = db, ): Promise { const out: CompanySkill[] = []; + const bundledFolders = new Map>>(); for (const skill of imported) { assertImportedSkillKeyAllowed(skill); assertImportedSkillSourceAllowed(skill); @@ -6412,9 +6413,11 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo const parsed = parseFrontmatterMarkdown(skill.markdown); const storeMetadata = readSkillStoreMetadata(parsed.frontmatter, metadata); const bundledCategory = paperclipBundledFolderCategory(skill.key, incomingMeta); - const bundledFolder = bundledCategory - ? await folderSvc.ensureBundledCategory(companyId, bundledFolderLabel(bundledCategory)) - : null; + let bundledFolder = bundledCategory ? bundledFolders.get(bundledCategory) : undefined; + if (bundledCategory && !bundledFolder) { + bundledFolder = await folderSvc.ensureBundledCategory(companyId, bundledFolderLabel(bundledCategory)); + bundledFolders.set(bundledCategory, bundledFolder); + } const projectId = asString(incomingMeta.projectId); const projectName = asString(incomingMeta.projectName); const projectFolder = !existing && incomingKind === "project_scan" && projectId && projectName diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index d78b215d16c..57420904e29 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -1659,9 +1659,11 @@ describe("remote provider pack manifest", () => { return JSON.stringify(value); }; - it.each(["opencode", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => { + it.each(["opencode", "opencode-diagnostics", "opencode-diagnostics-stale", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => { const architecture = provider === "claude-arm64" ? "arm64" : "x64"; const claude = provider.startsWith("claude-"); + const opencode = provider.startsWith("opencode"); + const diagnostics = provider.startsWith("opencode-diagnostics"); const root = await mkdtemp(join(tmpdir(), "paperclip-bundled-image-")); const digest = "sha256:" + "a".repeat(64); const artifacts = Object.fromEntries(Object.entries({ @@ -1669,7 +1671,7 @@ describe("remote provider pack manifest", () => { productionLock: "pnpm-lock.yaml", opencodeCommand: "node_modules/.bin/opencode", opencodeExecutable: "node_modules/opencode-ai/bin/opencode.exe", opencodeProxy: "dist/cli/opencode-app-server-proxy.cjs", acpxSidecar: "dist/cli/acpx-runtime-sidecar.cjs", - }).map(([key, path]) => [key, { path, sha256: digest }])) as Record<"nodeCommand" | "opencodeCommand", { path: string; sha256: string }>; + }).map(([key, path]) => [key, { path, sha256: digest }])) as Record<"nodeCommand" | "opencodeCommand" | "opencodeProxy", { path: string; sha256: string }>; const payload = { compatibility: { opencode: 1, acpx: 1 }, target: { platform: "linux", architecture }, runnerSourceRevision: "a".repeat(40), @@ -1695,16 +1697,17 @@ describe("remote provider pack manifest", () => { daemon.writeUInt16LE(architecture === "arm64" ? 183 : 62, 18); await writeFile(state.bundledRunnerBinary, daemon); if (claude) state.resolveRemoteRunnerBinary.mockResolvedValue(state.bundledRunnerBinary); - const remoteExecution = { ...execution, provider: provider === "opencode" + const remoteExecution = { ...execution, provider: opencode ? { kind: "opencode", model: null } : { kind: "acpx", agent: claude ? "claude" : "cursor", model: null }, workspace: { ...execution.workspace, cwd: root }, - session: { ...execution.session, normalizedSessionId: `bundled-${provider}-${randomUUID()}`, driverKind: provider === "opencode" ? "opencode_server" : "acpx_runtime" }, + session: { ...execution.session, normalizedSessionId: `bundled-${provider}-${randomUUID()}`, driverKind: opencode ? "opencode_server" : "acpx_runtime" }, } as unknown as NativeExecutionInputV1; const syncIn = vi.fn(async (_operations: { files: { kind: string }[] }[]) => undefined); const installed = structuredClone(manifest); - if (provider === "opencode") { + if (opencode) { installed.payload.runnerSourceRevision = "c".repeat(40); installed.payload.artifacts.nodeCommand.sha256 = "sha256:" + "d".repeat(64); + if (provider !== "opencode-diagnostics") installed.payload.artifacts.opencodeProxy.sha256 = "sha256:" + "e".repeat(64); installed.digest = "sha256:" + createHash("sha256").update(canonical(installed.payload)).digest("hex"); } const execute = vi.fn(async (command: { command: string; args?: string[] }) => ({ @@ -1717,12 +1720,18 @@ describe("remote provider pack manifest", () => { }) : "", })); try { - await createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution, + const prepare = createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution, runnerInstanceId: "bundled-image-runner", runnerIngressAuthorized: true, - ...(provider === "opencode" ? { runnerEnvironment: { PAPERCLIP_NATIVE_MCP_NAME: "assigned-fixture", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-fixture", PAPERCLIP_NATIVE_MCP_TOKEN: "private-fixture" } } : {}), + ...(opencode ? { runnerEnvironment: { ...configuredEnvironmentProjection(diagnostics ? { PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" } : {}), PAPERCLIP_NATIVE_MCP_NAME: "assigned-fixture", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-fixture", PAPERCLIP_NATIVE_MCP_TOKEN: "private-fixture" } } : {}), runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute, syncIn } } as never }); + if (provider === "opencode-diagnostics-stale") { + await expect(prepare).rejects.toThrow("image provider pack did not match the bundled release identity"); + expect(state.createBackend).not.toHaveBeenCalled(); + return; + } + await prepare; state.createTransport.mockClear(); state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); const transport = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { @@ -1730,9 +1739,10 @@ describe("remote provider pack manifest", () => { environment?: NodeJS.ProcessEnv; }; expect(transport.runnerBinary).toBe(state.bundledRunnerBinary); - if (provider === "opencode") { + if (opencode) { expect(state.createAssignedMcpTools).toHaveBeenCalledWith(expect.objectContaining({ gatewayPublicId: "assigned-fixture", bearerToken: "private-fixture", delivery: "on_demand" })); for (const name of ["PAPERCLIP_NATIVE_MCP_NAME", "PAPERCLIP_NATIVE_MCP_URL", "PAPERCLIP_NATIVE_MCP_TOKEN"]) expect(transport.environment?.[name]).toBeUndefined(); + expect(transport.environment?.PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS).toBe(diagnostics ? "1" : undefined); } if (claude) { expect(state.resolveRemoteRunnerBinary).toHaveBeenCalledWith(expect.objectContaining({ remoteCwd: "/workspace", configuredPath: undefined })); @@ -1750,7 +1760,8 @@ describe("remote provider pack manifest", () => { expect(execute.mock.calls.filter(([call]) => call.args?.[0] === "--build-metadata")).toHaveLength(1); expect(execute.mock.calls.some(([call]) => call.args?.[0] === "-e" || call.args?.[0] === "--version")).toBe(false); expect(syncIn.mock.calls.every(([operations]) => operations.every(operation => operation.files.every(file => file.kind === "file")))).toBe(true); - if (provider === "claude-x64") { + if (provider === "claude-x64" || provider === "opencode") { + const admitted = opencode ? installed : manifest; const identity = transport.prpIdentity!; const artifact = { version: "0.0.0", digest: "sha256:" + createHash("sha256").update(daemon).digest("hex") }; resolveRemoteProviderPackAuthority({ stateDirectory: transport.stateDirectory!, @@ -1758,7 +1769,7 @@ describe("remote provider pack manifest", () => { workspaceId: remoteExecution.binding.executionWorkspaceId, sessionId: remoteExecution.session.normalizedSessionId!, runnerInstanceId: identity.runnerInstanceId, environmentLeaseId: identity.environmentLeaseId, remoteCwd: "/workspace", providerDigest: nativeSha256(remoteExecution.provider) }, - reattach: false, current: () => manifest, runnerArtifact: () => artifact }).verified(); + reattach: false, current: () => admitted, runnerArtifact: () => artifact }).verified(); await mkdir(join(transport.stateDirectory!, "control-plane"), { recursive: true }); await writeFile(join(transport.stateDirectory!, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity))); const next = structuredClone(manifest); @@ -1771,7 +1782,7 @@ describe("remote provider pack manifest", () => { schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], - providerPacks: [{ root: "/opt/paperclip-runner/provider-pack", manifest, immutable: true }], + providerPacks: [{ root: "/opt/paperclip-runner/provider-pack", manifest: admitted, immutable: true }], }) }; if (request.args[0] === "-e" || request.args[0] === "--version") return { exitCode: 0, timedOut: false, stderr: "", stdout: request.args[0] === "--version" ? "1.18.34" : "" }; const script = request.args[1]; @@ -1781,6 +1792,7 @@ describe("remote provider pack manifest", () => { }); await createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution, runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true, + ...(opencode ? { runnerEnvironment: configuredEnvironmentProjection({ PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" }) } : {}), runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace", environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute: recoveryExecute }, sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never, @@ -1790,7 +1802,7 @@ describe("remote provider pack manifest", () => { }); state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); const recovered = state.createTransport.mock.calls.at(-1)![0]; - expect(recovered).toMatchObject({ providerPackAuthorityDigest: manifest.digest, runnerArtifact: artifact, adoptExistingRunner: { pid: 123 } }); + expect(recovered).toMatchObject({ providerPackAuthorityDigest: admitted.digest, runnerArtifact: artifact, adoptExistingRunner: { pid: 123 } }); } } finally { state.bundledPackManifestPath = null; state.bundledRunnerBinary = null; diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index b1cb4ccdade..630027317ab 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -12444,6 +12444,8 @@ async function createRunnerdBackendWithinSessionClaim( manifest: expected, installation, provider: input.execution.provider as { kind: "opencode" } | { kind: "acpx"; agent: string }, requireExactIdentity: reattachProviderPack || explicitProviderPackSelection, + requireCurrentOpenCodeProxy: !reattachProviderPack + && runnerEnvironment?.PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS === "1", })); if (!reattachProviderPack) providerPackAuthority!.select(manifest); expectedProviderPackManifest = manifest; diff --git a/server/src/services/native-runtime/remote-provider-pack-verification.test.ts b/server/src/services/native-runtime/remote-provider-pack-verification.test.ts index e5c8f742329..70f110045f2 100644 --- a/server/src/services/native-runtime/remote-provider-pack-verification.test.ts +++ b/server/src/services/native-runtime/remote-provider-pack-verification.test.ts @@ -61,11 +61,12 @@ describe("installed provider pack compatibility", () => { } }); const reseal = (pack: ProviderPackIdentity) => ({ ...pack, digest: sha(canonicalProviderPackJson(pack.payload)) }); - const check = (installed: ProviderPackIdentity, options: { immutable?: boolean; requireExactIdentity?: boolean } = {}) => { + const check = (installed: ProviderPackIdentity, options: { immutable?: boolean; requireExactIdentity?: boolean; requireCurrentOpenCodeProxy?: boolean } = {}) => { const execute = vi.fn(); return verifyRemoteProviderPack({ runner: { execute }, cwd: "/workspace", packRoot: "/opt/paperclip-runner/provider-pack", manifest: manifest("b"), installation: { root: "/opt/paperclip-runner/provider-pack", manifest: installed, immutable: options.immutable ?? true }, - provider: { kind: "opencode" }, requireExactIdentity: options.requireExactIdentity }); + provider: { kind: "opencode" }, requireExactIdentity: options.requireExactIdentity, + requireCurrentOpenCodeProxy: options.requireCurrentOpenCodeProxy }); }; it.each(["interface", "platform", "schema", "artifact_path", "digest", "legacy"])("rejects an incompatible %s", async field => { let pack = manifest("a"); @@ -85,6 +86,15 @@ describe("installed provider pack compatibility", () => { await expect(check(manifest("a"), { requireExactIdentity: true })).rejects.toThrow("incompatible"); await expect(check(manifest("b"), { requireExactIdentity: true })).resolves.toEqual(manifest("b")); }); + it("requires the current proxy for requested OpenCode diagnostics", async () => { + const installed = manifest("a"); + await expect(check(installed)).resolves.toEqual(installed); + await expect(check(installed, { requireCurrentOpenCodeProxy: true })) + .rejects.toThrow("requested OpenCode diagnostics"); + installed.payload.artifacts.opencodeProxy = manifest("b").payload.artifacts.opencodeProxy; + await expect(check(reseal(installed), { requireCurrentOpenCodeProxy: true })) + .resolves.toEqual(reseal(installed)); + }); it("ignores unrelated provider and dependency changes for OpenCode", async () => { const pack = manifest("a"); pack.payload.compatibility!.acpx += 1; diff --git a/server/src/services/native-runtime/remote-provider-pack-verification.ts b/server/src/services/native-runtime/remote-provider-pack-verification.ts index 68170e23089..64f0bf0d649 100644 --- a/server/src/services/native-runtime/remote-provider-pack-verification.ts +++ b/server/src/services/native-runtime/remote-provider-pack-verification.ts @@ -34,7 +34,7 @@ export type PackProvider = { kind: "opencode" } | { kind: "acpx"; agent: string */ function compatibleInstallation( installation: ProviderPackInstallation, expected: T, packRoot: string, - provider: PackProvider, requireExactIdentity: boolean, + provider: PackProvider, requireExactIdentity: boolean, requireCurrentOpenCodeProxy: boolean, ): T { const actual = installation.manifest as T | null; const payload = actual?.payload; @@ -56,6 +56,10 @@ function compatibleInstallation( throw new Error("runner_remote_provider_artifact_incompatible: installed provider pack artifact identity is invalid"); } } + if (provider.kind === "opencode" && requireCurrentOpenCodeProxy + && payload.artifacts.opencodeProxy?.sha256 !== expected.payload.artifacts.opencodeProxy?.sha256) { + throw new Error("runner_remote_provider_artifact_incompatible: installed proxy does not match the requested OpenCode diagnostics"); + } if (provider.kind === "acpx") { const agent = provider.agent; const currentProfile = expected.payload.acpxProfileDigests?.[agent]; @@ -72,7 +76,7 @@ function compatibleInstallation( return structuredClone(actual); } -export async function verifyRemoteProviderPack({ runner, cwd, packRoot, manifest, installation, provider, requireExactIdentity = false }: { +export async function verifyRemoteProviderPack({ runner, cwd, packRoot, manifest, installation, provider, requireExactIdentity = false, requireCurrentOpenCodeProxy = false }: { runner: CommandManagedRuntimeRunner; cwd: string; packRoot: string; @@ -80,10 +84,11 @@ export async function verifyRemoteProviderPack({ installation?: ProviderPackInstallation; provider?: PackProvider; requireExactIdentity?: boolean; + requireCurrentOpenCodeProxy?: boolean; }): Promise { if (installation) { if (!provider) throw new Error("runner_remote_provider_artifact_incompatible: provider interface is required"); - return compatibleInstallation(installation, manifest, packRoot, provider, requireExactIdentity); + return compatibleInstallation(installation, manifest, packRoot, provider, requireExactIdentity, requireCurrentOpenCodeProxy); } const expected = Buffer.from( canonicalProviderPackJson(manifest),